Introduction
Scientific fields such as physics and math are considered to be exact sciences. Biology, including microbiology, is different. Microbiological systems possess a greater number of uncontrollable variables which results in increased experimental complexity and decreased experimental accuracy. The great number of variables at play in a given microbiological study arise from the inherent complexity of living systems. Each variable contributes a certain degree of error which, depending on the system, may propagate linearly or non-linearly. The more variables involved, the more errors expected, resulting in changes to the outcome of a given study. Practicing microbiologists are used to – even count on – such variability in their studies. However, customers of Microchem Laboratory are sometimes puzzled or surprised, since they expect all sciences to yield accurate and precise results. This article discusses the causes of the inherent variability in antimicrobial testing.
Human Studies as a Window into the Variability of Microbiological Studies
Most people appreciate the challenges of measuring the effect of a single variable among many others in the context of human studies. For example, consider clinical studies, where large trials based on promising smaller preclinical studies often fail (1). Large trials would not be required if biological experiments were precisely replicable. In some instances even the results of large-scale trials are difficult to reproduce. Or consider research done to determine whether or not using a cell phone causes brain cancer. Of more than a dozen studies carried out, roughly one third indicate cell phone usage increases the risk of brain tumors, roughly another third of studies indicate cell phones have no effect on tumor formation, and the rest indicate cell phone usage protects people from brain tumors. In such studies, researchers attempt to isolate the effect of a single variable among the thousands of potential variables inherent to the biological systems under study, ranging from geographical location to chemical exposures to genetics.
While bacteria and fungi are not as complex as humans and animals, they are able to carry out a wide range of chemical transformations and are sensitive to subtle changes in the experimental environment. Small changes in the surrounding environment can affect proteins and other molecules in and on the cells, causing chemical, physical, and biological reactions (2, 3, 4, 5). And although an individual microorganism is less complex than an individual human, they propagate much more quickly and number in the billions or trillions in most studies. Just like humans, bacteria and fungi do not always behave predictably under study.
The cause for variability in antimicrobial efficacy studies can be roughly divided into three parts: The test system (e.g. microorganism and environmental conditions), the scientist(s) performing the study, and the test substance.
Variables in the Test System
Each part of a test system, which includes the microorganisms, the test surfaces or reaction vessel, and the various environmental conditions, includes numerous variables and each one can have an affect on the outcome of a study. Temperature and humidity is an important consideration because it can vary greatly between testing laboratories or even on a daily basis within the same laboratory. Most standardized test methods try to control for those differences by giving a narrow temperature range at which bacteria have to be cultured (e.g. 36°C ± 2°C) or by requiring the test substance to be held at a constant temperature during the study. However, these ranges are often set by scientific consensus; scant empirical evidence suggests the ranges are tight enough to limit study-to-study variability. The differences in the environmental conditions have an impact on the microorganisms, causing them to contribute an important source of variability to the study.
Bacteria grown in liquid cultures (and fungi grown on agar) are not homogenous. The cells pass through different growth stages during their short lives and some cells within the culture might be more susceptible to a test substance than others. Bacteria, some more than others, also tend to form clumps, which may protect the bacteria on the inside of the group from the harsh testing environment or even the test substance itself. In an attempt to control this source of variability, some testing methods require multiple subcultures of the microorganism before it can be used and several methods include steps designed to reduce, but not completely eliminate, the presence of clumps of microorganisms. There is little evidence to suggest that multiple subcultures decreases variability and it certainly results in older cultures. Older cultures have a different cell composition than fresher cultures and can therefore react differently to antimicrobials. Additionally, many of the official testing methods are silent about whether bacterial cultures are incubated statically or dynamically. As dynamically incubated cultures are exposed to more gas exchange, their physiology differs from statically grown bacteria (6).
The strain of test microorganism under study can also affect the outcome. In the United States, the Environmental Protection Agency (EPA) tries to minimize this source of variability as much as possible by recommending the usage of specific bacterial and fungal strains (7). For example, there are over 50 different strains of the bacterium Staphylococcus aureus listed on the ATCC (American Type Culture Collection) website, a nonprofit biological resource center that focuses on the distribution of standard reference microorganisms. Each of these strains exhibits different genetic properties and potentially reacts differently when exposed to the same test substance. Therefore, the EPA recommends only one specific S. aureus strain (S. aureus ATCC 6538) for disinfectant and sanitizer testing. Despite all the effort, the level of variability within a strain is poorly understood. For example, it is well known that the S. aureus ATCC 6538 strain may present as both white and yellow colonies on the same agar plate, indicating at least a moderate level of phenotypic (visually apparent) variance from one colony of this organism to the next
Another important cause of variability is microbiological growth media and the growth conditions used. Different brands of media used in different laboratories, each with different laboratory water, can contain different amounts of minerals and nutrients or have different pH values, and therefore can have different affects on the physiology and metabolism of the bacteria (3). Again, the EPA tries to control for that by recommending specific growth media. However, media vary by standard method and most large laboratories make growth media in the laboratory, so each batch will differ slightly from one another. Consider that automobiles run differently with just a couple of percent differences in fuel octane levels. Given the growth medium literally is transformed by the microorganisms into more microorganisms, it stands to reason that microorganisms will propagage differently in growth mediums that are slightly different. Unfortunately, the effect of subtle differences in growth medium composition on microbial makeup and proliferation have not been well studied and are not well understood.
Variability Caused by the Operator
As previously mentioned, humans are complex beings. Despite the best scientific efforts, people will not perform a movement exactly the same way twice in a row. In studies where movements such as spraying a disinfectant or wiping a test surface are involved, this adds variability that can hardly be avoided. Every time when test samples or test surfaces are manipulated, they will be treated slightly differently. Most methods used in the United States for disinfectant registration involve some sort of inherently variable manipulation of the test substance by the scientist. Other examples of experimental variability brought on by operator actions include inoculation of test surfaces differs from operator to operator and from one test surface to the next.
Small changes in pipetting and mixing cultures, inoculation of the carriers, drying of microorganisms onto the test surface or treatment of the test surface add variability to a test. A good example is the AOAC Germicidal Spray Products Test, Modified for Wipes. In this test, glass carriers with dried bacteria are treated with a test wipe. Different scientists will apply different pressure onto the wipe and have slightly different ways of holding the wipe. Recognizing this as a great potential source of experimental variability, the EPA recently convened a workshop to discuss the best way of wiping, folding, and holding a wipe during studies. Factors like the pressure used when wiping are difficult if not impossible to standardize and represent an example of variability from the test system.
Variability Caused by the Test Substance
The third potential source of variability is the test substance itself. Disinfectants are like any other chemicals; many factors can influence their stability and activity. For example, the age of the test substance is a consideration. Like other chemicals, the active ingredients in disinfectants have a half life that is influenced by factors such as pH and temperature (8). Adding complication, the half life of a disinfectant like bleach will be much less than the half life of a quaternary disinfectant, and may vary depending on the container in which it is stored or how frequently it is mixed.
In an attempt to evaluate this element of test substance variability, for new disinfectant claims, the EPA historically required that three different lots of a disinfectant be tested, one of which was required to be at least 60 days old. This requirement was introduced because the same disinfectant can perform differently when tested at different ages. However, the EPA now appears to be relaxing this requirement in exchange for manufacturers testing their products at artificially low active ingredient concentration levels. Regardless of EPA’s current policy, it important for manufacturers of antimicrobial substances to perform stability tests to determine the activity of their product over time, since the age of the product is so closely tied to activity.
The requirement for testing three different lots of an antimicrobial agent was introduced by the EPA since different lots can vary slightly in their composition and therefore in their activity. Disinfectants rarely consist of only one ingredient. They are most often mixtures of active and inactive ingredients at different concentrations. Differences between disinfectant lots can either be caused by variability during the production process or by differences between the active ingredients. While companies strive to keep the quality and composition of their antimicrobial products the same, slight variations caused by factors such as different personnel or different environmental conditions will result in slight differences between each lot. Therefore, testing three lots rather than just one increases the robustness of a study and makes sure that the expected small differences in the ingredient composition do not affect the efficacy of the antimicrobial.
Synthesizing a disinfectant or other antimicrobial agent can be a complex process in which multiple chemicals are being added to each other to create the final product. The order at which they are being added can have an affect on the outcome of the final product as chemicals can react as soon as they come in contact with each other. This can result in the change of reactant concentrations, changing the composition of the disinfectant. It is therefore important to take the order of ingredient addition into account when evaluating the efficacy variability of an antimicrobial substance.
Some disinfectants are designed to be diluted. Most often, they are diluted for antimicrobial efficacy tests using so-called hard water, which is formulated on-site at efficacy testing labs and is similar to tap water in terms of mineral content. The hardness of water (as PPM calcium) is known to decrease the efficacy of antimicrobial solutions, some being more affected than others. Therefore, disinfectants that are sold to be diluted by the consumer are usually tested with water of a certain hardness (e.g. 300 ppm). It can be expected, that a disinfectant will perform differently if the water used for the dilution exhibits different levels of hardness, and this adds another layer of variability if different batches of laboratory-made hard water are used in studies.
Antimicrobials come in all kind of forms and shapes. They can be liquid or viscous, they can be coated onto a hard surface or integrated into a textile. Some forms are more prone to variability than others. For liquid antimcrobials, the way the fluid covers a surface covered with microorganisms is important and another potential source of study variability. Generally speaking, liquids with low surface tension (with surfactants) outperform those without surfactants when all other things are held equal.
A viscous substance is more difficult to mix with microorganisms than a liquid substance which can increase the degree of error. The preservative efficacy screen or the USP <51>, for example, are testing methods affected by this variable. These methods are used to test the effectiveness of preservatives in samples such as creams or lotions by inoculating them with challenge microorganisms and observing their viability over the course of 1-4 weeks. A more viscous substance is also more difficult to sample, which can further increase the variability.
Surfactants are helpful in the context of surfaces that are coated or impregnated with antimicrobial agents. Surfactants are often necessary to facilitate spreading of the bacteria on hydrophobic hard surfaces and textiles. This is important as hydrophobic properties of surfaces can cause beading of the microbial inoculum, preventing some bacteria from interacting with the antimicrobial coating. At the same time, surfactant or detergent containing liquid antimicrobial substances will show improved coverage of a test surface versus substances that do not contain these supplemental ingredients, resulting in reduced efficacy.
Another problem can arise when during production, antimicrobial substances are not evenly incorporated onto hard surfaces or textiles. This can result in some parts of the same test sample being more antimicrobial than others, causing differences in reduction of microbial numbers. This is an example of a source of variability outside the control of the laboratory, and possibly outside the control of the manufacturer as well.
Other factors that can interfere with the active ingredient are culture supplements such as “soil” (required for one-step disinfectant claims) or blood, which some microorganisms need to grow (9). When exposed to the bacterial or fungal inoculum, the activity of a test substance can be reduced through self-quenching over time. The higher the initial inoculum, the larger the effect. Because of the reasons discussed above, getting the exact same amount of microorganisms on each carrier is impossible, which adds yet another source of variability into antimicrobial efficacy studies.
Variability from Unknown Sources
A number of known causes of variability have been discussed, but it is possible there are other sources of variability that are just as substantial but are unknown to microbiologists. For example, scientists may one day discover that certain test organisms engage a defense mechanism to certain disinfectants when in the presence of other members of the same strain, but only if grown in the presence of light. Or it could be the case that a given microorganism – say E. coli ATCC 11229 for the sake of example – mutates at a much, much higher rate than other bacteria, to the extent that it is nearly impossible for two laboratories to compare results after just a few subcultures. There is precedent for such “unknown” sources of variability, such as when some bacteria were shown to recover from UV damage when incubated with the lights on but not when incubated in the dark (a phenomenon now understood to be caused by a bacterial DNA repair enzyme called photolyase).
What Level of Variability is to be Expected in an Antimicrobial Efficacy Test?
To consider ordinary levels of variability, we must first consider the broad range in numbers of microorganisms in a study over time. For example, an AOAC Use Dilution Test may utilize nearly 1 trillion microorganisms at the outset as part of the initial inoculum for the penicylinders, but be reduced to 3 surviving cells after the test. These kinds of numbers are much different than studies, say, of wolf populations in Yellowstone Park
Accordingly, microbial populations are not usually expressed using a linear scale, but rather on a log scale. An easy way to understand the logarithmic scale is to view it as powers of 10. For example the logarithm (log) of 10,000 for the base of 10 is 4 (=104). A four log reduction would mean that only one bacterial (or fungal) cell out of 10,000 survived which is equal to a percent reduction of 99.99%. A three log reduction corresponds to a 99.9% reduction, a two log reduction corresponds to a 99% reduction, and a one log reduction would translate into 90%.
In the microbial world the difference between 1 log and 4 logs is large, especially when you start out with numbers like 1 million cells. A one log reduction would still leave 100,000 live cells behind while a four log reduction would bring that number down to 10. A 40% or 50% reduction might seem like a good results at first glance, but actually the microbial numbers are not reduced by a significant amount. As an example, the EPA requires at least a 99.9% (or 3 log) reduction for a non-food contact sanitizer test. For a food contact sanitizer test, that reduction increases to a 99.999% (or 5 log) reduction. With all that in mind, a variability of 50%-90% is quite common. As a rule of thumb, for suspension tests a 0.5 log variability is ordinary while for surface tests that number increases to 1 log and sometimes more.
However, that can also have its advantages. If experimental repeats are performed by different scientists and/or on different days and the study outcome is the same, it indicates a low level of variability and adds robustness to the results, making them more meaningful.
What Can be Done to Reduce the Impact of Variability?
In general, experimental results have to be reproducible in order to be meaningful. Now that we know that variability in biological systems is impossible to avoid, what can be done to reduce its impact on meaningfulness? The best strategy of enhancing the likelihood of reproducibility of a study is to choose a laboratory with experienced microbiologists and an eye toward controlling variability, then increase experimental robustness through greater numbers of replicates in studies. Increasing the number of replicates or individual measurements decreases the contribution of errors (4). One good example for this is the AOAC Use-Dilution Method or the AOAC Germicidal Spray Products Test. For an initial disinfectant registration, the EPA requires both of these tests to be performed with 60 carriers out of which 59 have to pass the test, giving a 95% confidence level. This high number of replicates ensures that a positive or negative outcome did not just happen by chance.
References:
- Lowenstein, P.R. and M.G. Castro, Uncertainty in the translation of preclinical experiments to clinical trials. Why do most phase III clinical trials fail? Curr Gene Ther, 2009. 9(5): p. 368-74.
- Shapiro, R.S. and L.E. Cowen, Thermal control of microbial development and virulence: molecular mechanisms of microbial temperature sensing. MBio, 2012. 3(5).
- Mceldowney, S. and M. Fletcher, Effect of Growth Conditions and Surface Characteristics of Aquatic Bacteria on Their Attachment to Solid Surfaces. Journal of General Microbiology, 1986. 132(2): p. 513-523.
- Casadevall, A. and F.C. Fang, Reproducible science. Infect Immun, 2010. 78(12): p. 4972-5.
- Giacomini, A., et al., Experimental conditions may affect reproducibility of the beta-galactosidase assay. FEMS Microbiol Lett, 1992. 79(1-3): p. 87-90.
- Stepanovic, S., et al., Influence of dynamic conditions on biofilm formation by staphylococci. Eur J Clin Microbiol Infect Dis, 2001. 20(7): p. 502-4.
- McDonnell, G. and A.D. Russell, Antiseptics and disinfectants: activity, action, and resistance. Clin Microbiol Rev, 1999. 12(1): p. 147-79. 9. Ruano, M., J. El-Attrache, and P. Villegas, Efficacy comparisons of disinfectants used by the commercial poultry industry. Avian Dis, 2001. 45(4): p. 972-7.
