RESEARCH DESIGN

How Do You Study Outcomes of Tailoring?
Tailoring seeks to improve the efficacy of interventions by varying the ingredients of the treatment according to observable characteristics of respondents. The key to successful tailoring is to determine the causal factors that influence the outcome, and the presence of interactions between these factors and individual characteristics: if the effect of a factor is not modified by individual characteristics, then there is no gain in tailoring the factor according to those characteristics.

Successful tailoring needs to identify the active treatment components, investigate their interrelationships, model for whom and when they impact outcomes, and select their settings judiciously to optimize the potency of the resulting treatment formulation. There are many research designs that can help identify the active components of tailoring.

The landscape of disease prevention in general, and of computer-based tailored interventions in particular, is changing rapidly. Advances in the science of prevention and in communications technologies are enabling dramatic changes in the types of feasible treatment delivery systems, measurement, and new treatment formulations. In fact, new communication technologies are moving well beyond our understanding of basic message content, presentation, and delivery principles. Current design strategies are inadequate in coping with the challenges posed by these developments. Large intervention package trials are unable to systematically unravel the active components of the "black box" [1]. If we continue to use traditional constructive or dismantling studies, in which components are added or subtracted one at a time, we would need years or decades to explore a small number of potentially active ingredients. By the time the findings of these endeavors are disseminated, the technology changes, populations change (e.g., become more sophisticated in their understanding of a communications channel), and the field continues to lag behind.

The novel designs and associated collaborations of this Center are motivated by the need for a new experimental paradigm that permits scientists to quickly and efficiently: (a) identify active psychosocial and communication intervention components, (b) investigate their interrelationships, (c) model for whom and when they impact prevention outcomes, and (d) select their settings judiciously to optimize the dissemination of the resulting treatment formulation. This paradigm must efficiently allocate resources among these multiple goals while maximizing power, given finite resources and timeframes.

A primary mission of this Center, therefore, is to develop, test, and disseminate a new paradigm for screening, refining, and optimizing health communication intervention components based on the use of sequential experimentation strategies and fractional factorial designs. Similar methods have been successfully applied in engineering, pharmacology, biochemistry, and other areas requiring the systematic development of products and materials from a large number of potentially active components [2-12]. This sequential strategy is an experimental approach to the iterative process of discovery -- screening, refining, and optimizing -- used in any scientific endeavor. Unlike classical experiments typified by randomized control trials in medicine and public health, in this new methodology, screening and refining are just as important as optimizing. The sequential experimentation strategies use scientific experience, results from previous experiments, and theory to improve and refine the treatment prior to conducting a classical optimization experiment.

This endeavor could not be undertaken without significant collaborations among: (a) statistical and design experts who have focused their research in other fields; (b) behavioral experts who have different theoretical perspectives on health behavior change and decision making; (c) information technology experts who have experience developing sophisticated health communication interventions; and (d) investigators who have a deep understanding of a relevant population setting (i.e., HMOs) for research and dissemination. Collaboration among these four domains is essential to the successful implementation of our Center. This collaboration is reflected in the strong, interconnected Cores of the Center.

The components of our Center should provide the basis for future studies, utilizing larger populations of subjects, in larger sequential experimental designs, using automated data collection methods. Reflecting directions being taken in the proprietary marketplace, these studies could be designed to model optimal treatment components and timing regimens that could be administered to millions at relatively low cost.

Simple Design
A simple way to determine the operational effect of a particular factor is to compare outcomes in a simple one-way layout that randomly assigns half the participants to one level of a factor and half to the other level. This design allows a valid measure of the causal effect of the factor, and of interactions between that factor and individual characteristics: if the factor is gain/loss framing, and gain framing is better for contemplators, and loss framing is better for precontemplators, then tailoring gain/loss framing according to stage is indicated. However, if there are 6 potential factors to manipulate in a particular treatment, 6 studies would be needed to study each of their individual effects; also, varying each factor individually does not provide any information about interactions between the factors themselves, which may be important in tailoring the treatment.

Factorial Design
To illustrate the advantage of factorial designs, consider the simple case of two two-level factors X(1) and X(2). For a fixed sample size n, compare (a) two experiments of size n/2 that manipulate one factor at a time, and (b) a single factorial experiment that assigns n/4 participants to each of the 4 treatment combinations. The factorial design is superior, since (1) it allows the main effects of each factor to be estimated from the full sample of size n, rather than a reduced sample of size n/2; and (2) it provides information about the interaction between the factors, which is completely absent in design (a).

A full factorial (or fully crossed) design investigates all possible combinations of factors, and it leads to an unworkable number of treatment combinations in our setting due to the developmental cost of communications, one per treatment combination. Although a full factorial design provides estimates of all main effects and interactions of all orders, in practice only a small proportion of these effects are usually important. For example, third and higher-order interactions are usually not important or of primary interest (at least in the initial stage).

Fractional Factorial Design
Fractional factorial designs are very useful for reducing the number of treatment combinations needed to study a large number of factors in these situations. As George Box writes (Box et al., 1978, page 375), "...there tends to be a redundancy in [full factorial designs] - redundancy in terms of an excess number of interactions that can be estimated and sometimes in an excess number of variables [components] that are studied. Fractional factorial designs exploit this redundancy."

In the screening phase we make working assumptions concerning the non-significance of selected higher order interactions. These working assumptions are based on scientific theory, clinical experience or past experimental evidence in similar situations. A fractional factorial design is a balanced subset of a full factorial design that exploits the working assumptions to estimate main effects and possibly significant interactions using a reduced number of cells. For example, consider a study with 4 factors, A, B, C, D, each with two levels (e.g. low and high). Following Box's suggestion, suppose we assume that all 3-factor interactions (ABC, ABD, etc.) are nonexistent or so small so that they can be ignored. Then we can use the full factorial design for 3 factors (with just 8 cells) to study 4 factors in a fractional factorial design! Subjects are randomized into 8 cells according to Table 2, with an equal number of subjects in each cell.

The first 3 columns of Table 2 show the levels of factors A, B and C in each cell, where "-" denotes "low" and "+" denotes "high." We set factor D levels according to the column ABC. That is, in cell 1, factor D will be set at the low (-) level, cell 2 at the high (+) level, and so on. The settings for factor D are the same as those for the 3-factor interaction ABC, obtained by multiplying the levels in the first three columns. In this case, we say the factors ABC and D are aliased. When we use this column to estimate an effect, we cannot tell whether the effect is due to D or ABC or both. Under our assumption that the ABC interaction is negligible, if we find a significant effect, we attribute this to the main effect of factor D.

Aliasing D with ABC induces aliasing of other interactions. The others can be found by the following mnemonic: beginning with D=ABC, multiply both sides by a letter and simply replace any letter that is squared by the identity. Thus, multiplying both sides of D=ABC by A, we get AD = A2BC = BC. Since D=ABC, we also have BAD=C, ADC=B, DBC =A, so each main effect is aliased with a three way interaction. We also have aliasing of second order interactions, AD=BC, AB=CD, etc. Thus the column in Table 2 labeled "BC=AD" measures the effect of BC, AD, or both.

Why are fractional factorial designs useful despite the aliasing of factors? The key is to select a design that capitalizes on knowledge (based on theory, clinical experience, or previous experiments) that some interactions are important and others are small. Suppose we are particularly interested in the AD interaction and are willing to make the working assumption that the BC interaction is small. Then, we can use the design in Table 2 that aliases the AD interaction with BC. If the estimated effect turns out to be large, then we know that it must be an AD interaction. If the effect is insignificant then we conclude that there is no AD interaction (along with no BC interaction). All is not lost even if our working assumption about BC is wrong. If we find a significant effect associated with this column, we can use follow-up experiments in Phase II (the refining phase) to untangle the two effects. A 2-cell experiment where, say, AD is at the same level in both cells but BC is at the low level in one and the high-level accomplishes this. Another concern is that if the interactions AD and BC are of opposite sign, the aliased AD=BC effect may be non-significant if the AD and BC cancel out. However, if we can predict the sign of the interaction a priori, then we can choose the fractional factorial design appropriately to avoid this cancellation.

There are many elegant properties associated with factorial and fractional factorial designs (Box et al., 1978). For example, when we average over the high and low levels of factor A to estimate the main effect, the effect of factors B, C, and the interactions cancel out. This implies that the main effect of factor A, say, is the difference in means between all the observations at the high level and all at the low level. So the power is similar to that of a single factor design - power is not sacrificed by increasing the number of factors. The beauty of the design in Table 2 is that it is a full factorial design for every subset of 3-factors. So regardless of which three factors are important, we are implicitly studying them using a full factorial design. The number of treatment combinations is reduced from 16 in a full factorial setting to 8, considerably reducing the development costs associated with creating the treatments.

Every fractional factorial design has a "resolution", depending on the nature of the aliasing relationships. The design in Table 2 is a resolution IV design, meaning that main effects are aliased with third and higher-order interactions, and 2-way interactions are aliased with other 2-way or higher-order interactions. For 2-way interactions to be aliased only with 3-way and higher-order interactions, we need a resolution V design, which requires more cells. Resolution III design require fewer cells, but have the undesirable property that main effects are aliased with 2-way interactions. Resolution IV designs provide a good compromise between number of cells and aliasing. The savings of fractional factorial designs over full factorial designs is even greater when there are more than 4 factors as in Table 2; one can choose the aliasing structure to estimate a large number of 2-way interactions. Tables of fractional factorial designs are available in the literature (Box et al., 1978; Wu and Hamada, 2000), and many software packages can be used to construct these designs.

It should be noted that the analysis of data from factorial and fractional factorial designs is more complicated when there is differential drop out across the cells, leading to unequal number of participants in the cells. Inference has to be based on weighted least squares and likelihood-based methods rather than the simpler least squares methods.

There has been some discussion of fractional factorial experiments in the prevention literature. For example, West et al. (1993), West and Aiken (1997) describe fractional factorial designs as a promising technique for investigating the effects of individual components in multi-component interventions. Challenges pointed out by West et al. (1993) include: (a) the large number of intervention combinations still needed in large fractional factorial designs, and (b) the potential for confounding main and 2-way effects by higher-order interactions. The first concern is addressed through the modular design of our web-based interventions, which can rapidly assemble multiple treatment combinations. Testing six psychosocial and communication factors through the development of 16 treatment component combinations in a fractional factorial design, however, is clearly a tremendous reduction in time and resources relative to the development of the 64 treatment combinations required in a full factorial design. Addressing the second point, West et al. (1993) state that: "...designs and analyses should be developed to the extent possible to be capable of probing the plausibility of the assumptions that have been made, particularly if they are compatible with other competing theoretical viewpoints (Coie et al., 1993)." This is precisely the purpose of our Phase II experiments. Moreover, we have spent a great deal of time in each Research Project to alias the most interesting and plausible interactions with the least plausible interactions.

The Projects in this proposal are all designed using the unified two-phase strategy discussed in Section C. In Phase I, we use screening experiments to explore the large-dimensional factor space to identify factors that have large effects and interactions with selected individual characteristics. Highly-fractionated factorial experiments are used to accomplish this economically with a manageable number of treatment combinations. The effects of factors that are identified from this phase are then studied more thoroughly in Phase II experiments. These include the use of higher-resolution designs with a small number of identified factors to resolve the aliasing structure among higher-order interactions, use of response surface designs with more than two levels to identify the optimal settings of continuously-scaled factors, and so on. If necessary, additional samples will be studied in selected cells to increase the power of detection. The sample sizes from the various stages will be combined appropriately to give adequate power in determining the active effects. These details are discussed in the context of the specific Research Projects.

The Phase I experiments in all three Projects investigate six treatment components, each at two levels, using 16-cell fractional factorial designs. These are resolution IV designs that use only one-quarter of the 26=64 possible treatment combinations (cells) in a full factorial design. In all three Projects, the designs were constructed to estimate all main effects and specified two-way interactions.

A primary goal of each Project is to determine if there are interactions between the treatments and selected individual characteristics. It is these interactions that allow for the possibility of tailoring. We will collect data and analyze interactions associated with several interesting characteristics. Moreover, selected characteristics have been block randomized in each experiment so that there is sufficient power to detect these key interactions. These variables are: participant's stage at the beginning of study (pre-contemplation, contemplation, or preparation) in Project 1, ethnic identity and motivational disposition in Project 2, and self-reported numeracy in Project 3. The results from the Phase I studies will be used to develop appropriate Phase II experiments and investigate the hypotheses and questions of interest more fully. These are discussed in detail in the individual Project descriptions.

There is a world of difference between data and information. To extract information from data you have to make assumptions about the system that generated the data. Using these assumptions and physical theory you may be able to develop a mathematical model of the system.

Generally, even rigorously formulated models have some unknown constants. The goal of experimentation is to acquire data that enable you to estimate these constants.

But why do you need to experiment at all? You could instrument the system you want to study and just let it run. Eventually you would have all the data you could use.

In fact, this is a fairly common approach. There are three characteristics of historical data that pose problems for statistical modeling:

• Suppose you observe a change in the operating variables of a system followed by a change in the outputs of the system. That does not necessarily mean that the change in the system caused the change in the outputs.
• A common assumption in statistical modeling is that the observations are independent of each other. This is not the way a system in normal operation works.
• Controlling a system in operation often means changing system variables in tandem. But if two variables change together, it is impossible to separate their effects mathematically.

Designed experiments directly address these problems. The overwhelming advantage of a designed experiment is that you actively manipulate the system you are studying. With Design of Experiments (DOE) you may generate fewer data points than by using passive instrumentation, but the quality of the information you get will be higher.

The Statistics Toolbox provides several functions for generating experimental designs appropriate to various situations. These are discussed in the following sections:

Experimental design methods offer valuable insights into the improvement and understanding of the response from a system and are considered very general and versatile [4,7-10]. They can be employed for both optimization and for comparing treatment effects. A treatment is a stimulus or perturbation applied to one or more of the variables of a system in order to induce a change in the response. Each experimental run in an experimental design study requires that one or more treatments be applied to the system and the response be measured. Statistical analysis methods are then employed to determine whether a particular treatment or combination of treatments was statistically significant in influencing the system response. Calculation of the treatment effects can be used to identify which variables lead to an optimal response. In experimental design methods, the values or settings for each variable are called levels. The number of levels for each variable as well as the settings at each level are chosen prior to the study.

text below is from: (http://ei.cs.vt.edu/~cs5014/fall.95/courseNotes/WebPages/3.ExperimentDesign/ch16/abrams445.html) More stuff on site! Revisit.

Simple Design
Step 1: Choose a baseline level for each factor. Measure performance.
Step 2: Vary the first factor, measuring performance at each level.

Example: For workload design, vary the CPU type first to see which CPU type is best.

Step 3: Repeat step 2 for each level.

Disadvantages:

• Design may give false conclusions about factor interaction
• Alternate design can give more information with same number of experiments

Factorial
Exhaustively try every possible combination of all levels of all factors.

Advantage:

• We can find effect of every factor (including secondary factors) and their interactions.

Disadvantage:

• Cost of study: Infeasible if number of factors or levels is large.

When an experimenter is interested in the effects of two or more independent variables, it is usually more efficient to manipulate these variables in one experiment than to run a separate experiment for each variable. Moreover, only in experiments with more than one independent variable is it possible to test for interactions among variables.

Suppose you want to determine whether the variability of a machining process is due to the difference in the lathes that cut the parts or the operators who run the lathes.

If the same operator always runs a given lathe then you cannot tell whether the machine or the operator is the cause of the variation in the output. By allowing every operator to run every lathe you can separate their effects.

This is a factorial approach. fullfact is the function that generates the design. Suppose you have four operators and three machines. What is the factorial design?

Each row of d represents one operator/machine combination. Note that there are 4*3 = 12 rows.

Fractional Factorial
Advantage: Fewer experiments than full factorial design.

Disadvantage: Indicates interactions among some but not all factors. (O.K. if you know a priori that certain interactions are negligible.)

Text below from: http://chemdiv-www.nrl.navy.mil/6110/6112/chemometrics/optexpd.html

A simple, commonly used experimental design method is the two-level factorial design. For systems with a small number of variables, this approach is very powerful. Using only a small number of experiments, this approach can obtain maximum information by measuring the influence of several variables simultaneously. In this type of design, the treatments consist of all possible combinations of two levels of each of the experimental variables. For example, a two-level, three-variable, full factorial design employs 23= 8 experiments. Although these designs only explore a small region of the response surface, they can determine a promising direction for further exploration. In that sense, they can be used for optimization. However, their most powerful application is for understanding how the treatments affect the response of the system. This is performed by determining the main and interaction effects of the treatments. Main effects measure the influence that a particular variable has on the response. Interaction effects measure the influence that a particular combination of variables has on the response. For example, if three variables (i.e., a treatment combination) jointly influence the response, this would be a three-way interaction effect. A treatment is judged statistically significant if the variation in the response caused by changing the variable setting (or combination of variable settings) is larger than the experimental error in the measurement of the response. Several statistical methods have been employed for the determination of the main and interaction effects including Yate's algorithm and analysis of variance (ANOVA). Discussion of Yate's algorithm is beyond the scope of this review, but is fully described in several textbooks on experimental design methodology [7,9]. A brief description of the working principles of ANOVA can also be found in references 2-5.

Fractional factorial designs, a subset of factorial designs, are very useful when each experiment is expensive or time-consuming. In factorial designs, the number of experiments increases exponentially as more variables are added to the design. Fractional factorial designs reduce the overall number of experiments that need to be performed by reducing redundancy. In most experimental situations, the higher-order interaction effects are negligible and can be disregarded. Fractional factorial designs exploit this redundancy. In the most commonly employed fractional design, the half-fraction design, exactly one half of the experiments of a full design are performed. For example, for a half-fractional, two-level, five-factor design, 2 5-1 = 16 experiments are performed. In this design, however, the calculation of the three-way and higher order interaction effects is not possible due to insufficient degrees of freedom. The statistical analysis of fractional factorial designs can again be performed through ANOVA.

text below is from: (http://www.statsoftinc.com/textbook/stexdes.html#2)
In many cases, it is sufficient to consider the factors affecting the production process at two levels. For example, the temperature for a chemical process may either be set a little higher or a little lower, the amount of solvent in a dyestuff manufacturing process can either be slightly increased or decreased, etc. The experimenter would like to determine whether any of these changes affect the results of the production process. The most intuitive approach to study those factors would be to vary the factors of interest in a full factorial design, that is, to try all possible combinations of settings. This would work fine, except that the number of necessary runs in the experiment (observations) will increase geometrically. For example, if you want to study 7 factors, the necessary number of runs in the experiment would be 2**7 = 128. To study 10 factors you would need 2**10 = 1,024 runs in the experiment. Because each run may require time-consuming and costly setting and resetting of machinery, it is often not feasible to require that many different production runs for the experiment. In these conditions, fractional factorials are used that "sacrifice" interaction effects so that main effects may still be computed correctly.

the text below is from: (http://www.itl.nist.gov/div898/handbook/pri/section3/pri334.htm)

The ASQC (1983) Glossary & Tables for Statistical Quality Control defines fractional factorial design in the following way: "A factorial experiment in which only an adequately chosen fraction of the treatment combinations required for the complete factorial experiment is selected to be run."

Even if the number of factors, k, in a design is small, the 2k runs specified for a full factorial can quickly become very large. For example, 26 = 64 runs is for a two-level, full factorial design with six factors. To this design we need to add a good number of centerpoint runs and we can thus quickly run up a very large resource requirement for runs with only a modest number of factors.

The solution to this problem is to use only a fraction of the runs specified by the full factorial design. Which runs to make and which to leave out is the subject of interest here. In general, we pick a fraction such as �, �, etc. of the runs called for by the full factorial. We use various strategies that ensure an appropriate choice of runs.

from: (http://www.nmt.edu/~design/mate482lect/FractionalFactorialDesign.html)

As k increases the number of experiments required for 2k grows rapidly and exceeds costs and time resources of the engineer. For example a 26 requires 64 runs or experiments assuming no replications, but if we look more closely only 6 of the 63d.f. correspond to the main effects, 15 are two factor, and 42 are 3 or more factor interactions.

If you, the experimenter can assume that higher order interactions are negligible then you only need to run a factor of the experiments. This technique is often used for product and process design as well as process trouble shooting. Based on result of screening experiments, input factors can be identified then investigated more thoroughly in subsequent experiments.

Three factors for successful fractional factorial design:

1. Scarcity Effects Principle- When there are several variables, the process is likely driven purely by some of the main effects and low order interactions.
2. Projective Property-Fractal designs can be projected into larger designs by the significant factors.
3. Sequential Experiments-Runs of 2 or more fractal factors can be assembled into a larger design to estimate further effects and interactions.

text below from: (http://www.mathworks.com/access/helpdesk/help/toolbox/stats/doe4.shtml)

One difficulty with factorial designs is that the number of combinations increases exponentially with the number of variables you want to manipulate.

For example, the sensitivity study discussed above might be impractical if there were seven variables to study instead of just three. A full factorial design would require 27 = 128 runs!

If you assume that the variables do not act synergistically in the system, you can assess the sensitivity with far fewer runs.

A drawback of this design is that if the effect of one variable does vary with the value of another variable, then the estimated effects will be biased (that is, they will tend to be off by a systematic amount).

At a cost of a somewhat larger design, you can find a fractional factorial that is much smaller than a full factorial, but that does allow estimation of main effects independent of interactions between pairs of variables. You can do this by specifying generators that control the confounding between variables.

As an example, suppose you create a design with the first four variables varying independently as in a full factorial, but with the other three variables formed by multiplying different triplets of the first four. With this design the effects of the last three variables are confounded with three-way interactions among the first four variables. The estimated effect of any single variable, however, is not confounded with (is independent of) interaction effects between any pair of variables. Interaction effects are confounded with each other.