4. Process Modeling
4.3. Data Collection for Process Modeling

4.3.5.

How can I tell if a particular experimental design is good for my application?

Assess Relative to the Six Design Principles

If you have a design, generated by whatever method, in hand, how can you assess its after-the-fact goodness? Such checks can potentially parallel the list of the six general design principles. The design can be assessed relative to each of these six principles. For example, does it have capacity for the primary model, does it have capacity for an alternative model, etc.

Some of these checks are quantitative and complicated; other checks are simpler and graphical. The graphical checks are the most easily done and yet are among the most informative. We include two such graphical checks and one quantitative check.

Graphically Check for Univariate Balance

If you have a design that claims to be globally good in \(k\) factors, then generally that design should be locally good in each of the individual \(k\) factors. Checking high-dimensional global goodness is difficult, but checking low-dimensional local goodness is easy. Generate \(k\) counts plots, with the levels of factors \(x_i\) plotted on the horizontal axis of each plot and the number of design points for each level in factor \(x_i\) on the vertical axis. For most good designs, these counts should be about the same (equal balance) for all levels of a factor. Exceptions exist, but such balance is a low-level characteristic of most good designs.

Graphically Check for Bivariate Balance

If you have a design that is purported to be globally good in \(k\) factors, then generally that design should be locally good in all pairs of the individual \(k\) factors. Graphically check for such 2-way balance by generating plots for all pairs of factors, where the horizontal axis of a given plot is \(x_i\) and the vertical axis is \(x_j\). The response variable \(y\) does NOT come into play in these plots. We are only interested in characteristics of the design, and so only the \(x\) variables are involved. The 2-way plots of most good designs have a certain symmetric and balanced look about them--all combination points should be covered and each combination point should have about the same number of points.

Check for Minimal Variation

For optimal designs, metrics exist (D-efficiency, A-efficiency, etc.) that can be computed and that reflect the quality of the design. Further, relative ratios of standard deviations of the coefficient estimators and relative ratios of predicted values can be computed and compared for such designs. Such calculations are commonly performed in computer packages which specialize in the generation of optimal designs.