5. Process Improvement
5.4. Analysis of DOE data
5.4.7. Examples of DOE's

5.4.7.2.

Fractional factorial example

A "Catapult" Fractional Factorial Experiment

A step-by-step analysis of a fractional factorial "catapult" experiment

This experiment was conducted by a team of students on a catapult, a table-top wooden device used to teach design of experiments and statistical process control. The catapult has several controllable factors and a response easily measured in a classroom setting. It has been used for over 10 years in hundreds of classes.

Catapult

Description of Experiment: Response and Factors

The experiment has five factors that might affect the distance the golf ball travels

Purpose: To determine the significant factors that affect the distance the ball is thrown by the catapult, and to determine the settings required to reach three different distances (30, 60 and 90 inches).

Response Variable: The distance in inches from the front of the catapult to the spot where the ball lands. The ball is a plastic golf ball.

Number of observations: 20 (a 2^5-1 resolution V design with 4 center points).

Variables:

1. Response Variable Y = distance

2. Factor 1 = band height (height of the pivot point for the rubber bands, levels were 2.25 and 4.75 inches with a centerpoint level of 3.5)

3. Factor 2 = start angle (location of the arm when the operator releases, starts the forward motion of the arm, levels were 0 and 20 degrees with a centerpoint level of 10 degrees)

4. Factor 3 = rubber bands (number of rubber bands used on the catapult, levels were 1 and 2 bands)

5. Factor 4 = arm length (distance the arm is extended, levels were 0 and 4 inches with a centerpoint level of 2 inches)

6. Factor 5 = stop angle (location of the arm where the forward motion of the arm is stopped and the ball starts flying, levels were 45 and 80 degrees with a centerpoint level of 62 degrees)

Design matrix and responses (in run order)

The design matrix appears below in (randomized) run order. (The reader can download the data as a text file.)

distance height start bands length stop order
   28.00   3.25     0    1      0   80     1
   99.00      4    10    2      2   62     2
  126.50   4.75    20    2      4   80     3
  126.50   4.75     0    2      4   45     4
   45.00   3.25    20    2      4   45     5
   35.00   4.75     0    1      0   45     6
   45.00      4    10    1      2   62     7
   28.25   4.75    20    1      0   80     8
   85.00   4.75     0    1      4   80     9
    8.00   3.25    20    1      0   45    10
   36.50   4.75    20    1      4   45    11
   33.00   3.25     0    1      4   45    12
   84.50      4    10    2      2   62    13
   28.50   4.75    20    2      0   45    14
   33.50   3.25     0    2      0   45    15
   36.00   3.25    20    2      0   80    16
   84.00   4.75     0    2      0   80    17
   45.00   3.25    20    1      4   80    18
   37.50      4    10    1      2   62    19
  106.00   3.25     0    2      4   80    20

One discrete factor

Note that four of the factors are continuous, and one, number of rubber bands, is discrete. Due to the presence of this discrete factor, we actually have two different centerpoints, each with two runs. Runs 7 and 19 are with one rubber band, and the center of the other factors, while runs 2 and 13 are with two rubber bands and the center of the other factors.

Five confirmatory runs

After analyzing the 20 runs and determining factor settings needed to achieve predicted distances of 30, 60 and 90 inches, the team was asked to conduct five confirmatory runs at each of the derived settings.

Analysis of the Experiment

The analyses shown in this page can be generated using R code.

Step 1: Look at the data

Histogram, box plot, normal probability plot, and run order plot of the response

We start by plotting the data several ways to see if any trends or anomalies appear that would not be accounted for by the models.

We can see the large spread of the data and a pattern to the data that should be explained by the analysis. The run order plot does not indicate an obvious time sequence. The four highlighted points in the run order plot are the center points in the design. Recall that runs 2 and 13 had two rubber bands and runs 7 and 19 had one rubber band. There may be a slight aging of the rubber bands in that the second center point resulted in a distance that was a little shorter than the first for each pair.

Plots of responses versus factor columns

Next look at the plots of responses sorted by factor columns.

Several factors appear to change the average response level and most have large spread at each of the levels.

Step 2: Create the theoretical model

The resolution V design can estimate main effects and all two-factor interactions

With a resolution V design we are able to estimate all the main effects and all two-factor interactions without worrying about confounding. Therefore, the initial model will have 16 terms: the intercept term, the 5 main effects, and the 10 two-factor interactions.

Step 3: Create the actual model from the data

Variable coding

Note we have used the orthogonally coded columns for the analysis, and have abbreviated the factor names as follows:

Height (h) = band height
Start (s) = start angle
Bands (b) = number of rubber bands
Stop (e) = stop angle
Length (l) = arm length.

Trial model with all main factors and two-factor interactions

The results of fitting the trial model that includes all main factors and two-factor interactions follow.

Source      Estimate  Std. Error  t value  Pr(>|t|) 
---------   --------  ----------  -------  --------
Intercept    57.5375      2.9691   19.378  4.18e-05 ***
h            13.4844      3.3196    4.062   0.01532 *  
s           -11.0781      3.3196   -3.337   0.02891 *  
b            19.4125      2.9691    6.538   0.00283 ** 
l            20.1406      3.3196    6.067   0.00373 ** 
e            12.0469      3.3196    3.629   0.02218 *  
h*s          -2.7656      3.3196   -0.833   0.45163    
h*b           4.6406      3.3196    1.398   0.23467    
h*l           4.7031      3.3196    1.417   0.22950    
h*e           0.1094      3.3196    0.033   0.97529    
s*b          -3.1719      3.3196   -0.955   0.39343    
s*l          -1.1094      3.3196   -0.334   0.75502    
s*e           2.6719      3.3196    0.805   0.46601    
b*l           7.6094      3.3196    2.292   0.08365 .  
b*e           2.8281      3.3196    0.852   0.44225    
l*e           3.1406      3.3196    0.946   0.39768    

Significance codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

Residual standard error:  13.28, based on 4 degrees of freedom
Multiple R-squared:  0.9709     
Adjusted R-squared:  0.8619 
F-statistic:  8.905, based on 15 and 4 degrees of freedom
p-value: 0.02375 

Use p-values and a normal probability plot to help select significant effects

The model has a good R² value, but the fact that R² adjusted is considerably smaller indicates that we undoubtedly have some terms in our model that are not significant. Scanning the column of p-values (labeled Pr(>|t|)) for small values shows five significant effects at the 0.05 level and another one at the 0.10 level.

A normal probability plot of effects is a useful graphical tool to determine significant effects. The graph below shows that there are nine terms in the model that can be assumed to be noise. That would leave six terms to be included in the model. Whereas the output above shows a p-value of 0.0836 for the interaction of Bands (b) and Length (l), the normal plot suggests we treat this interaction as significant.

Refit using just the effects that appear to matter

Remove the non-significant terms from the model and refit to produce the following analysis of variance table.

Source           Df Sum of Sq  Mean Sq   F value   Pr(>F)
-----------      -- ---------  -------   -------   ------
Model             6  22148.55   3691.6
Total error      13   2106.99    162.1     22.77  3.5e-06

Lack-of-fit      11   1973.74    179.4
Pure error        2    133.25     66.6      2.69   0.3018

Residual standard error:  12.73 based on 13 degrees of freedom
Multiple R-squared:  0.9131
Adjusted R-squared:  0.873 
p-value:  

R² is OK and there is no significant model "lack of fit"

The R² and R² adjusted values are acceptable. The ANOVA table shows us that the model is significant, and the lack-of-fit test is not significant. Parameter estimates are below.

Source      Estimate  Std. Error  t value  Pr(>|t|)  
---------   --------  ----------  -------  --------  
Intercept     57.537       2.847   20.212  3.33e-11 ***
h             13.484       3.183    4.237   0.00097 ***
s            -11.078       3.183   -3.481   0.00406 ** 
b             19.412       2.847    6.819  1.23e-05 ***
l             20.141       3.183    6.328  2.62e-05 ***
e             12.047       3.183    3.785   0.00227 ** 
b*l            7.609       3.183    2.391   0.03264 *  

Significance codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

Step 4: Test the model assumptions using residual graphs (adjust and simplify as needed)

Diagnostic residual plots

To examine the assumption that the residuals are approximately normally distributed, are independent, and have equal variances, we generate four plots of the residuals: a normal probability plot, box plot, histogram, and a run-order plot of the residuals. In the run-order plot, the highlighted points are the centerpoint values. Recall that run numbers 2 and 13 had two rubber bands while run numbers 7 and 19 had only one rubber band.

The residuals do appear to have, at least approximately, a normal distributed.

Plot of residuals versus predicted values

Next we plot the residuals versus the predicted values.

There does not appear to be a pattern to the residuals. One observation about the graph, from a single point, is that the model performs poorly in predicting a short distance. In fact, run number 10 had a measured distance of 8 inches, but the model predicts -11 inches, giving a residual of 19 inches. The fact that the model predicts an impossible negative distance is an obvious shortcoming of the model. We may not be successful at predicting the catapult settings required to hit a distance less than 25 inches. This is not surprising since there is only one data value less than 28 inches. Recall that the objective is to achieve distances of 30, 60, and 90 inches.

Plots of residuals versus the factor variables

Next we look at the residual values versus each of the factors.

The residual graphs are not ideal, although the model passes the lack-of-fit test

Most of the residual graphs versus the factors appear to have a slight "frown" on the graph (higher residuals in the center). This may indicate a lack of fit, or sign of curvature at the centerpoint values. The lack- of-fit test, however, indicates that the lack of fit is not significant.

Consider a transformation of the response variable to see if we can obtain a better model

At this point, since there are several unsatisfactory features of the model we have fit and the resultant residuals, we should consider whether a simple transformation of the response variable (Y = "Distance") might improve the situation.

There are at least two good reasons to suspect that using the logarithm of distance as the response might lead to a better model.

1. A linear model fit to ln(Y) will always predict a positive distance when converted back to the original scale for any possible combination of X factor values.
2. Physical considerations suggest that a realistic model for distance might require quadratic terms since gravity plays a key role - taking logarithms often reduces the impact of non-linear terms.

To see whether using ln(Y) as the response leads to a more satisfactory model, we return to step 3.

Step 3a: Fit the full model using ln(Y) as the response

First a main effects and two-factor interaction model is fit to the log distance responses

Proceeding as before, using the coded values of the factor levels and the natural logarithm of distance as the response, we obtain the following parameter estimates.

Source      Estimate  Std. Error  t value  Pr(>|t|) 
---------   --------  ----------  -------  --------   
(Intercept)  3.85702     0.06865   56.186  6.01e-07 ***
h            0.25735     0.07675    3.353   0.02849 *  
s           -0.24174     0.07675   -3.150   0.03452 *  
b            0.34880     0.06865    5.081   0.00708 ** 
l            0.39437     0.07675    5.138   0.00680 ** 
e            0.26273     0.07675    3.423   0.02670 *  
h*s         -0.02582     0.07675   -0.336   0.75348    
h*b         -0.02035     0.07675   -0.265   0.80403    
h*l         -0.01396     0.07675   -0.182   0.86457    
h*e         -0.04873     0.07675   -0.635   0.55999    
s*b          0.00853     0.07675    0.111   0.91686    
s*l          0.06775     0.07675    0.883   0.42724    
s*e          0.07955     0.07675    1.036   0.35855    
b*l          0.01499     0.07675    0.195   0.85472    
b*e         -0.01152     0.07675   -0.150   0.88794    
l*e         -0.01120     0.07675   -0.146   0.89108    

Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

Residual standard error:  0.307 based on 4 degrees of freedom
Multiple R-squared:  0.9564
Adjusted R-squared:  0.7927 
F-statistic:  5.845 based on 15 and 4 degrees of freedom
p-value:  0.0502

A simpler model with just main effects has a satisfactory fit

Examining the p-values of the 16 model coefficients, only the intercept and the 5 main effect terms appear significant. Refitting the model with just these terms yields the following results.

Source           Df Sum of Sq  Mean Sq   F value    Pr(>F)
-----------      -- ---------  -------   -------    ------
Model             5   8.02079   1.60416   36.285   1.6e-07
Total error      14   0.61896   0.04421

Lack-of-fit      12   0.58980   0.04915
Pure error        2   0.02916   0.01458    3.371    0.2514


Source      Estimate  Std. Error  t value  Pr(>|t|) 
---------   --------  ----------  -------  --------   
Intercept    3.85702     0.04702   82.035   < 2e-16 ***
h            0.25735     0.05257    4.896  0.000236 ***
s           -0.24174     0.05257   -4.599  0.000413 ***
b            0.34880     0.04702    7.419  3.26e-06 ***
l            0.39437     0.05257    7.502  2.87e-06 ***
e            0.26273     0.05257    4.998  0.000195 ***

Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

Residual standard error:  0.2103 based on 14 degrees of freedom
Multiple R-squared:  0.9284
Adjusted R-squared:  0.9028 

This is a simpler model than previously obtained in Step 3 (no interaction term). All the terms are highly significant and there is no indication of a significant lack of fit.

We next look at the residuals for this new model fit.

Step 4a: Test the (new) model assumptions using residual graphs (adjust and simplify as needed)

Normal probability plot, box plot, histogram, and run-order plot of the residuals

The following normal plot, box plot, histogram and run-order plot of the residuals shows no problems.

Residuals plotted versus run order again show a possible slight decreasing trend (rubber band fatigue?).

Plot of residuals versus predicted ln(Y) values

A plot of the residuals versus the predicted ln(Y) values looks reasonable, although there might be a tendency for the model to overestimate slightly for high predicted values.

Plot of residuals versus the factor variables

Next we look at the residual values versus each of the factors.

The residuals for the main effects model (fit to natural log of distance) are reasonably well behaved

These plots still appear to have a slight "frown" on the graph (higher residuals in the center). However, the model is generally an improvement over the previous model and will be accepted as possibly the best that can be done without conducting a new experiment designed to fit a quadratic model.

Step 5: Use the results to answer the questions in your experimental objectives

Final step: Predict the settings that should be used to obtain desired distances

Based on the analyses and plots, we can select factor settings that maximize the log-transaformed distance. Translating from "-1", "0", and "+1" back to the actual factor settings, we have: band height at "0" or 3.5 inches; start angle at "0" or 10 degrees; number of rubber bands at "1" or 2 bands; arm length at "1" or 4 inches , and stop angle at "0" or 80 degrees.

"Confirmation" runs were successful

In the confirmatory runs that followed the experiment, the team was successful at hitting all three targets, but did not hit them all five times. The model discovery and fitting process, as illustrated in this analysis, is often an iterative process.