2. Measurement Process Characterization
2.3. Calibration
2.3.3. What are calibration designs?
2.3.3.3. Uncertainties of calibrated values

2.3.3.3.2.

Repeatability and level-2 standard deviations

Repeatability standard deviation comes from the data of a single design

The repeatability standard deviation of the instrument can be computed in two ways.

1. It can be computed as the residual standard deviation from the design and should be available as output from any software package that reduces data from calibration designs. The matrix equations for this computation are shown in the section on solutions to calibration designs. The standard deviation has degrees of freedom $$ \nu = n - m + 1 $$ for n difference measurements and m items. Typically the degrees of freedom are very small. For two differences measurements on a reference standard and test item, the degrees of freedom is \( \nu = 1 \).

A more reliable estimate comes from pooling over historical data

2. A more reliable estimate of the standard deviation can be computed by pooling variances from K calibrations (and then taking its square root) using the same instrument (assuming the instrument is in statistical control). The formula for the pooled estimate is $$ {\large s}_1 = \sqrt{\frac{1}{\sum_{k} \nu_k}\sum_{k} \nu_k {\large s}_k^2} \,\, . $$

Level-2 standard deviation is estimated from check standard measurements

The level-2 standard deviation cannot be estimated from the data of the calibration design. It cannot generally be estimated from repeated designs involving the test items. The best mechanism for capturing the day-to-day effects is a check standard, which is treated as a test item and included in each calibration design. Values of the check standard, estimated over time from the calibration design, are used to estimate the standard deviation.

Assumptions

The check standard value must be stable over time, and the measurements must be in statistical control for this procedure to be valid. For this purpose, it is necessary to keep a historical record of values for a given check standard, and these values should be kept by instrument and by design.

Computation of level-2 standard deviation

Given K historical check standard values, $$ C_1, \, C_2, \, \ldots , \, C_K $$

the standard deviation of the check standard values is computed as $$ {\large s}_C = {\large s}_2 = \sqrt{\frac{1}{K-1} \sum_{k=1}^{K} \left( C_k - \overline{C}_{\small{\bullet}} \right)^2} $$ where $$ \overline{C}_{\small{\bullet}} = \frac{1}{K} \sum_{k=1}^{K} C_k $$ with degrees of freedom \( \nu = K - 1 \).