MEDIAN CONFIDENCE LIMITS

MEDIAN CONFIDENCE LIMITS

Analysis Command

Generates a median based confidence interval for the location of a variable.

Mosteller and Tukey (see Reference section below) define two types of robustness:
1. resistance means that changing a small part, even by a large amount, of the data does not cause a large change in the estimate

2. robustness of efficiency means that the statistic has high efficiency in a variety of situations rather than in any one situation. Efficiency means that the estimate is close to optimal estimate given that we distribution that the data comes from. A useful measure of efficiency is:
   Efficiency = (lowest variance feasible)/ (actual variance)

Standard confidence intervals are based on the mean and variance. These are the optimal estimators if the data are in fact from a Gaussian population. However, the mean lacks both resistance and robustness of efficiency. The median is less affected by outliers (i.e., resistance) than the mean. However, the median is not particularly robust with regards to efficiency.

Dataplot generates confidence intervals for the median using the following two methods:

1. Method 1 is the Hettmansperger-Sheather interpolation method. The steps in this method are:
   • Suppose W is a binomial random variable with n trials and a probability of success p = 0.5. For any integer, k, between 0 and [n/2], let \( \gamma_{k} = P(k \le W \le n - k) \).

     A 95% confidence interval for the median is:

     (X_k, X_n-k+1)

     with X denoting the sorted observations.

   • Determine k such that \( \gamma_{k+1} < 1 - \alpha < \gamma_{k} \).

   • Compute
     \( I = \frac{\gamma_k -1 - \alpha}{\gamma_k - \gamma_{k+1}} \)

     and

     \( \lambda = \frac{(n-k)I}{k + (n - 2k)I} \)

   • An approximate (1-\( \alpha \)) confidence interval is
     \( LCL = \lambda X_{k+1} + (1 - \lambda) X_k \)
     \( UCL = \lambda X_{n-k} + (1 - \lambda) X_{n-k+1} \)

2. Method 2, based on a method given by Wilcox (see Reference below) on page 87, is based on the Maritz-Jarrett estimate of the standard error for a quantile. Specifically,
   \( \hat{x}_{q} \pm \Phi^{-1}(1-\alpha/2)\hat{\sigma}_{mj} \)

   where

   q = the desired quantile (q = 0.5 for the median)

   \( \hat{x} \) = the estimated sample quantile

   \( \Phi^{-1} \) = the percent point function of the standard normal distribution

   \( \alpha \) = the significance level

   \( \hat{\sigma}_{mj} \) = the quantile standard error based on the Maritz-Jarrett method

   Note that this method can be applied to quantiles other than the median. However, the accuracy of this method has not been studied for quantiles other than 0.5.

MEDIAN CONFIDENCE LIMITS <y>       <SUBSET/EXCEPT/FOR qualification>
where <y> is the response variable,
and where the <SUBSET/EXCEPT/FOR qualification> is optional.

QUANTILE CONFIDENCE LIMITS <y>
                        <SUBSET/EXCEPT/FOR qualification>
where <y> is the response variable, and where the <SUBSET/EXCEPT/FOR qualification> is optional.

MEDIAN CONFIDENCE LIMITS Y1


LET P100 = 0.25
QUANTILE CONFIDENCE LIMITS Y1 SUBSET TAG = 2

For quantiles other than the median, the desired quantile is specified with the LET command. Specifically, define the parameter P100. For example,
LET P100 = 0.25

Only the method based on the Maritz-Jarrett standard error is supported for quantiles other than the median.

A table of confidence intervals is printed for alpha levels of 50.0, 75.0, 90.0, 95.0, 99.0, 99.9, 99.99, and 99.999.

Alternative methods for generating confidence intervals for medians or quantiles are available.
1. You can use the BOOTSTRAP MEDIAN PLOT or the BOOTSTRAP QUANTILE PLOT command. For example,

   LET Y = CAUCHY RANDOM NUMBERS FOR I = 1 1 100
   BOOTSTRAP MEDIAN PLOT Y
   LET LCL = B025
   LET UCL = B975
   .
   LET XQ = 0.75
   BOOTSTRAP QUANTILE PLOT Y
   LET LCL = B025
   LET UCL = B975
              

2. Wilcox suggests the following method for quantiles (the median is a special case with the quantile = 0.5) on page 86.
   \( \hat{\theta} = \pm \; \hat{c} \; \hat{\sigma}_{hd} \)

   where

   \( \hat{\theta} \) = the Herrell-Davis quantile estimate

   \( \hat{\sigma}_{hd} \) = the bootstrap estimate of the Herrell-Davis quantile standard error

   \( \hat{c} \) = 0.5064*N**(-0.25) + 1.96 for N >= 11 and 0.3 <= q <= 0.7
         for N > 21 and 0.2 <= q <= 0.8
         for N > 41 0.1 ≤ q ≤ 0.9
      = -6.23*(1/N) + 5.01 for 11 ≤ N ≤ 21, q = 0.2, 0.8
      = 36.2*(1/N) + 1.31 for N > 41, q = 0.1, 0.9

   This can be coded in the following Dataplot macro:

   SET QUANTILE METHOD HERRELL DAVIS
   LET P100 = 0.5
   LET THETAHAT = QUANTILE Y
   BOOTSTRAP QUANTILE STANDARD ERROR PLOT Y
   LET SIGMAHAT = B50
   LET N = SIZE Y
   IF N < 11
     QUIT
   END OF IF
   LET C = 0.5064*N**(-0.25) + 1.96
   LET IQFLAG = 1
   IF P100 <= 0.19
     IF N > 41
       LET C = 36.2*(1/N) + 1.31
     END OF IF
   ELSEIF P100 <= 0.29
     IF N <= 21
       LET C = -6.23*(1/N) + 5.01
     END OF IF
   ELSE IF P100 >= 0.81
     IF N > 41
       LET C = 36.2*(1/N) + 1.31
     END OF IF
   ELSE IF P100 >= 0.71
     IF N <= 21
       LET C = -6.23*(1/N) + 5.01
     END OF IF
   ENDIF
   LET LOWLIMIT = THETAHAT - C*B50
   LET UPPLIMIT = THETAHAT + C*B50
              

None

MEDIAN CONFIDENCE INTERVAL

MEDIAN	= Compute the median.
MEDIAN PLOT	= Generate a median (versus subset) plot.
BOOTSTRAP PLOT	= Generate a bootstrap plot.
CONFIDENCE LIMITS	= Compute a Gaussian based confidence limit.
BIWEIGHT CONFIDENCE LIMITS	= Compute a biweight location based confidence limit.
TRIMMED MEAN CONFIDENCE LIMITS	= Compute a trimmed mean based confidence limit.
T-TEST	= Perform a t-test.

Rand Wilcox (1997), "Introduction to Robust Estimation and Hypothesis Testing," Academic Press, p. 87.

T. P. Hettmansperger and S. J. Sheather (1986), "Confidence Interval Based on Interpolated Order Statistics," Statistical Probability Letters 4, pp. 75-79.

Robust Data Analysis

2003/2

 
LET Y1 = NORMAL RANDOM NUMBERS FOR I = 1 1 100
LET Y2 = LOGISTIC RANDOM NUMBERS FOR I = 1 1 100
LET Y3 = CAUCHY RANDOM NUMBERS FOR I = 1 1 100
LET Y4 = DOUBLE EXPONENTIAL RANDOM NUMBERS FOR I = 1 1 100
SET WRITE DECIMALS 4
MEDIAN CONFIDENCE LIMITS Y1 TO Y4
    

Dataplot generates the following output:

             Confidence Limits for the Median
             (Based on Maritz-Jarrett Standard Error for Quantiles)
  
 Response Variable: Y1
  
 Summary Statistics:
 Number of Observations:                  100
 Sample Minimum:                          -3.4580
 Sample Maximum:                          2.0059
 Sample Median:                           0.0015
 Sample Quantile Standard Error:          0.1119
  
  
  
 -----------------------------------------------------------------
   Confidence       Z      Z-Value X          Lower          Upper
    Value (%)   Value         StdErr          Limit          Limit
 -----------------------------------------------------------------
       50.000   0.674         0.0755        -0.0739         0.0770
       75.000   1.150         0.1287        -0.1272         0.1302
       90.000   1.645         0.1840        -0.1825         0.1856
       95.000   1.960         0.2193        -0.2177         0.2208
       99.000   2.576         0.2882        -0.2866         0.2897
       99.900   3.291         0.3682        -0.3666         0.3697
       99.990   3.891         0.4353        -0.4337         0.4368
       99.999   4.417         0.4942        -0.4927         0.4957
  
  
             Hettmansperger-Sheater Median Confidence Limits
  
 -----------------------------------
   Confidence   Lower          Upper
    Value (%)   Limit          Limit
 -----------------------------------
       50.000  -0.064         0.0410
       75.000  -0.102         0.1053
       90.000  -0.132         0.2318
       95.000  -0.187         0.2382
       99.000  -0.352         0.2681
       99.900  -0.383         0.3789
       99.990  -0.446         0.4064
       99.999  -0.467         0.4250
  
  
             Confidence Limits for the Median
             (Based on Maritz-Jarrett Standard Error for Quantiles)
  
 Response Variable: Y2
  
 Summary Statistics:
 Number of Observations:                  100
 Sample Minimum:                          -5.0249
 Sample Maximum:                          5.3818
 Sample Median:                           0.1507
 Sample Quantile Standard Error:          0.2162
  
  
  
 -----------------------------------------------------------------
   Confidence       Z      Z-Value X          Lower          Upper
    Value (%)   Value         StdErr          Limit          Limit
 -----------------------------------------------------------------
       50.000   0.674         0.1458         0.0048         0.2965
       75.000   1.150         0.2487        -0.0981         0.3994
       90.000   1.645         0.3556        -0.2050         0.5063
       95.000   1.960         0.4238        -0.2731         0.5744
       99.000   2.576         0.5569        -0.4063         0.7076
       99.900   3.291         0.7115        -0.5608         0.8621
       99.990   3.891         0.8412        -0.6905         0.9919
       99.999   4.417         0.9551        -0.8044         1.1057
  
  
             Hettmansperger-Sheater Median Confidence Limits
  
 -----------------------------------
   Confidence   Lower          Upper
    Value (%)   Limit          Limit
 -----------------------------------
       50.000  -0.032         0.3550
       75.000  -0.048         0.5086
       90.000  -0.069         0.5268
       95.000  -0.105         0.5454
       99.000  -0.139         0.5622
       99.900  -0.543         0.7184
       99.990  -0.779         0.8084
       99.999  -1.021         0.9848
  
  
             Confidence Limits for the Median
             (Based on Maritz-Jarrett Standard Error for Quantiles)
  
 Response Variable: Y3
  
 Summary Statistics:
 Number of Observations:                  100
 Sample Minimum:                          -27.0517
 Sample Maximum:                          8.6177
 Sample Median:                           0.0212
 Sample Quantile Standard Error:          0.1866
  
  
  
 -----------------------------------------------------------------
   Confidence       Z      Z-Value X          Lower          Upper
    Value (%)   Value         StdErr          Limit          Limit
 -----------------------------------------------------------------
       50.000   0.674         0.1258        -0.1046         0.1470
       75.000   1.150         0.2146        -0.1934         0.2358
       90.000   1.645         0.3069        -0.2856         0.3281
       95.000   1.960         0.3656        -0.3444         0.3869
       99.000   2.576         0.4805        -0.4593         0.5018
       99.900   3.291         0.6139        -0.5927         0.6351
       99.990   3.891         0.7258        -0.7046         0.7470
       99.999   4.417         0.8241        -0.8028         0.8453
  
  
             Hettmansperger-Sheater Median Confidence Limits
  
 -----------------------------------
   Confidence   Lower          Upper
    Value (%)   Limit          Limit
 -----------------------------------
       50.000  -0.086         0.1580
       75.000  -0.158         0.2898
       90.000  -0.225         0.3791
       95.000  -0.333         0.4389
       99.000  -0.380         0.4515
       99.900  -0.412         0.6212
       99.990  -0.485         0.8956
       99.999  -0.683         0.9482
  
  
             Confidence Limits for the Median
             (Based on Maritz-Jarrett Standard Error for Quantiles)
  
 Response Variable: Y4
  
 Summary Statistics:
 Number of Observations:                  100
 Sample Minimum:                          -9.6504
 Sample Maximum:                          3.0304
 Sample Median:                           0.0233
 Sample Quantile Standard Error:          0.0698
  
  
  
 -----------------------------------------------------------------
   Confidence       Z      Z-Value X          Lower          Upper
    Value (%)   Value         StdErr          Limit          Limit
 -----------------------------------------------------------------
       50.000   0.674         0.0471        -0.0238         0.0703
       75.000   1.150         0.0803        -0.0570         0.1036
       90.000   1.645         0.1148        -0.0915         0.1381
       95.000   1.960         0.1368        -0.1135         0.1601
       99.000   2.576         0.1798        -0.1565         0.2030
       99.900   3.291         0.2296        -0.2064         0.2529
       99.990   3.891         0.2715        -0.2482         0.2948
       99.999   4.417         0.3083        -0.2850         0.3315
  
  
             Hettmansperger-Sheater Median Confidence Limits
  
 -----------------------------------
   Confidence   Lower          Upper
    Value (%)   Limit          Limit
 -----------------------------------
       50.000  -0.032         0.0478
       75.000  -0.063         0.0691
       90.000  -0.083         0.1003
       95.000  -0.137         0.1559
       99.000  -0.213         0.2189
       99.900  -0.312         0.3046
       99.990  -0.389         0.4273
       99.999  -0.479         0.4552