SED navigation bar go to SED home page go to Dataplot home page go to NIST home page SED Home Page SED Staff SED Projects SED Products and Publications Search SED Pages
Dataplot Vol 1 Vol 2

FISHER TWO SAMPLE RANDOMIZATION TEST

Name:
    FISHER TWO SAMPLE RANDOMIZATION TEST
Type:
    Analysis Command
Purpose:
    Perform a Fisher two sample randomization test for the equality of the means of two independent samples.
Description:
    The two sample t-test is the standard test for the equality of the means from two samples. This test is based on the following aassumptions

    1. the samples are randomly selected from infinite populations (equivalently the observations are independent)

    2. the samples come from normal populations

    3. the two populations have equal variances

    Randomization tests can be used when these assumptions are questionable. Fisher introduced randomization tests (also referred to as permutation tests) in 1935.

    The randomization test for the equality of the means for two samples is computed as follows:

    1. Given that sample one has n1 observations and sample two has n2 observations, randomly assign the n1 + n2 observations so that n1 observations are assigned to sample one and n2 observations and compute the difference of the means. This is a single permuation for the test.

    2. Generate all possible permutations of the n1 + n2 observations and compute the difference of the means for each permutation. The number of permutations is \( \left( \begin{array}{c} n1 \\ n2 \end{array} \right) = \frac{n1!}{n2!(n1-n2)!} \). Call this value NTOTAL for subsequent steps.

    3. Let DFULL denote the difference of the means for the original samples. Let Di denote the difference of the means for the i-th sample. Then the following p-values can be computed

        \( p(\mbox{upper tail}) = \frac{\mbox{number of } D_{i} \hspace{0.1 in} \ge \hspace{0.1 in} \mbox{DFULL}} {\mbox{NTOTAL}} \)

        \( p(\mbox{lower tail}) = \frac{\mbox{number of } D_{i} \hspace{0.1 in} \le \hspace{0.1 in} \mbox{DFULL}} {\mbox{NTOTAL}} \)

        \( p(\mbox{two tailed}) = \frac{\mbox{number of } |D_{i}| \hspace{0.1 in} \ge \hspace{0.1 in} \mbox{DFULL}} {\mbox{NTOTAL}} \)

    The primary drawback to this test is that NTOTAL grows rapidly as n1 and n2 increase. A test based on the full set of permutations may be computationaly prohibitive except for relatively small samples. For larger n1 and n2, one approach is to generate a random subset of the complete set of permutations (typically on the order of 4,000 to 10,000 random subsets will be generated).

    For this command, Dataplot is using the algorithm of Richards and Byrd. This algorithm generates the complete set of permutations. The advantage of this algorithm is that exact p-values are obtained for one-tailed tests and also for two-tailed tests when n1 = n2. If n1 is not equal n2, an approximate p-value is obtained for the two-tailed test. The primary drawback is that this test is limited to small sample sizes. Dataplot currently limits the maximum value of n1 and n2 to be 22. See the Note section below for some guidance to generating this test for larger samples based on randomly sampling the permutations.

    If the two samples are not randomly drawn from larger populations, the inference will be valid for the observations under study but not necesarily for the populations from which the observations are drawn.

Syntax 1:
    FISHER TWO SAMPLE RANDOMIZATION TEST <y1> <y2>
                            <SUBSET/EXCEPT/FOR qualification>
    where <y1> is the first response variable;
                <y2> is the second response variable;
    and where the <SUBSET/EXCEPT/FOR qualification> is optional.

    The <y1> and <y2> need not be the same length.

    Either <y1> or <y2> (or both) may be matrix arguments. If a matrix argument is given, the response variable will consist of all observations in that matrix. Although matrix arguments are allowed, they are rarely used for this command due to limitation on the size of the response variable.

Syntax 2:
    FISHER TWO SAMPLE RANDOMIZATION TEST <y1> ... <yk>
                            <SUBSET/EXCEPT/FOR qualification>
    where <y1> ... <yk> is a list of 1 to 30 response variables;
    and where the <SUBSET/EXCEPT/FOR qualification> is optional.

    This syntax will implement all the pairwise Fisher two sample randomization tests for the listed response variables. For example,

      FISHER TWO SAMPLE RANDOMIZATION TEST Y1 TO Y4

    is equivalent to

      FISHER TWO SAMPLE RANDOMIZATION TEST Y1 Y2
      FISHER TWO SAMPLE RANDOMIZATION TEST Y1 Y3
      FISHER TWO SAMPLE RANDOMIZATION TEST Y1 Y4
      FISHER TWO SAMPLE RANDOMIZATION TEST Y2 Y3
      FISHER TWO SAMPLE RANDOMIZATION TEST Y2 Y4
      FISHER TWO SAMPLE RANDOMIZATION TEST Y3 Y4

    The <y1>, ..., <yk> need not be the same length.

    Any of the listed response variables may be matrix arguments. If a matrix argument is given, the response variable will consist of all observations in that matrix. Although matrix arguments are allowed, they are rarely used for this command due to limitation on the size of the response variable.

Examples:
    FISHER TWO SAMPLE RANDOMIZATION TEST Y1 Y2
    FISHER TWO SAMPLE RANDOMIZATION TEST Y1 TO Y4

Note:
    According to Franks and Byrd, this test can also be used to obtained p-values for the Wilcoxon signed rank test and the Mann-Whitney rank sum test for small samples. These tests are essentially Fisher's test applied to the ranks of the original observations. So you can generate the ranks of the data (the ranks are based on the combined data) and apply the FISHER TWO SAMPLE RANDOMIZATION TEST to the ranked data.
Note:
    Dataplot saves the following internal parameters after this test:

      STATVALU = the value of the test statistic
      PVALUE = the p-value for the two-sided test
      PVALUELT = the p-value for the lower tailed test
Note:
    The following statistics are also supported:

      LET A = FISHER TWO SAMPLE RANDOMIZATION TEST Y1 Y2
      LET A = FISHET TWO SAMPLE RANDOMIZATION TEST PVALUE Y1 Y2

    In addition to the above LET command, built-in statistics are supported for about 20+ different commands (enter HELP STATISTICS for details).

Note:
    This test is specific to the two sample location test and generates all possible permutations of the data. For that reason, it is restricted to small sample sizes (currently a maximum sample size of 22 is allowed).

    The SAMPLE RANDOM PERMUTATION command can be used to implement other randomization tests (and to accomodate sample sizes greater than allowed here). The Program 2 and Program 3 examples demonstrate this. Although these examples demonstrate the difference of means statistic, other statistics can be easily substituted into these examples.

Default:
    None
Synonyms:
    None
Related Commands: Reference:
    Richards and Byrd (1996), "Fisher's Randomization Test for Two Small Independent Samples", Applied Statistics, Vol. 45, No. 3, pp. 394-398.

    Fisher (1935), "Design of Experiments", Edinburgh: Oliver and Boyd.

    Conover (1999), "Practical Non-Parametric Statistics", Third Edition, Wiley, p. 410.

    Higgins (2004), "Introduction to Modern Nonparametric Statistics", Thomson/Brooks/Cole, Duxbury Advanced Series, Chapter 2.

Applications:
    Nonparameteric statistics, two sample problem
Implementation Date:
    2011/06
Program:
     
    .  Example from p. 410 of Convover (1999), "Practical Nonparametric
    .  Statistics", Third Edition, Wiley.
    .
    let y1 = data 0 1 1 0 -2
    let y2 = data 6 7 7 4 -3 9 14
    let y3 = data 9 2 3 5 7
    let y4 = data 6 8 9 12 15
    set write decimals 5
    .
    let t    = fisher two sample rand test        y1 y2
    let pval = fisher two sample rand test pvalue y1 y2
    .
    print t pval
    .
    fisher two sample rand test y1 y2
    fisher two sample rand test y1 y2 y3 y4
        
    The following output is generated
     
     PARAMETERS AND CONSTANTS--
    
        T       --        0.00000
        PVAL    --        0.02778
     
    
                Two Sample Two-Sided Fisher Randomization Test
                            (Independent Samples)
     
    First Response Variable: Y1
    Second Response Variable: Y2
     
    H0: E(X) = E(Y)
    Ha: E(X) <> E(Y)
     
    Summary Statistics:
    Sample with Smaller Mean:
    Number of Observations:                  5
    Mean:                                    0.00000
    Sum of Observations:                     0.00000
    Sample with Larger Mean:
    Number of Observations:                  7
    Mean:                                    6.28571
    Sum of Observations:                     44.00000
    Difference of Means:                     -6.28571
     
    Test Statistic:                          0.00000
    Approximate P-Value (two-tailed test):   0.02778
    Exact P-Value (lower-tailed test):       0.01389
    
    
    
    
    
                Two Sample Two-Sided Fisher Randomization Test
                            (Independent Samples)
     
    First Response Variable: Y1
    Second Response Variable: Y2
     
    H0: E(X) = E(Y)
    Ha: E(X) <> E(Y)
     
    Summary Statistics:
    Sample with Smaller Mean:
    Number of Observations:                  5
    Mean:                                    0.00000
    Sum of Observations:                     0.00000
    Sample with Larger Mean:
    Number of Observations:                  7
    Mean:                                    6.28571
    Sum of Observations:                     44.00000
    Difference of Means:                     -6.28571
     
    Test Statistic:                          0.00000
    Approximate P-Value (two-tailed test):   0.02778
    Exact P-Value (lower-tailed test):       0.01389
     
     
                Two Sample Two-Sided Fisher Randomization Test
                            (Independent Samples)
     
    First Response Variable: Y1
    Second Response Variable: Y3
     
    H0: E(X) = E(Y)
    Ha: E(X) <> E(Y)
     
    Summary Statistics:
    Sample with Smaller Mean:
    Number of Observations:                  5
    Mean:                                    0.00000
    Sum of Observations:                     0.00000
    Sample with Larger Mean:
    Number of Observations:                  5
    Mean:                                    5.20000
    Sum of Observations:                     26.00000
    Difference of Means:                     -5.20000
     
    Test Statistic:                          0.00000
    Approximate P-Value (two-tailed test):   0.00794
    Exact P-Value (lower-tailed test):       0.00397
     
     
                Two Sample Two-Sided Fisher Randomization Test
                            (Independent Samples)
     
    First Response Variable: Y1
    Second Response Variable: Y4
     
    H0: E(X) = E(Y)
    Ha: E(X) <> E(Y)
     
    Summary Statistics:
    Sample with Smaller Mean:
    Number of Observations:                  5
    Mean:                                    0.00000
    Sum of Observations:                     0.00000
    Sample with Larger Mean:
    Number of Observations:                  5
    Mean:                                    10.00000
    Sum of Observations:                     50.00000
    Difference of Means:                     -10.00000
     
    Test Statistic:                          0.00000
    Approximate P-Value (two-tailed test):   0.00794
    Exact P-Value (lower-tailed test):       0.00397
     
     
                Two Sample Two-Sided Fisher Randomization Test
                            (Independent Samples)
     
    First Response Variable: Y2
    Second Response Variable: Y3
     
    H0: E(X) = E(Y)
    Ha: E(X) <> E(Y)
     
    Summary Statistics:
    Sample with Smaller Mean:
    Number of Observations:                  5
    Mean:                                    6.28571
    Sum of Observations:                     26.00000
    Sample with Larger Mean:
    Number of Observations:                  7
    Mean:                                    5.20000
    Sum of Observations:                     44.00000
    Difference of Means:                     1.08571
     
    Test Statistic:                          26.00000
    Approximate P-Value (two-tailed test):   0.72222
    Exact P-Value (lower-tailed test):       0.36111
     
     
                Two Sample Two-Sided Fisher Randomization Test
                            (Independent Samples)
     
    First Response Variable: Y2
    Second Response Variable: Y4
     
    H0: E(X) = E(Y)
    Ha: E(X) <> E(Y)
     
    Summary Statistics:
    Sample with Smaller Mean:
    Number of Observations:                  5
    Mean:                                    5.20000
    Sum of Observations:                     26.00000
    Sample with Larger Mean:
    Number of Observations:                  5
    Mean:                                    10.00000
    Sum of Observations:                     50.00000
    Difference of Means:                     -4.80000
     
    Test Statistic:                          26.00000
    Approximate P-Value (two-tailed test):   0.06349
    Exact P-Value (lower-tailed test):       0.03175
     
     
                Two Sample Two-Sided Fisher Randomization Test
                            (Independent Samples)
     
    First Response Variable: Y3
    Second Response Variable: Y4
     
    H0: E(X) = E(Y)
    Ha: E(X) <> E(Y)
     
    Summary Statistics:
    Sample with Smaller Mean:
    Number of Observations:                  5
    Mean:                                    5.20000
    Sum of Observations:                     26.00000
    Sample with Larger Mean:
    Number of Observations:                  5
    Mean:                                    10.00000
    Sum of Observations:                     50.00000
    Difference of Means:                     -4.80000
     
    Test Statistic:                          26.00000
    Approximate P-Value (two-tailed test):   0.06349
    Exact P-Value (lower-tailed test):       0.03175
     
        

Privacy Policy/Security Notice
Disclaimer | FOIA

NIST is an agency of the U.S. Commerce Department.

Date created: 12/18/2017
Last updated: 12/18/2017

Please email comments on this WWW page to alan.heckert.gov.