Q1: What exactly is NIST providing with the Statistical Reference Datasets?
Q2: What areas of statistical computing are covered by the Statistical
Reference Datasets?
A2: At the current time, datasets and associated certified values are
provided for four areas of statistical interest:
Q3: What led NIST to offer this service?
A3: The Statistical Engineering Division of NIST has been approached by
users of statistical software packages and by scientists within its own
organization about the accuracy of statistical software. This service is
provided to aid these communities in evaluating the accuracy of commercial
and home-grown software and the community of software developers in
evaluating their algorithms.
Q4: I have a machine which is capable of carrying a large number of
significant digits. Doesn't that guarantee the accuracy of results?
A4: No, for two reasons.
Q5: Can NIST recommend statistical software packages which give consistently accurate results?
A5: NIST does not endorse or recommend any commercial products.
Q6: How do I know if my software package is giving accurate results?
A6: One criterion for judging the accuracy of software output is
comparison with "certified values" from sources that are known to be
reliable. The NIST collection of datasets serves this purpose. The
certified values for the datasets in this collection should be compared
with the values estimated from your software package. The NIST datasets
are ordered according to level of difficulty, and for completeness,
comparisons should be made over all levels of difficulty. Further
details can be found in the background information sections for each
statistical area.
Q7: My software gives results which disagree with the NIST certified results
in the __ significant digit. Should this difference be worrisome?
A7: The answer to this question depends upon your needs and the application.
NIST results are quoted to
a high number of significant digits for comparison purposes, not to indicate
statistical significance. Results from a given package will
depend on the statistical area being tested, the individual problem and the precision used in your computations. Guidance for comparing your results with the certified values is given in the background information sections for
each statistical area.
Q8: What should I do if my software gives results which are very different from the NIST results?
A8: Start by checking to see if your computer program is attempting
the same calculation as that used for the certified results. It may be
necessary to invoke some specific option, which is not the default, in
order to reproduce the certified results. Also, you should verify that
your computer program obtains correct results for the lowest difficulty
datasets before worrying about the results for datasets of moderate or high
difficulty. If you become convinced that the computer algorithm is at
fault, you may need to contact the software developer for a resolution of
the discrepancy.
Q9: How accurate are the NIST results?
A9: The NIST results for the linear regression, analysis of variance, and univariate pages are accurate to the fifteen quoted significant digits.
Results for the non-linear
regression pages are quoted to eleven significant digits for comparison
purposes only. A description of how certified results are obtained is
given as background information on the web page for each statistical area.
Q10: I am interested in the accuracy of statistics other than those
certified in the NIST pages. Can you help me?
A10: NIST is not prepared at this time to evaluate statistics other than
those given in these pages. However, you are invited to express your
interest in further developments by sending email to
StRD@nist.gov.
Q11: Will NIST in the future offer a service for testing statistical
software packages?
A11: NIST has no plans to offer such a service. However, a specialized
service for testing the performance of geometric fitting software for
coordinate measurement systems is available through the
NIST Calibration
Program; this special test service is called the Algorithm
Testing and Evaluation Program for Coordinate Measurement Systems.
Q12: Can we advertise that our commercially available software package is traceable to NIST certified results?
A12: There is no mechanism for establishing software traceability. The
NIST Statistical Reference Datasets are provided simply as an aid to
software development, and use of these materials for software testing is
strictly voluntary.