Testing of distributions

[pitdicker]

We should add some testing for distributions. Unfortunately since distribution samples are random, this is inherently difficult. Original issue dhardy#72.

See for some direction rust-lang/rust#10084, and the random-tests.

[vks]

Also see #290.

[dhardy]

@vks has some plans for histogram-based testing, though possibly only after PDFs are added.

Separately, we should add some tests of value-stability.

[vks]

@vks has some plans for histogram-based testing, though possibly only after PDFs are added.

See tests/uniformity.rs, which could be generalized to arbitrary distributions (with given PDFs).

[dhardy]

@vks could I interest you (or potentially someone else) in adding at utility to rand_distr (perhaps an example) which plots histograms of each distribution from actual samples (not PDFs)? This would at least let us eye-ball that distributions are doing roughly the right thing.

[vks]

I was thinking about using sparklines for that. I'm interested, but I can't make promises that I'll get to it before the end of the year.

[vks]

Resolving this issue was started in #1121, more work is needed to extend it to the other distributions.

[vks]

I think we should either make this issue more actionable or close it.

For which distributions do we want histogram tests? Sometimes, calculating the PDF is non-trivial and requires special functions.

[dhardy]

I suggest the following approach:

1. Pick a representative parameter set (or possibly a small number of parametrisations).
2. Generate a histogram with this parametrisation and manually verify that the result looks reasonable.
3. Potentially lower the number of buckets and samples used by the histogram, and check the result is still similar. (We want tests to be reasonably fast to run.)
4. The test should enforce exact reproduction of this histogram.

This should allow fast and robust testing (within the limits of reproducibility), with the caveat that any value-breaking change must again be visually confirmed.

It should be applicable to all distributions, though may be hard to visualise for multi-dimensional outputs.

It cannot confirm perfect accuracy, especially in degenerate cases.

[JamboChen]

I am a university student with a background in statistics and would love to contribute to this repository. I have a few questions regarding this issue. I noticed that histograms are used in the tests for visual inspection by comparing the expected curve with the sample curve.

Would it be possible to use a chi-square test for validation? The process should be similar, but hypothesis testing might provide more convincing results.

I wrote a simple test for this: https://github.com/JamboChen/distr_test. The code is rough, and I'm unsure if this is the right approach to use here, I'd appreciate any feedback.

[dhardy]

@JamboChen feel free to take a look at #1494 by @benjamin-lieser implementing a Kolmogorov-Smirnov test.

[dhardy]

Implemented in #1504 thanks to @JamboChen and @benjamin-lieser.

Failure cases:

• Hypergeometric (100, 50, 49): Hypergeo fix #1510
• Poisson 1e9
• Poisson 1.844e19

[dhardy]

We have a PR for the hypergeo issue and #1515 for the Poisson issue, so this can be closed.