Future of distributions in rand

[vks]

When working with distributions it is very useful to have access to their density, their distribution function and their quantile function in addition to being able to sample from them. Furthermore, it is nice to be able to calculate their (theoretical, exact) moments. I think it makes sense to have all this functionality in a common interface.

This is implemented in the statrs crate. There is some overlap with rand, sampling is implemented there as well, but for a lot more distributions.

I see the following options:

1. Only implement sampling in rand. Port the missing distributions from statrs to rand.
2. Also implement the theoretical properties of the distributions mentioned above in rand, essentially duplicating statrs.
3. Remove the distributions from rand and suggest to use statrs instead.

What do you think?

[dhardy]

Very good question. I don't know; potentially I'm open to big changes (post 0.5).

Statrs focusses on f64 despite using a generic T in the Distribution trait; this probably means at least Range and Uniform should remain in rand.

The Distribution trait in statrs is identical to ours (aside from R: Rng+?Sized addition), so it doesn't make much sense to have both (after 0.5 introduces our Distribution trait). I don't think it makes sense to remove this or basic uniform sampling from rand so statrs could simply re-export our trait later (unfortunately meaning it can't reach 1.0 before rand does).

The other distributions we have (exponential, normal, gamma) do feel out-of-place in rand so potentially we could remove those. I'm embarrassed to say I wasn't aware of statrs until now.

One thing not clear about the statrs distributions is how precise the sampling is — e.g. is the Bernoulli distribution accurate for p < ε (but within the normal range of f64)?

I think we should also ask @boxtown what he thinks?

[dhardy]

I see the Bernoulli distribution in statrs simply wraps Binomial, and the latter has a simple O(n) implementation, in contrast to the rejection method used in @fizyk20's PR.

[boxtown]

I'm open to the idea of depending on rands Distribution trait. Although to be honest I don't see any issue with the duplication. I am of course open to PRs if there is a desire to move sampling code into Statrs. I'd be willing to tackle it myself if someone is willing to open issues, it might just take a little longer

[vks]

Statrs focusses on f64 despite using a generic T in the Distribution trait; this probably means at least Range and Uniform should remain in rand.

I think the main reason for this is simplicity. AFAIK, the sampling of f32 and integers is a desired feature. See statrs-dev/statrs#17.

The other distributions we have (exponential, normal, gamma) do feel out-of-place in rand so potentially we could remove those.

I agree, it makes sense to move them to statrs and advertise the create for generating floats. In the long term, the discrete distributions could be moved there as well.

I see the Bernoulli distribution in statrs simply wraps Binomial, and the latter has a simple O(n) implementation, in contrast to the rejection method used in @fizyk20's PR.

Yes, I think it makes sense to avoid duplicated work implementing the distributions, which is the main reason I brought up the issue.

[dhardy]

I labelled this low-priority mostly because it needs more planning and would probably be better to discuss after more distribution code has been implemented / optimised / reviewed, and also because it's beyond the scope of the next release, but this is an important topic to finalise before the 1.0 release.

[pitdicker]

Just my personal opinion. There are a few distributions I really want to keep/have in Rand:

• Some default distribution that all types can implement and that ties in with Rng::gen (Uniform).
• The Range distribution.
• Some way to get floats with high precision, including a range.
• A Bernoulli distribution, making use of the high precision float method to improve upon Rng::gen_bool.
• I have really started to like type Exp1 distribution. It is an easy way to improve the performance of code that would otherwise use rng.gen::<f64>().ln().
• The Alphanumeric distribution. It does not irreplaceable or perfect, but it is just a nice piece of functionality.

What I am not sure about is generating random numbers that fit some probability distribution. Because if have not used them myself, I can't care strongly whether they are part of Rand or not. Two not-so-strong reasons to have them in Rand (or a clear companion crate to point to):

• Generating values in those distributions is a big randomness-related topic. A lot of the literature is about this area, it seems much more to me than there is about RNGs.
• Sometimes using some distribution is just the right tool for the job. Simulations / statistical problems are not the only uses. It is nice to have them easily available.

On the other hand, stats-rs is certainly going beyond what rand should provide (in my opinion), and no doubt provides a more complete and useful library for some area's. And it is taking it's inspiration from a good source 😄.

I think it is best to keep generating values according to some distribution in Rand. For simple use cases people can just use Rand. Then it would be nice if stats-rs would use these implementations (work together on improving them), and we would link to stats-rs for the place to find more features.

[burdges]

It's sometimes useful if two projects do the same things from different perspectives, especially if those things involve tricky things like presenting mathematics well.

[dhardy]

It would be good to make some progress on this now. From @vks's suggestions, I like 1 and 3 the best.

1. Only implement sampling in rand. Port the missing distributions from statrs to rand.

This means all the distribution objects (e.g. the Normal struct) exist in Rand, as does the Distribution trait and implementations of it.

Traits like Mean and Variance (see statrs::statistics) remain in statrs and are implemented for the distribution objects in Rand.

3. Remove the distributions from rand and suggest to use statrs instead.

We should keep Standard, Uniform, Alphanumeric and Bernoulli in Rand (these could be re-exported from or duplicated in statrs). For everything else, users should use statrs.

This could be significantly simpler to maintain, and perhaps makes the scope of Rand clearer.

Note that several of the distribution implementations in Rand may be faster than those in statrs but I imagine the statrs implementations can be optimised.

4. New sub-crate

This is just a variant of 1. We move the distribution objects (and their implementations of Distribution) to a sub-crate, e.g. rand_dist, but keep the Distribution trait and a few of the distributions (Standard, Uniform, Alphanumeric, Bernoulli) in Rand.

Functionally, this is not much different than option 3. There main difference is probably that users do not have to import the full statrs to sample from the more complex distributions (though this may not make much difference, especially if we include all the less-commonly-used distributions in rand_dist).

---

There is some overlap with #494 in that if we use option 3 or 4 then we do not need to worry about most of the distributions should we implement high and low precision sampling variants for e.g. Uniform and Bernoulli.

This is also related to #431 in design concepts; do we prefer few large crates or many small crates? E.g. the statrs::statistics traits can be kept separate from the distributions, and potentially the distributions can be split into multiple sub-crates (though probably not a good idea).

[dhardy]

@vks don't you think we should resolve this issue before trying to duplicate all functionality in statrs?

There are several issues currently:

1. There is a lot of duplication between rand and statrs
2. Rand distributions aren't easy to compare (Comparing distributions #560) — having properties like in statrs may be useful
3. We are now accumulating a lot of distributions into Rand, which is a lot of code that most users are not going to need. On the other hand, some distributions, e.g. Bernoulli, Exp, Normal, may be used more commonly and there is some rationale in not having to import a big stats crate to access them.
4. All our distribution constructors report errors via panic, which is less useful when input comes from untrusted sources. Possibly we should switch to returning Result types, but this would be a big breaking change.
5. statrs still depends on Rand 0.3

I'm not sure where we should go from here...

1. Ignore statrs and implement the bits we want here. Not ideal, but this is what seems to be happening...
2. Implement sampling within Rand but extended properties in statrs. This seems to be what was suggested above, however I don't know how it can work without making the internal parameters public.
3. Remove most distributions from Rand and recommend statrs.
4. Same, but to a sub-crate (e.g. rand_stat)
5. Split distributions into families and handle each separately; e.g. we roughly have these families:
   • uniform and utility distributions (e.g. Alphanumeric)
   • Exp, Normal, LogNormal (exp and normal aren't really related, but both use Ziggurat sampling currently)
   • Cauchy, Binomial, Poisson
   • Gamma and derived (at least, the implementations of Dirichlet and Beta depend on it)
   • other: WeightedIndex, Pareto, Triangular, Bernoulli

[vks]

@dhardy The status quo is that Rand only implements sampling. Most of the work in statrs is calculating the statistics (PDF, CDF, moments etc.). In principle statrs could use the sampling implementations from Rand without introducing a breaking change, so moving them to Rand could actually reduce duplication.

However, as you noted this will result in duplication: A distribution struct has to be created for sampling and another for calculating statistics, unless the internal parameters are made accessible.

I think the main decision is whether we want the code for calculating statistics in Rand or not:

• If yes, I think it makes sense to essentially port statrs to rand_stat. We want at least PDF support for testing, and statistics would be nice for Comparing distributions #560.
• If not, I think we should remove all non-uniform distributions from Rand. This would be a breaking change but probably not affect most (?) users. statrs would reuse the code for uniform distributions.

Personally, I would prefer the first option, but the disadvantage is that this would add a lot of code that most users will not use. It would also make Rand a bit of a misnomer.

Maybe we should get some community feedback on Reddit/the Rust user forum.

All our distribution constructors report errors via panic, which is less useful when input comes from untrusted sources. Possibly we should switch to returning Result types, but this would be a big breaking change.

This could be done without breaking changes by introducing try_new constructors.

statrs still depends on Rand 0.3

That should be easy to fix.

[boxtown]

I think having a rand_stat sub-crate would be a good idea. I'm even open to doing some of the porting myself. I've had less and less time recently to update statrs or even keep it maintained due to work and personal reasons so I think having a more "official" community maintained statistics library would be beneficial.

[dhardy]

Sure. If @boxtown would prefer a community-maintained crate over statrs then adding rand_stat makes a lot of sense.

I guess we may as well take the opportunity to re-design as we see fit (potentially combine the Checked* traits with un-checked versions, via type Error parameter or blanket implementation). There is however quite a lot of functionality in statrs and it already has quite a good design.

@vks do you want to make a Reddit post about this then? It sounds like there is quite a bit of scope for input, though probably only a small set of interested users.

[pitdicker]

Maybe we should be careful to not take to much on our plate. @dhardy Don't you think this will add quite a lot of extra work? Is it something to start right now?

[dhardy]

I think it is important to know where we are headed before blindly adding every distribution we can. As for being too much for us to handle, sure, it is more than we can get to right now — except there are currently five people making significant contributions to Rand, and as @boxtown says, it probably makes sense to move to a community-maintained project for an important stats library at some point (I doubt any of us will continue working on Rand for the lifetime of the Rust project).

In the mean-time though, it would probably make more sense to port statrs to Rand 0.5.

@boxtown, as the developer of statrs, do you have any recommendations for the design of rand_stat? It is tempting to put the distributions at the top-level instead of in a sub-module, but there is quite a lot of additional functionality so this may not be a good approach. Roughly, statrs has:

• a few consts — these can stay in a sub-module either way I think
• the distributions
• PDF (continuous), PMF (discrete), CDF and inverse CDF traits
• a real modulus operator — is being added to std
• various functions — we should definitely keep these somewhere
• sequence generators — I think these could be removed or put in a separate crate? Or maybe just hide; they're only for testing I think?
• an almost_eq function and assert_almost_eq macro — more utilities that don't really belong (though comparable to rand_core::le etc.)
• a host of statistics traits (Min, Max, Mean, Variance, ...)
• StatsError and Result types

[vks]

@vks do you want to make a Reddit post about this then? It sounds like there is quite a bit of scope for input, though probably only a small set of interested users.

Sure, I can do that. What is our current consensus? We could do something like that:

1. Factor distributions in Rand out to rand_stat, while reexporting them in rand.
2. Port the remaining sampling code from statrs (and possibly GSL).
3. Port PDFs, CDFs etc.
4. Port statistics.

Maybe we should be careful to not take to much on our plate. @dhardy Don't you think this will add quite a lot of extra work? Is it something to start right now?

I think it can be done incrementally, as mentioned above.

[boxtown]

Finally cut 0.10.0 that includes @vks 's changes fyi

[dhardy]

Rand 0.6 is basically done, and resolving this issue should be next on the agenda.

We could remove all distributions other than Standard, Uniform, Bernoulli, Open*01 and WeightedIndex. This would:

• remove approx 2100 lines of code (of approx 12000 under src or 18%)
• remove the log_gamma and ziggurat utility functions
• remove around half our cfg(feature = "std") gates (most of the rest being related to OsRng / seeding)
• leave over 3000 lines of code in distributions (a large portion of this is uniform sampling for many different types)

Is this a good goal? It doesn't exactly make Rand a small lib, but does move us in that direction (though we couldn't go much further without either removing a lot of features or modularisation pushing OsRng etc. into a new lib — I don't believe either of those are goals, though I have considered moving "generators" to a sub-lib).

The other perspective is de-duplication with statrs. Doing the above would still leave Bernoulli and Uniform in both, which I don't want to remove from Rand. Note that while de-duplication has some advantages it might not be a goal we want — e.g. statrs explicitly supports reporting properties like mean, std-dev and calculating PDFs for its distributions, which sometimes has implications for internal representations.

[vks]

My plan was to move all distributions (including the ones you mentioned) to a rand_dist crate and make rand use that crate.

In principle I would like to have everything about distributions in one place. This is the approach taken by R and Julia, which I consider best-in-class for statistics. However, for now it should be enough to focus on sampling.

statrs explicitly supports reporting properties like mean, std-dev and calculating PDFs for its distributions, which sometimes has implications for internal representations.

I'm also not sure whether we can get away with one internal representation for everything. If a different internal representation is needed, it can probably be implemented on top of the representation used for sampling.

[dhardy]

In principle I would like to have everything about distributions in one place.

But this is what statrs already does?

You mean you would like to remove even Bernoulli, Uniform and Standard from rand? That doesn't make much sense to me; they are the building blocks of Rng::gen_bool, Rng::gen_range and Rng::gen.

[vks]

But this is what statrs already does?

Yes, basically. In any case, this is something to consider after moving the sampling to rand_dist.

You mean you would like to remove even Bernoulli, Uniform and Standard from rand?

rand would still depend on rand_dist, so Rng would still work as of now. (Assuming there are no problems with circular dependencies... In principle, rand_dist should be fine with using rand_core.) We might consider dropping the reexports of the distributions in rand, but that is a breaking change anyway.

[dhardy]

So you are proposing modularisation... I don't really see the point though; it doesn't help us and won't help most users (who will still depend on all the same code, just in two crates instead of one).

Part of the point of removing many distributions (from my POV) is that most users do not use those distributions, therefore will depend on less code. (There is probably also a significant subset who only want a randomness source like thread_rng but no sampling algorithms.)

[vks]

I don't really see the point though; it doesn't help us and won't help most users

It helps us to iterate on the distributions without having to worry about rand. Are you suggesting to implement some distributions in rand and some in rand_dist? This seems like an unfortunate separation to me. It would make it more difficult to work on the design of rand_dist without changing rand as well.

There is probably also a significant subset who only want a randomness source like thread_rng but no sampling algorithms.

This is not possible, because of the convenience methods in Rng. However, it would be possible if thread_rng and friends were moved to their own crate (which might actually make sense).

[burdges]

It helps us to iterate on the distributions without having to worry about rand.

No. You cannot iterate on the distributions upon which rand depends, like Uniform, just by having them in another crate. rand even reexports them most likely.

It helps us to iterate on the distributions without having to worry about rand. Are you suggesting to implement some distributions in rand and some in rand_dist? This seems like an unfortunate separation to me.

If I understand, you want nonsensical things:

• consolidate all-ish distributions into one rand_dist crate, but extremely niche distributions clearly exist.
• rand to depend on rand_dist making it impossible to use rand to implement distributions.

[dhardy]

However, it would be possible if thread_rng and friends were moved to their own crate (which might actually make sense).

Yes, this is what I meant — without having Rng in scope (RngCore is sufficient for thread_rng and OsRng). But this is another issue and related to #579.

[vks]

No. You cannot iterate on the distributions upon which rand depends, like Uniform, just by having them in another crate. rand even reexports them most likely.

You could release newer versions of rand_dist while rand uses an older version, couldn't you?

consolidate all-ish distributions into one rand_dist crate, but extremely niche distributions clearly exist.

I'm not sure it's worth it to keep them in two different places.

rand to depend on rand_dist making it impossible to use rand to implement distributions.

I don't see how that would be a problem.

[dhardy]

@vks yes, this would partially decouple rand and rand_dist versions, but since users would still be dependent on both, I don't see the advantage.

It would make it more difficult to work on the design of rand_dist without changing rand as well.

Is this your real rationale? I wasn't intending on doing much redesign anyway, myself.

I don't see any good rationale for your idea so far.

[vks]

but since users would still be dependent on both, I don't see the advantage.

Even if we keep the distributions used by Rng in rand and reexport them in rand_dist, we will have this problem, because we need to reexport the moved distributions in rand. So rand will have to depend on rand_dist until we decide to break backwards compatibility. In any case, I don't have a strong preference.

(Ideally, I would like to see rand being a façade for the rand_* crates, similar to num.)

Is this your real rationale?

No, my real rationale is that I think it makes sense to modularize that functionality into a different crate. The decoupling is one of the advantages.

[dhardy]

Well, potentially we could push:

• the trait plus core distributions (Standard, Uniform, Bernoulli, ...?) into rand_dist, depending only on rand_core
• other distributions into rand_stat, depending on rand_dist

then re-export rand_dist as rand::distributions. I'm not sure whether to do the whole, but we can already start on rand_stat (though without bothering with the deprecations).

(Ideally, I would like to see rand being a façade for the rand_* crates, similar to num.)

To fully realise this, we'd also need at least rand_thread (or _entropy or both), _seq and _rng. To me this just seems unnecessary?

Further, we currently use Rng as the "front-end trait for users" and RngCore just for implementation. We'd need to tweak this a bit, e.g. replacing fn sample<R: Rng + ?Sized> with fn sample<R: RngCore + ?Sized>. This is fine, but does mean users get to see more of the complexity of this crate (though that's kind of the case already, e.g. when calling Uniform::new with unsupported types the error message is complex).

Edit: until we've finished dealing with the fall-out from 0.6.2 we should hold off on big PRs.

[dhardy]

I think we should start on this now — the goal being to reduce rand itself to the functionality we think it should have by default, parking the rest in another crate like rand_stat for now. This is not the ideal approach, but making this move at least allows us to get the rand crate closer to where we want it.

Eventually this could still merge with statrs or not — this would allow some de-duplication, but I don't like to confuse users with overlap like which Bernoulli or Uniform implementation to use (statrs has both, but we definitely want to keep our versions in Rand for use by Rng::gen_range etc.).

To do items:

• create new crate — in a new repo?
• move distributions (most which are not depended on by Rng) to this crate
• move over relevant issues

[dhardy]

No, my real rationale is that I think it makes sense to modularize that functionality into a different crate. The decoupling is one of the advantages.

We've already had complaints about Rand using two many crates. Adding another crate doesn't really gain anything, since if rand depends on rand_dist, then breaking changes in rand_dist still require a new release in rand (and delaying doing so would just be a nuisance to users since the latest version of each crate would have a different API for the same stuff).

So lets do the following:

• release rand_distr, versioned in lock-step with rand (so users of both should use the same minor version)
• keep some parts of rand::distributions (e.g. Distribution, Uniform) where they are and re-export from rand_distr
• move other items to rand_distr

Thus from a user's point of view, they can just use rand for the basics or use rand_distr for the full set of distributions (with rand_distr being a pure superset of rand::distributions).

This change lets us do two things:

• slim down Rand for many users
• make breaking changes to the less common distributions now (while moving to rand_distr)

[vks]

Sounds good to me! I'll presumably try to implement this in a week or so.

[dhardy]

Keeping the versions in lock-step does have disadvantages:

• breaking changes in rand_distr must wait on a version bump in rand
• this approach differs from all other Rand crates, thus will surprise users

On the other hand, without this, re-exporting all of rand::distributions makes less sense, because it is difficult to predict whether the re-exported contents are currently equivalent or not (though since the Distribution trait is implemented by external libs we will likely have to use upgrade shims at some point; I believe this is already an issue when using statrs).

However, I think having all distributions available through rand_distr is still going to be more user-friendly than forcing users to remember which distributions are in which package.

This means that rand_distr v0.1 can just be a copy of the current state of distributions, and v0.2 can make breaking changes, while hopefully both versions can be compatible with rand v0.6-0.7.

FYI I've started working on this

[vks]

rand_distr has been implemented. The duplication between statrs and rand_distr still needs to be addressed.

[dhardy]

Does this duplication need to be addressed? @boxtown

Statrs claims to be a port from the C# Math.NET library. It implements a bunch of things (e.g. error function, CDFs) that are quite specialised (not widely used). Its distributions are implemented such that properties like mean etc. can easily be calculated.

rand_distr on the other hand has a narrower focus: only implement sampling algorithms, and optimise for speed.

Because of this I think there is scope for the on-going existance of both libraries, and I have documented this here and here.

(Note: another possibility would be including the extra functionality in rand_distr or another crate in order to make statrs obsolete since @boxtown said above he has less interest in continued maintenance. However I don't have an interest in doing this either, hence it seems more likely we will continue with two crates. This goes beyond the scope of this issue however, so lets close #290.)

[vks]

We will need some of statrs functionality (namely PDFs) to properly test sampling.

[boxtown]

I don't really see a need to de-dupe. Like @burdges mentioned above I don't see a harm in having two approaches to sampling and like you mentioned, the bulk of the work Statrs is doing is in calculating attributes of various distributions

[dhardy]

We will need some of statrs functionality (namely PDFs) to properly test sampling.

🤣

Last time I tried histogram testing, I think even with 10 million samples I was struggling to get useful results. I think it's probably a dead-end, at least for standard integration testing (though it may be somewhat usable for offline analysis). OTOH "black box testing" (aka testing a few samples produced the same result as last time) is easy to do and somewhat useful.

[vks]

I was able to get errors smaller than 0.001 with a million samples, but this is for (kind of) uniform distributions: https://github.com/rust-random/rand/blob/master/rand_distr/tests/uniformity.rs

For non-uniform distributions it will be more tricky.