Auto merge of #5921 - Eh2406:proptest, r=alexcrichton

use proptest to fuzz the resolver

This has been a long time goal. This uses proptest to generate random registry indexes and throws them at the resolver.

It would be simple to generate a registry by,
1. make a list of name and version number each picked at random
2. for each pick a list of dependencies by making a list of name and version requirements at random.

Unfortunately, it would be extremely unlikely to generate any interesting cases, as the chance that the random name you depend on was also generated as the name of a crate is vanishingly small. So this implementation works very hard to ensure that it only generates valid dependency requirements.

This is still a WIP as it has many problems:
- [x] The current strategy is very convoluted. It is hard to see that it is correct, and harder to see how it can be expanded. Thanks to @Centril for working with me on IRC to get this far. Do you have advice for improving it?
- [X] It is slow as molasses when run without release. I looked with a profilere and we seem to spend 2/3 of the time in `to_url`. Maybe we can special case `example.com` for test, like we do for `crates.io` or something? Edit: Done. `lazy_static` did its magic.
- [x] `proptest` does not yet work with `minimal-versions`, a taste of my own medicine.
- [x] I have not verified that, if I remove the fixes for other test that this regenerates them.

The current strategy does not:
- [x] generate interesting version numbers, it just dose 1.0.0, 2.0.0 ...
- [x] guarantee that the version requirements are possible to meet by the crate named.
- [ ] generate features.
- [ ] generate dev-dependencies.
- [x] build deep dependency trees, it seems to prefer to generate crates with 0 or 1 dependents so that on average the tree is 1 or 2 layers deep.

And last but not least, there are no interesting properties being tested. Like:
- [ ] If resolution was successful, then all the transitive requirements are met.
- [x] If resolution was successful, then unpublishing a version of a crate that was not selected should not change that.
- [x] If resolution was unsuccessful, then it should stay unsuccessful even if any version of a crate is unpublished.

- [ ] @maurer suggested testing for consistency. Same registry, same cargo version, same lockfile, every time.
- [ ] @maurer suggested a pareto optimality property (if all else stays the same, but new package versions are released, we don't get a new lockfile where every version is <= the old one, and at least one is < the old one)

Author: bors

Cargo.toml

@@ -94,6 +94,7 @@ features = [
 
 [dev-dependencies]
 bufstream = "0.1"
+proptest = "0.8.7"
 
 [[bin]]
 name = "cargo"

src/cargo/core/resolver/mod.rs

@@ -200,7 +200,7 @@ fn activate_deps_loop(
         }
     }
 
-    let mut ticks = 0;
+    let mut ticks = 0u16;
     let start = Instant::now();
     let time_to_print = Duration::from_millis(500);
     let mut printed = false;
@@ -240,6 +240,18 @@ fn activate_deps_loop(
                 config.shell().status("Resolving", "dependency graph...")?;
             }
         }
+        // The largest test in our sweet takes less then 5000 ticks
+        // with all the algorithm improvements.
+        // If any of them are removed then it takes more than I am willing to measure.
+        // So lets fail the test fast if we have ben running for two long.
+        debug_assert!(ticks < 50_000);
+        // The largest test in our sweet takes less then 30 sec
+        // with all the improvements to how fast a tick can go.
+        // If any of them are removed then it takes more than I am willing to measure.
+        // So lets fail the test fast if we have ben running for two long.
+        if cfg!(debug_assertions) && (ticks % 1000 == 0) {
+            assert!(start.elapsed() - deps_time < Duration::from_secs(90));
+        }
 
         let just_here_for_the_error_messages = deps_frame.just_for_error_messages;
 

tests/testsuite/main.rs

@@ -9,8 +9,12 @@ extern crate flate2;
 extern crate git2;
 extern crate glob;
 extern crate hex;
+#[macro_use]
+extern crate lazy_static;
 extern crate libc;
 #[macro_use]
+extern crate proptest;
+#[macro_use]
 extern crate serde_derive;
 #[macro_use]
 extern crate serde_json;