So far, we've only seem examples of very simple & trivial properties, doing little more than showcasing syntax. However, what if we're operating on some more complicated datastructure and want to check whether the operations we can perform on it uphold the invariants we expect? This too can, in a for now basic form, be done with Supposition.jl.
Consider this example from the movie Die Hard With A Vengeance:
\n
","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"The problem John McClane & Zeus Carver have to solve is the well known 3L & 5L variation on the water pouring puzzle. You have two jugs, one that can hold 3L of liquid and one that can hold 5L. The task is to measure out precisely 4L of liquid, using nothing but those two jugs. Let's model the problem and have Supposition.jl solve it for us:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"struct Jugs\n small::Int\n large::Int\nend\nJugs() = Jugs(0,0)","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"We start out with a struct holding our two jugs; one Int for the small jug and one Int for the large jug. Next, we need the operations we can perform on these jugs. These are","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Filling a jug to the brim\nNo partial filling! That's not accurate enough.\nEmptying a jug\nNo partial emptying! That's not accurate enough.\nPouring one jug into the other\nAny leftover liquid stays in the jug we poured from - don't spill anything!","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Defining them as functions returning a new Jugs, we get:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"# filling\nfill_small(j::Jugs) = Jugs(3, j.large)\nfill_large(j::Jugs) = Jugs(j.small, 5)\n\n# emptying\nempty_small(j::Jugs) = Jugs(0, j.large)\nempty_large(j::Jugs) = Jugs(j.small, 0)\n\n# pouring\nfunction pour_small_into_large(j::Jugs)\n nlarge = min(5, j.large + j.small)\n nsmall = j.small - (nlarge - j.large)\n Jugs(nsmall, nlarge)\nend\n\nfunction pour_large_into_small(j::Jugs)\n nsmall = min(3, j.small + j.large)\n nlarge = j.large - (nsmall - j.small)\n Jugs(nsmall, nlarge)\nend\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"From the top, we have filling either jug (note that we can only fill the small jug up to 3L, and the large up to 5L), emptying either jug, and finally pouring one into the other, taking care not to spill anything (i.e., any leftovers stay in the jug we poured out of).","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"We can very easily now generate a sequence of operations:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"using Supposition\n\nraw_ops = (fill_small, fill_large, empty_small, empty_large, pour_small_into_large, pour_large_into_small)\ngen_ops = Data.Vectors(Data.SampledFrom(raw_ops))\ngen_ops = Data.Vectors(Data.SampledFrom(raw_ops); min_size=5, max_size=10) # hide\nexample(gen_ops)","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Generating a sequence of operations is simply generating a vector from all possible ones! This is the input to our property. We declare that for all sequences of operations we can do with a Jug, all invariants we expect must hold true.","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Speaking of invariants, we need three of them that must be preserved at all times:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"The small jug must ALWAYS have a fill level between 0 and 3 (inclusive).\nThe large jug must ALWAYS have a fill level between 0 and 5 (inclusive).\nThe large just must NEVER have a fill level of exactly 4.","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"The last invariant may look a bit odd, but remember that Supposition.jl is trying to find a falsifying example. The first two invariants are sanity checks to make sure that our pouring functions are well behaved; the last invariant is the solution we want to find, by combining the operations above in an arbitrary order. Let's translate these into functions as well:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"small_jug_invariant(j::Jugs) = 0 <= j.small <= 3\nlarge_jug_invariant(j::Jugs) = 0 <= j.large <= 5\nlevel_invariant(j::Jugs) = j.large != 4\ninvariants = (small_jug_invariant, large_jug_invariant, level_invariant)\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"And now, to finally combine all of these:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"gen_ops = Data.Vectors(Data.SampledFrom(raw_ops)) # hide\n# do a little dance so that this expected failure doesn't kill doc building # hide\ntry # hide\n@check function solve_die_hard(ops = gen_ops)\n jugs = Jugs()\n\n for op in ops\n # apply the rule\n jugs = op(jugs)\n\n # check invariants\n for f in invariants\n f(jugs) || return false\n end\n end\n\n return true\nend\ncatch # hide\nend # hide\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"This pattern is very extensible, and a good candidate for the next UX overhaul (getting a reported failure for the target we actually want to find is quite bad UX). Nevertheless, it already works right now!","category":"page"},{"location":"Examples/stateful.html#Balancing-a-heap","page":"Stateful Testing","title":"Balancing a heap","text":"","category":"section"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"The previous example showed how we can check these kinds of operations based invariants on an immutable struct. There is no reason why we can't do the same with a mutable struct (or at least, a struct containing a mutable object) though, so let's look at another example: ensuring a heap observes its heap property. As a quick reminder, the heap property for a binary heap is that each child of a node is <= than that node, resulting in what's called a \"Max-Heap\" (due to the maximum being at the root). Similarly, if the property for children is >=, we get a \"Min-Heap\". Here, we're going to implement a Min-Heap.","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"First, we need to define our datastructure:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"struct Heap{T}\n data::Vector{T}\nend\nHeap{T}() where T = Heap{T}(T[])","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"as well as the usual operations (isempty, push!, pop!) on that heap:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"isempty: Whether the heap has elements\npush!: Put an element onto the heap\npop!: Retrieve the smallest element of the heap (i.e., remove the root)","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Written in code, this might look like this:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Base.isempty(heap::Heap) = isempty(heap.data)\n\nfunction Base.push!(heap::Heap{T}, value::T) where T\n data = heap.data\n push!(data, value)\n index = lastindex(data)\n while index > firstindex(data)\n parent = index >> 1\n if data[parent] > data[index]\n data[parent], data[index] = data[index], data[parent]\n index = parent\n else\n break\n end\n end\n heap\nend\n\nBase.pop!(heap::Heap) = popfirst!(heap.data)","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"In this implementation, we're simply using an array as the backing store for our heap. The first element is the root, followed by the left subtree, followed by the right subtree. As implemented, pop! will return the correct element if the heap is currently balanced, but because pop! doesn't rebalance the heap after removing the root, pop! may leave it in an invalid state. A subsequent pop! may then remove an element that is not the smallest currently stored.","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"We can very easily test this manually:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"using Supposition\n\nintvec = Data.Vectors(Data.Integers{UInt8}())\n\ntry # hide\n@check function test_pop_in_sorted_order(ls=intvec)\n h = Heap{eltype(ls)}()\n\n # push all items\n for l in ls\n push!(h, l)\n end\n\n # pop! all items\n r = eltype(ls)[]\n while !isempty(h)\n push!(r, pop!(h))\n end\n\n # the pop!ed items should be sorted\n r == sort(ls)\nend\ncatch # hide\nend # hide\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"And as expected, the minimal counterexample is [0x0, 0x1, 0x0]. We first pop! 0x0, followed by 0x1 while it should be 0x0 again, and only then 0x1, resulting in [0x0, 0x0, 0x1] instead of [0x0, 0x1, 0x0].","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Replacing this with a (presumably) correct implementation looks like this:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"function fixed_pop!(h::Heap)\n isempty(h) && throw(ArgumentError(\"Heap is empty!\"))\n data = h.data\n isone(length(data)) && return popfirst!(data)\n result = first(data)\n data[1] = pop!(data)\n index = 0\n while (index * 2 + 1) < length(data)\n children = [ index*2+1, index*2+2 ]\n children = [ i for i in children if i < length(data) ]\n @assert !isempty(children)\n sort!(children; by=x -> data[x+1])\n broke = false\n for c in children\n if data[index+1] > data[c+1]\n data[index+1], data[c+1] = data[c+1], data[index+1]\n index = c\n broke = true\n break\n end\n end\n !broke && break\n end\n return result\nend","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Me telling you that this is correct though should only be taken as well-intentioned, but not necessarily as true. There might be more bugs that have sneaked in after all, that aren't caught by our naive \"pop in order and check that it's sorted\" test. There could be a nasty bug waiting for us that only happens when various push! and pop! are interwoven in just the right way. Using stateful testing techniques and the insight that we can generate sequences of operations on our Heap with Supposition.jl too! We're first going to try with the existing, known broken pop!:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"gen_push = map(Data.Integers{UInt}()) do i\n (push!, i)\nend\ngen_pop = Data.Just((pop!, nothing))\ngen_ops = Data.Vectors(Data.OneOf(gen_push, gen_pop); max_size=10_000)\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"We either push! an element, or we pop! from the heap. Using (pop!, nothing) here will make it a bit easier to actually define our test. Note how the second element acts as the eventual argument to pop!.","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"There's also an additional complication - because we don't have the guarantee anymore that the Heap contains elements, we have to guard the use of pop! behind a precondition check. In case the heap is empty, we can just consume the operation and treat it as a no-op, continuing with the next operation:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"# let's dance again, Documenter.jl! # hide\ntry # hide\n@check function test_heap(ops = gen_ops)\n heap = Heap{UInt}()\n\n for (op, val) in ops\n if op === push!\n # we can always push\n heap = op(heap, val)\n else\n # check our precondition!\n isempty(heap) && continue\n\n # the popped minimum must always == the minimum\n # of the backing array, so retrieve the minimum\n # through alternative internals\n correct = minimum(heap.data)\n val = op(heap)\n\n # there's only one invariant this time around\n # and it only needs checking in this branch:\n val != correct && return false\n end\n end\n\n # by default, we pass the test!\n # this happens if our `ops` is empty or all operations\n # worked successfully\n return true\nend\ncatch # hide\nend # hide\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Once again, we find our familiar example UInt[0x0, 0x1, 0x0], though this time in the form of operations done on the heap:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"ops = Union{Tuple{typeof(pop!), Nothing}, Tuple{typeof(push!), UInt64}}[\n (push!, 0x0000000000000001),\n (push!, 0x0000000000000000),\n (push!, 0x0000000000000000),\n (pop!, nothing),\n (pop!, nothing)\n]","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"We push three elements (0x1, 0x0 and 0x0) and when popping two, the second doesn't match the expected minimum anymore!","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Now let's try the same property with our (hopefully correct) fixed_pop!:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"gen_fixed_pop = Data.Just((fixed_pop!, nothing))\ngen_fixed_ops = Data.Vectors(Data.OneOf(gen_push, gen_fixed_pop); max_size=10_000)\n# Documenter shenanigans require me to repeat this. # hide\nfunction test_heap(ops) # hide\n heap = Heap{UInt}() # hide\n # hide\n for (op, val) in ops # hide\n if op === push! # hide\n # we can always push # hide\n heap = op(heap, val) # hide\n else # hide\n # check our precondition! # hide\n isempty(heap) && continue # hide\n # hide\n # the popped minimum must always == the minimum # hide\n # of the backing array, so retrieve the minimum # hide\n # through alternative internals # hide\n correct = minimum(heap.data) # hide\n val = op(heap) # hide\n # hide\n # there's only one invariant this time around # hide\n # and it only needs checking in this branch: # hide\n val != correct && return false # hide\n end # hide\n end # hide\n # hide\n # by default, we pass the test! # hide\n return true # hide\nend # hide\n\n@check test_heap(gen_fixed_ops)\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Now this is much more thorough testing!","category":"page"},{"location":"resources.html#PBT-Resources","page":"PBT Resources","title":"PBT Resources","text":"","category":"section"},{"location":"resources.html","page":"PBT Resources","title":"PBT Resources","text":"This page contains a collection of PBT tutorials and other useful resources for learning PBT techniques. Most, if not all, should be directly translatable to Supposition.jl in one form or another. If you find a new tutorial or resource that helped you test your code with Supposition.jl in some manner, please don't hesitate to open a PR adding the resource here!","category":"page"},{"location":"resources.html","page":"PBT Resources","title":"PBT Resources","text":"The purpose of Hypothesis by David R. MacIver\n[...], the larger purpose of Hypothesis is to drag the world kicking and screaming into a new and terrifying age of high quality software.\nHypothesis testing with Oracle functions by Hillel Wayne\nA blogpost about using existing (but slower/partially incorrect) implementations to make sure a refactored or new implementation still conforms to all expected contracts of the old implementation.\nSolving the Water Jug Problem from Die Hard 3 with TLA+ and Hypothesis by Nicholas Chammas\nA blogpost about helping out John McClane (Bruce Willis) and Zeus Carver (Samuel L. Jackson) ~defuse a bomb~ solve fun children's games.\nThis blogpost has been translated to Supposition.jl! Check it out in the examples.\nRule Based Stateful Testing by David R. MacIver\nA blogpost from the main developer behind Hypothesis, showing how to test stateful systems with Hypothesis.\nThis blogpost has been translated to Supposition.jl! Check it out in the examples.\nNote: Not all features of Hypothesis have been ported to Supposition.jl, in particular the UX for stateful testing is very bare bones. The linked example contains a very manual implementation of the features utilized by Hypothesis for much the same thing, but should be easily adaptable for all kinds of stateful tests.\nProprty Testing Stateful Code in Rust by Raphael Gashignard\nA blogpost about fuzzing internal datastructures of nushell using PBT and the Rust library proptest.\nAutomate Your Way to Better Code: Advanced Property Testing (with Oskar Wickström) by Kris Jenkins from Developer Voices\nMy Job as a programmer is to be lazy in the smart way - I see that many unit tests, and I just want to automate the problem away. Well that's the promise of property testing - write a bit of code that describes the shape of your software, and it will go away and create 10_000 unit tests to see if you're right, if it actually does work that way. [..] we're also going to address my biggest disappointment so far with property testing: which is that it only seems to work in theory. It's great for textbook examples, I'm sold on the principle, but I've struggled to make it work on my more gnarly real world code.\nThis is an absolutely delightful listen! A nice definition of what property based testing is, as well as a lot of discussion on how to start out with property based testing and continue with the approach onto more difficult pastures. Don't let yourself be intimidated by the length - take your time with this one, it's well worth it!\nThe Magic of Property Testing by Kris Jenkins from Developer Voices\nThis is a followup to \"Automate Your Way to Better Code\", showcasing an example of property based testing in PureScript. The fuzzing framework used here is a port of QuickCheck, but the general flow should be translatable to Supposition.jl. One feature being showcased (generation of objects through reflection) is not yet available in Supposition.jl; see this discussion for the state of things. Nevertheless, even without that, the generation capabilities of random data in Supposition.jl are just as powerful.","category":"page"},{"location":"Examples/recursive.html#Recursive-Generation","page":"Recursive Generation","title":"Recursive Generation","text":"","category":"section"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"In some situations, it is required to generate objects that can nest recursively. For example, JSON is an often used data exchange format that consists of various layers of dictionaries (with string keys) and one dimensional arrays, as well as strings, numbers, booleans an Nothing.","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"In Supposition.jl, we can generate these kinds of recursively nested objects using the Data.Recursive Possibility. For this, we need a generator of a basecase, as well as a function that wraps one generated example in a new layer by returning a new Possibility.","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"We can construct the Possibility that generates the basecase like so:","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"using Supposition\n\nstrings = Data.Text(Data.AsciiCharacters())\nstrings = Data.Text(Data.AsciiCharacters(); max_len=10) # hide\nbools = Data.Booleans()\nnone = Data.Just(nothing)\nnumbers = Data.Floats{Float64}()\nbasecase = strings | numbers | bools | none","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"which gives us a Data.OneOf, a Possibility that can generate any one of the objects generated by the given Possibility.","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"For wrapping into new layers, we need a function that wraps our basecase Possibility and gives us a new Possibility generating the wrapped objects. For the JSON example, this means we can wrap an object either in a Vector, or a Dict, where the latter has String keys.","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"note: Wrapping order\nRecursive expects a function that takes a Possibility for generating the children of the wrapper object, which you should pass into a generator. The generator for the wrapper can be any arbitrary Possibility.","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"Defining that function like so:","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"function jsonwrap(child)\n vecs = Data.Vectors(child)\n dicts = Data.Dicts(strings, child)\n vecs = Data.Vectors(child; max_size=5) # hide\n dicts = Data.Dicts(strings, child; max_size=5) # hide\n vecs | dicts\nend","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"allows us to construct the Possibility for generating nested JSON-like objects:","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"json = Data.Recursive(basecase, jsonwrap; max_layers=3)\nexample(json)\n# a little bit of trickery, to show a nice example # hide\nprintln( # hide\n\"Dict{String, Union{Nothing, Bool, Float64, Dict{String, Union{Nothing, Bool, Float64, String}}, String, Vector{Union{Nothing, Bool, Float64, String}}}} with 5 entries:\" * # hide\n\"\\n \\\"!\\\" => -1.58772e111\" * # hide\n\"\\n \\\"\\\\e^Y\\\\x1cq\\\\bEj8\\\" => -4.31286e-135\" * # hide\n\"\\n \\\"^\\\" => Union{Nothing, Bool, Float64, String}[false]\" * # hide\n\"\\n \\\"\\\\x0f \\\\t;lgC\\\\e\\\\x15\\\" => nothing\" * # hide\n\"\\n \\\"Y266uYkn6\\\" => -5.68895e-145\" # hide\n) # hide\nnothing # hide","category":"page"},{"location":"index.html#Supposition.jl-Documentation","page":"Main Page","title":"Supposition.jl Documentation","text":"","category":"section"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"This is the documentation for Supposition.jl, a property based testing framework inspired by Hypothesis.","category":"page"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"It features choice-sequence based generation & shrinking of examples, which can smartly shrink initial failures to smaller example while preserving the invariants the original input was generated under. It's also easy to combine generators into new generators.","category":"page"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"Check out the Examples in the sidebar to get an introduction to property based testing and to learn how to write your own tests!","category":"page"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"Here's also a sitemap for the rest of the documentation:","category":"page"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"Pages = [ \"index.md\", \"intro.md\", \"faq.md\", \"interfaces.md\", \"api.md\" ]\nDepth = 3","category":"page"},{"location":"index.html#Goals","page":"Main Page","title":"Goals","text":"","category":"section"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"Good performance\nA test framework should not be the bottleneck of the testsuite.\nComposability\nIt should not be required to modify an existing codebase to accomodate Supposition.jl\nHowever, for exploratory fuzzing it may be advantageous to insert small markers into a codebase\nReusability\nIt should be easily possible to reuse large parts of existing definitions (functions/structs) to build custom generators\nRepeatability\nIt should be possible to replay previous (failing) examples and reproduce the sequence of steps taken exactly. The only cases where this isn't going to work is if your code relies on external state, such as querying a hardware RNG for random data or similar objects that are not under the control of the testing framework itself (such as the capacity of your harddrive, for example).\nDiscoverability (of API boundaries)\nSupposition.jl should be easy to use to find the actual API boundaries of a given function, if that is not yet known or not sufficiently specified in the docstring of a function. It should be enough to know the argument types to start fuzzing a function (at least in the simplest sense of \"does it error\").\nEase of Use\nIt should be relatively straightforward to write custom generators.","category":"page"},{"location":"index.html#Limitations","page":"Main Page","title":"Limitations","text":"","category":"section"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"Due to its nature as a fuzzing framework and the (usually) huge associated statespace, Supposition.jl cannot give a formal proof of correctness. It's only an indicator (but a pretty good one).","category":"page"}]
+[{"location":"Examples/docalignment.html#Aligning-Behavior-and-Documentation","page":"Alignment of Documentation","title":"Aligning Behavior & Documentation","text":"","category":"section"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"When it comes to property based testing, the question \"but how/what should I start testing?\" quickly arises. After all, a documented & tested code base should already have matching documentation & implementation!","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"In reality, this is often not the case. Docstrings can bitrot when performance optimizations or bugfixes subtly change the semantics of a function, generic functions often can't easily write out all conditions a passed-in function must follow and large code bases are often so overwhelmingly full of possible invocations that the task to check for conformance with each and every single docstring can be much too daunting, to say the least.","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"There is a silver lining though - there are a number of simple checks a developer can use to directly & measurably improve not only the docstring of a function, but code coverage of a testsuite.","category":"page"},{"location":"Examples/docalignment.html#Does-it-error?","page":"Alignment of Documentation","title":"Does it error?","text":"","category":"section"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"The simplest property one can test is very straightforward: On any given input, does the function error? In theory, this is something that any docstring should be able to precisely answer - under which conditions is the function expected to error? The use of Supposition.jl helps in confirming that the function only errors under those conditions.","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"Take for example this function and its associated docstring:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"\"\"\"\n sincosd(x)\n\nSimultaneously compute the sine and cosine of x, where x is in degrees.\n\n!!! compat \"Julia 1.3\"\n This function requires at least Julia 1.3.\n\"\"\"\nsincosd","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"sincosd is a function from Base Julia - I don't mean to pick on it, but it serves as a good example (and I also have an open PR to improve this very docstring!).","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"As written, we don't really know a whole lot about sincosd, other than that it computes both sin and cosin in some simultaneous fashion and interprets whatever we give it in degrees. In particular, there is no mention of what types and instances of those types are expected to work at all, and no indication of when or even if the function can throw an error. Nevertheless, it does:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"julia> sincosd(Inf)\nERROR: DomainError with Inf:\n`x` cannot be infinite.\nStacktrace:\n [1] sind(x::Float64)\n @ Base.Math ./special/trig.jl:1185\n [2] sincosd(x::Float64)\n @ Base.Math ./special/trig.jl:1250\n [3] top-level scope\n @ REPL[3]:1\n\njulia> sincosd(\"Inf\")\nERROR: MethodError: no method matching deg2rad(::String)\n\nClosest candidates are:\n deg2rad(::AbstractFloat)\n @ Base math.jl:346\n deg2rad(::Real)\n @ Base math.jl:348\n deg2rad(::Number)\n @ Base math.jl:350\n\nStacktrace:\n [...]","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"So at the very least, due to its dependence on deg2rad, sincosd expects a \"number-like\" object, not a String. At the same time, not all \"number-like\" objects are accepted either. It may be obvious to a mathematician or to a developer who regularly uses sin that Inf can't possibly be a valid input, but that is not necessarily obvious to someone just starting out with using trigonometric functions. It may be entirely reasonable that sin returns NaN on an Inf input, to signal to a user that there is no number that can represent the result of that call! For a developer, that is a huge difference in error checking behavior - one case can be a quick if isnan(sin(z)) check, while the other requires a costly try/catch. For this reason, it's very important to be accurate in documentation about what kinds of inputs are valid and clearly specify what happens when & why things go wrong, and not just think of the \"happy path\".","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"So an improved version of this docstring might look like so:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":" \"\"\"\n sincosd(x::Number)\n\n Simultaneously compute the sine and cosine of x, where x is in degrees.\n+\n+ Throws a `DomainError` if `isinf(x)` is `true`.\n\n !!! compat \"Julia 1.3\"\n This function requires at least Julia 1.3.\n \"\"\"\n sincosd","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"By adding the ::Number type restriction to the argument x, we clearly communicate what kind of object we expect to work on sincosd. If other user-defined types have their own implementation of sincosd, they should have their own docstrings specifying the peculiarities of their implementation on their own types.","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"We've checked some simple examples through knowing special values of Float64, but are there any other special values we should know about? Let's use Supposition.jl to define the simplest possible test that tries all kinds of different Float64 inputs:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"using Supposition\n\n@check function sincosd_float64(f=Data.Floats{Float64}())\n try\n return sincosd(f) isa Tuple{Float64, Float64}\n catch e\n e isa DomainError && isinf(f) && return true\n rethrow()\n end\nend\nnothing # hide","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"We first define our data generator, in this case sampling from all possible Float64 values (including all NaNs and Infs!). For each value, we try to call sincosd and check whether it returns two Float64 values. In case an error occurs, we can check whether the error is a DomainError and the input was an infinity. If so, the property holds; if not, we simply rethrow the error. Supposition.jl will take the thrown error as a signal that something has gone wrong, and try to shrink the input to the minimal example that reproduces the same error.","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"Looking at that test, there's another assumption that we should document: sincosd returns a tuple! The companion function sincos (differing from sincosd insofar as it takes its argument in radians, not degrees) does document this, so we should match that documentation here too:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":" \"\"\"\n sincosd(x::Number)\n\n- Simultaneously compute the sine and cosine of x, where x is in degrees.\n+ Simultaneously compute the sine and cosine of x, where x is in degrees,\n+ returning a tuple (sine, cosine).\n\n Throws a `DomainError` if `isinf(x)` is `true`.\n\n !!! compat \"Julia 1.3\"\n This function requires at least Julia 1.3.\n \"\"\"\n sincosd","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"This is especially important because of the order the sine and cosine are returned in. If this isn't documented, users can only assume which returned number is which trigonometric result, and without actually checking, they have no way to confirm that behavior, which we will get to in the next subchapter.","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"Now that we've followed an example, let's reflect on what even simple \"does it error?\" style tests can do. At the end of this process, a developer should now have","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"a clear understanding of when the tested function errors,\nknowledge about the requirements a function has so that it can be called without error, ready to be added to the documentation of the function,\nready-made tests that can be integrated into a testsuite running in CI,\npotentially found & fixed (or tracked on an issue tracker) a few bugs that were found during testing.","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"all of which should help a user make an informed choice in how they use e.g. sincosd, as well as inform a developer about a deviation in expected behavior.","category":"page"},{"location":"Examples/docalignment.html#Docstring-guarantees-of-single-functions","page":"Alignment of Documentation","title":"Docstring guarantees of single functions","text":"","category":"section"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"The next level up from error checks is checking requirements & guarantees on valid input, i.e. the input that is not expected to error but must nonetheless conform to some specification. Once we can generate such a valid input, we can check that the output of the function actually behaves as we expect it to.","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"Continuing on from the sincosd example from above, let's start out with a generator for all non-throwing Float64. Since there are just a few of these, we can filter them out easily, without having to worry about rejecting too many samples:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"using Supposition\n\nnon_throw_sincos = filter(!isinf, Data.Floats{Float64}())\nprint(non_throw_sincos) # hide\nnon_throw_sincos = Data.Just(NaN) # hide\nnothing # hide","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"Now let's think about what we'd like sin and cos to obey. For starters, we could use some mathematical identities:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"@check function pythagorean_identity(degrees=non_throw_sincos)\n s, c = sincosd(degrees)\n (s^2 + c^2) == one(s)\nend\nnothing # hide","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"Right away, we can find a very easy counterexample - NaN! Not to worry, we can simply amend the docstring to mention this too:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":" \"\"\"\n sincosd(x::Number)\n\n Simultaneously compute the sine and cosine of x, where x is in degrees,\n returning a tuple (sine, cosine).\n\n Throws a `DomainError` if `isinf(x)` is `true`.\n+\n+ If `isnan(x)`, return a 2-tuple of `NaN` of type `typeof(x)`.\n\n !!! compat \"Julia 1.3\"\n This function requires at least Julia 1.3.\n \"\"\"\n sincosd","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"and try again, this time with NaN values filtered out too:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"pure_float = filter(Data.Floats{Float64}()) do f\n !(isinf(f) || isnan(f))\nend\nfunction pythagorean_identity(degrees) # hide\n s, c = sincosd(degrees) # hide\n (s^2 + c^2) ≈ one(s) # hide\nend # hide\n@check pythagorean_identity(pure_float)\nnothing # hide","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"The property holds! Very nice. What other properties do we have? Wikipedia maintains a list, so we just have to pick and choose some that are to our liking.","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"For example, there's these:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"sin(alpha pm beta) = sin(alpha)cos(beta) pm cos(alpha)sin(beta) \ncos(alpha pm beta) = cos(alpha)cos(beta) pm sin(alpha)sin(beta) \nsin(2theta) = 2sin(theta)cos(theta) = (sin(theta) + cos(theta))^2 \ncos(2theta) = cos^2(theta) - sin^2(theta) = 2cos^2(theta)-1 = 1-2sin^2(theta)","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"and lots more - there's just one problem: due to floating point addition not being associative, almost none of these are numerically stable:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"using Test\n\ntry # hide\n@testset \"offset identity\" begin\n @check function sin_offset(a=pure_float, b=pure_float)\n sin_a, cos_a = sincosd(a)\n sin_b, cos_b = sincosd(b)\n sind(a+b) == sin_a*cos_b + cos_a*sin_b\n end\nend\ncatch # hide\nend # hide\nnothing # hide","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"We have to make do with a subset of all properties then, such as \"the output of sincos should == the outputs of sin and cos on their own\", or properties that avoid addition & subtraction.","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"@testset \"sincos properties\" begin\n@check function sincos_same(theta=pure_float)\n s, c = sincosd(theta)\n s == sind(theta) && c == cosd(theta)\nend\n@check function twice_sin(theta=pure_float)\n s, c = sincosd(theta)\n twice_theta = 2*theta\n assume!(!isinf(twice_theta))\n isapprox(sind(twice_theta), 2*s*c)\nend\nend\nnothing # hide","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"These somewhat milder properties seem to pass, nice!","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"Note how we have to make use of assume! to not have sin error out with a DomainError too! This indicates that there are situations where sincos & the longer form of sin is preferable to the real thing. I'm unsure whether that should be noted in a docstring of sincos (this seems more appropriate for a computer numerics course), but it's a good example of how a developer can learn about the properties & tradeoffs a function can have compared to its counterparts. Perhaps a warning like the following would be best:","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":" \"\"\"\n sincosd(x::Number)\n\n Simultaneously compute the sine and cosine of x, where x is in degrees,\n returning a tuple (sine, cosine).\n\n Throws a `DomainError` if `isinf(x)` is `true`.\n\n If `isnan(x)`, return a 2-tuple of `NaN` of type `typeof(x)`.\n+\n+ !!! warning \"Numerical Stability\"\n+ Due to floating point addition not being associative, not all\n+ trigonometric identies can hold for all inputs. Choose carefully\n+ and consider the operation you're doing when using trigonometric\n+ identities to transform your code.\n\n !!! compat \"Julia 1.3\"\n This function requires at least Julia 1.3.\n \"\"\"\n sincosd","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"We've now not only documented the erroring behavior of sincosd, but also special values and their special behavior. In the process we've also found that for this particular function, not everything we might expect actually can hold true in general - and documenting that has good chances to be a helpful improvement for anyone stumbling over trigonometric functions for the first time (in the long run, most developers are newbies; experts are rare!)","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"At the end of this process, a developer should now have","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"a better understanding of the guarantees a function gives,\nsome behavioral tests of a function, ready to be integrated into a testsuite running in CI,\na clear understanding that care must be taken when talking about what a computer ought to compute \"correctly\".","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"Of course, there could be numerous other properties we'd like to test. For example, we may want to confirm that the output of sincos is always a 2-Tuple, or that if the input was non-NaN, the output values are in the closed interval [-1, 1], or that the output values are evenly distributed in that interval (up to a point - this is surprisingly difficult to do for large inputs!)","category":"page"},{"location":"Examples/docalignment.html#Interactions-between-functions","page":"Alignment of Documentation","title":"Interactions between functions","text":"","category":"section"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"Once the general knowledge about single functions has been expanded and appropriately documented, it's time to ask the bigger question - how do these functions interact, and do they interact in a way that a developer would expect them to? Since this is more involved, I've dedicated an entire section to this: Stateful Testing.","category":"page"},{"location":"Examples/docalignment.html","page":"Alignment of Documentation","title":"Alignment of Documentation","text":"These three sections present a natural progression from writing your first test with Supposition.jl, over documenting guarantees of single functions to finally investigating interactions between functions and the effect they have on the datastructures of day-to-day use.","category":"page"},{"location":"Examples/events.html#Events-and-Oracle-testing","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"","category":"section"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"One of the most useful applications of Supposition.jl is to assist developers when porting code from another language to Julia, while trying to optimize that same code a bit. In these cases, it can be extremely useful to compare the output of the port with the output of the original implementation (which is usually called a test oracle).","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"A typical fuzzing test making use of an oracle looks like this:","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"@check function checkOracle(input=...)\n new_res = optimized_func(input)\n old_res = old_func(input)\n new_res == old_res\nend","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"which generates some valid input and simply compares whether the new implementation produces the same output as the old implementation.","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"In order not to get bogged down in the abstract, let's look at a concrete example of how this can be used in practice. The following example is inspired by the work @tecosaur has recently done in StyledStrings.jl, huge shoutout for giving permission to use it as an example!","category":"page"},{"location":"Examples/events.html#Porting-an-example","page":"Events & Oracle testing","title":"Porting an example","text":"","category":"section"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"The task is seemingly simple - map an NTuple{3, UInt8} representing an RGB color to an 8-bit colour space, via some imperfect mapping.","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"The original function we're trying to port here is the following, from tmux:","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"/*\n * Copyright (c) 2008 Nicholas Marriott \n * Copyright (c) 2016 Avi Halachmi \n *\n * Permission to use, copy, modify, and distribute this software for any\n * purpose with or without fee is hereby granted, provided that the above\n * copyright notice and this permission notice appear in all copies.\n *\n * THE SOFTWARE IS PROVIDED \"AS IS\" AND THE AUTHOR DISCLAIMS ALL WARRANTIES\n * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF\n * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR\n * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES\n * WHATSOEVER RESULTING FROM LOSS OF MIND, USE, DATA OR PROFITS, WHETHER\n * IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING\n * OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.\n */\n\n#include \n\nstatic int\ncolour_dist_sq(int R, int G, int B, int r, int g, int b)\n{\n\treturn ((R - r) * (R - r) + (G - g) * (G - g) + (B - b) * (B - b));\n}\n\nstatic int\ncolour_to_6cube(int v)\n{\n\tif (v < 48)\n\t\treturn (0);\n\tif (v < 114)\n\t\treturn (1);\n\treturn ((v - 35) / 40);\n}\n\nint\ncolour_find_rgb(u_char r, u_char g, u_char b)\n{\n\tstatic const int\tq2c[6] = { 0x00, 0x5f, 0x87, 0xaf, 0xd7, 0xff };\n\tint\t\t\tqr, qg, qb, cr, cg, cb, d, idx;\n\tint\t\t\tgrey_avg, grey_idx, grey;\n\n\t/* Map RGB to 6x6x6 cube. */\n\tqr = colour_to_6cube(r); cr = q2c[qr];\n\tqg = colour_to_6cube(g); cg = q2c[qg];\n\tqb = colour_to_6cube(b); cb = q2c[qb];\n\n\t/* If we have hit the colour exactly, return early. */\n\tif (cr == r && cg == g && cb == b)\n\t\treturn (16 + (36 * qr) + (6 * qg) + qb);\n\n\t/* Work out the closest grey (average of RGB). */\n\tgrey_avg = (r + g + b) / 3;\n\tif (grey_avg > 238)\n\t\tgrey_idx = 23;\n\telse\n\t\tgrey_idx = (grey_avg - 3) / 10;\n\tgrey = 8 + (10 * grey_idx);\n\n\t/* Is grey or 6x6x6 colour closest? */\n\td = colour_dist_sq(cr, cg, cb, r, g, b);\n\tif (colour_dist_sq(grey, grey, grey, r, g, b) < d)\n\t\tidx = 232 + grey_idx;\n\telse\n\t\tidx = 16 + (36 * qr) + (6 * qg) + qb;\n\treturn idx;\n}","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"note: Modifications\nI've modified the code slightly, since we don't really care for the flags tmux is storing in the upper bits of the result. The transform would be simple enough to undo when fuzzing/using this as an oracle, but would mostly just distract from the core of what I'm trying to show here.","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"That is quite a handful of code! There's lots of magic numbers, two basically-one-liner helper functions and very specific input types, the behavior of which must all be taken into account.","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"One attempt at a port to Julia might look like this (courtesy of @tecosaur):","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"function termcolor8bit(r::UInt8, g::UInt8, b::UInt8)\n # Magic numbers? Lots.\n cdistsq(r1, g1, b1) = (r1 - r)^2 + (g1 - g)^2 + (b1 - b)^2\n to6cube(value) = (value - 35) ÷ 40\n from6cube(r6, g6, b6) = 16 + 6^2 * r6 + 6^1 * g6 + 6^0 * b6\n sixcube = (0, 95:40:255...)\n r6cube, g6cube, b6cube = to6cube(r), to6cube(g), to6cube(b)\n rnear, gnear, bnear = sixcube[r6cube+1], sixcube[g6cube+1], sixcube[b6cube+1]\n colorcode = if r == rnear && g == gnear && b == bnear\n from6cube(r6cube, g6cube, b6cube)\n else\n grey_avg = Int(r + g + b) ÷ 3\n grey_index = if grey_avg > 238 23 else (grey_avg - 3) ÷ 10 end\n grey = 8 + 10 * grey_index\n if cdistsq(grey, grey, grey) <= cdistsq(rnear, gnear, bnear)\n 16 + 6^3 + grey_index\n else\n from6cube(r6cube, g6cube, b6cube)\n end\n end\n UInt8(colorcode)\nend","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"The basic structure is the same, albeit formatted a bit differently and with inner helper functions instead of outer ones. There are also some additional subtleties, such as the +1 that are necessary when indexing into sixcube and the explicit call to construct an Int for the average.","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"So without further ado, let's compare the two implementations! We can compile the C code with gcc -shared colors.c -o colors.o and wrap the resulting shared object from Julia like so:","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"# `colors.o` is stored in a subdirectory relative to this file!\nso_path = joinpath(@__DIR__, \"tmux_colors\", \"colors.o\")\ntmux_8bit_oracle(r::UInt8, g::UInt8, b::UInt8) = @ccall so_path.colour_find_rgb(r::UInt8, g::UInt8, b::UInt8)::UInt8","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"Great, this now allows us to call the oracle function and compare to our ported implementation:","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"tmux_8bit_oracle(0x4, 0x2, 0x3)\ntermcolor8bit(0x4, 0x2, 0x3)","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"And at least for this initial example, the 8-bit colorcodes we get out are the same.","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"But are they the same for all colors? You're reading the docs of Supposition.jl, so let's @check that out, following the pattern for oracle tests we saw earlier:","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"using Supposition\nuint8gen = Data.Integers{UInt8}()\n@check function oracle_8bit(r=uint8gen,g=uint8gen,b=uint8gen)\n jl = termcolor8bit(r,g,b)\n tmux = tmux_8bit_oracle(r,g,b)\n jl == tmux\nend\nnothing # hide","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"and indeed, we find a counterexample where our port doesn't match the original code for some reason. It would be really interesting to see what exactly these two functions put out, so that it's easier to reconstruct where the port has gone wrong (or perhaps even the oracle!). Now, while we could go in and add @info or manually call termcolor8bit and tmux_8bit_oracl with the minimized input, neither of those is really attractive, for multiple reasons:","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"@info will output logs for every invocation of our property, which is potentially quite a lot. That's a lot of text to scroll back through, both locally and in CI. In the worst case, we might even hit the scrollback limit of our terminal, or fill the disk on a resource-limited CI.\nCalling the functions manually can quickly get cumbersome, since you have to copy the output of @check, make sure you format it correctly to call the various functions etc. The interfaces between the two functionalities may be entirely different too in a more complex example than this one, adding more complexity that a developer has to keep in mind.","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"No, \"manual\" checking & reading logs is certainly not in the spirit of Supposition.jl! Indeed, there's a feature we can make use of here that gives us exactly the information we wanted to have in the first place - event!. We simply insert calls to it into our property, and it records any events we consider to be interesting enough to record. We can also optionally give these events a label, though that's only used for display purposes:","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"@check function oracle_8bit(r=uint8gen,g=uint8gen,b=uint8gen)\n jl = termcolor8bit(r,g,b)\n event!(\"Port\", jl)\n tmux = tmux_8bit_oracle(r,g,b)\n event!(\"Oracle\", tmux)\n jl == tmux\nend\nnothing # hide","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"The first argument in the 2-arg form of event! is any AbstractString, so even the (upcoming with Julia 1.11) AnnotatedStrings from the styled\"\" macro are an option, for fancy labelling. The recorded object can be arbitrary, but be aware that it will be kept alive for the entire duration of the testsuite, so it's better to record small objects. Only the events associated with the \"most important\" test case encountered during fuzzing will be kept alive; if a better one comes around, the existing events are deleted.","category":"page"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"The printing is currently quite ugly, but I do expect that to improve with #23 once StyledStrings.jl is released ;)","category":"page"},{"location":"Examples/events.html#When-to-use-this","page":"Events & Oracle testing","title":"When to use this","text":"","category":"section"},{"location":"Examples/events.html","page":"Events & Oracle testing","title":"Events & Oracle testing","text":"Events are a great way to diagnose & trace how a minimal input affects deeper parts of your code, in particular when you have a failing minimal example that you can't quite seem to get a handle on when debugging. In those cases, you can add event! calls into your code base and check the resulting trace for clues what might be going wrong.","category":"page"},{"location":"Examples/basic.html#Basic-Usage","page":"Basic Usage","title":"Basic Usage","text":"","category":"section"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"At its core, property based testing (PBT) is about having a function (or set of functions) to test and a set of properties that should hold on that function. If you're already familiar with PBT, this basic example will be familiar to you already.","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"Consider this add function, which simply forwards to +:","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"function add(a,b)\n a + b\nend","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"How can we test that this function truly is the same as +? First, we have to decide what input we want to test with. In Supposition.jl, this is done through the use of Possibilitiy objects, which represent an entire set of objects of a shared type. In other frameworks like Hypothesis, this is known as a strategy. In our case, we are mostly interested in integers, so the generator Data.Integers{UInt} is what we're going to use:","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"using Supposition, Supposition.Data\n\nintgen = Data.Integers{UInt}()","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"Now that we have our input generator, we have to decide on the properties we want to enforce. Here, we simply want to check the mathematical properties of addition, so let's start with commutativity:","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"Supposition.@check function commutative(a=intgen, b=intgen)\n add(a,b) == add(b,a)\nend\nnothing # hide","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"@check takes a function definition where each argument is given a Possibility, runs those generators, feeds the generated values into the given function and shrinks any failing examples. Note that the name given in the arguments is the same as used in the function.","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"Here's an example for a failing property:","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"try # hide\nSupposition.@check function failprop(x=intgen)\n add(x, one(x)) < x\nend\ncatch # hide\nend # hide\nnothing # hide","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"Supposition.jl successfully found a counterexample and reduced it to a more minimal counterexample, in this case just UInt(0).","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"note: Overflow\nThere is a subtle bug here - if x+1 overflows when x == typemax(UInt), the resulting comparison is true: typemin(UInt) < typemax(UInt) after all. It's important to keep these kinds of subtleties, as well as the invariants the datatype guarantees, in mind when choosing a generator and writing properties to check the datatype and its functions for.","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"We've still got three more properties to test, taking two or three arguments each. Since these properties are fairly universal, we can also write them out like so, passing a function of interest:","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"associative(f, a, b, c) = f(f(a,b), c) == f(a, f(b,c))\nidentity_add(f, a) = f(a,zero(a)) == a\nfunction successor(a, b)\n a,b = minmax(a,b)\n sumres = a\n for _ in one(b):b\n sumres = add(sumres, one(b))\n end\n\n sumres == add(a, b)\nend","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"And check that they hold like so. Of course, we can also test the property implicitly defined by @check earlier:","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"using Test\n\nSupposition.@check associative(Data.Just(add), intgen, intgen, intgen)\nSupposition.@check identity_add(Data.Just(add), intgen)\nSupposition.@check successor(intgen, intgen)\nSupposition.@check commutative(intgen, intgen)\nnothing # hide","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"In this way, we can even reuse properties from other invocations of @check with new, perhaps more specialized, inputs. For generalization, we can use Data.Just to pass our add function to the generalized properties.","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"note: Nesting @testset\nFrom Julia 1.11 onwards, @check can also report its own results as part of a parent @testset. This is unfortunately unsupported on 1.10 and earlier.","category":"page"},{"location":"Examples/basic.html","page":"Basic Usage","title":"Basic Usage","text":"Be aware that while all checks pass, we do not have a guarantee that our code is correct for all cases. Sampling elements to test is a statistical process and as such we can only gain confidence that our code is correct. You may view this in the light of Bayesian statistics, where we update our prior that the code is correct as we run our testsuite more often. This is also true were we not using property based testing or Supposition.jl at all - with traditional testing approaches, only the values we've actually run the code with can be said to be tested.","category":"page"},{"location":"Examples/target.html#Targeting","page":"Targeted Operation","title":"Targeting","text":"","category":"section"},{"location":"Examples/target.html","page":"Targeted Operation","title":"Targeted Operation","text":"As fuzzing targets become larger and the total state space of possible inputs becomes sparser in the number of possible counterexamples, it can become rarer to encounter randomized test failures. It can be challenging to design fuzzing inputs in just the right way to still get good results; Supposition.jl however has a tool that can make the process of finding inputs that are more likely to be useful for the property at hand easier.","category":"page"},{"location":"Examples/target.html","page":"Targeted Operation","title":"Targeted Operation","text":"Consider this example, where we assert that no matter what number we generate, it won't be equal to some other number we picked at random:","category":"page"},{"location":"Examples/target.html","page":"Targeted Operation","title":"Targeted Operation","text":"using Supposition, Random, Logging\n\nrand_goal = rand()\n\n@info \"Our random goal is:\" Goal=rand_goal\n\n# The RNG is fixed for doc-building purposes - you can try to reproduce\n# this example with any RNG you'd like!\n@check rng=Xoshiro(1) function israndgoal(f=Data.Floats{Float64}())\n f != rand_goal\nend\nnothing # hide","category":"page"},{"location":"Examples/target.html","page":"Targeted Operation","title":"Targeted Operation","text":"The default for the number of attempts @check tries to feed to israndgoal is 10_000; the test still passes, which means Supposition.jl was unable to find the number we claim can't be generated. We can increase this by an almost arbitrary amount, without having the test fail:","category":"page"},{"location":"Examples/target.html","page":"Targeted Operation","title":"Targeted Operation","text":"@check max_examples=1_000_000 rng=Xoshiro(1) israndgoal(Data.Floats{Float64}())\nnothing # hide","category":"page"},{"location":"Examples/target.html","page":"Targeted Operation","title":"Targeted Operation","text":"Clearly, there needs to be something done so that we can hint to Supposition.jl what we consider to be \"better\" inputs, so that Supposition.jl can focus on them. This functionality is target!. target! takes a number and records it as the score for the given generated inputs. During the generation phase, Supposition.jl tracks which example was considered \"the best\", i.e. which had the highest score, and subsequently attempts to find further examples that further increase this score, hopefully finding a maximum. For our example here, we can simply use the absolute distance from our input to the artificial goal as a score:","category":"page"},{"location":"Examples/target.html","page":"Targeted Operation","title":"Targeted Operation","text":"sr = @check rng=Xoshiro(1) function israndgoal(f=Data.Floats{Float64}())\n # negative absolute distance, because we want to _minimize_ the distance\n target!(-abs(rand_goal - f))\n f != rand_goal\nend\nnothing # hide","category":"page"},{"location":"Examples/target.html","page":"Targeted Operation","title":"Targeted Operation","text":"which results in Supposition.jl finding the sole counterexample in a comparatively very small number of inputs:","category":"page"},{"location":"Examples/target.html","page":"Targeted Operation","title":"Targeted Operation","text":"Supposition.num_testcases(sr)","category":"page"},{"location":"Examples/target.html","page":"Targeted Operation","title":"Targeted Operation","text":"In more complex situations where you don't have a very clear goal to minimize or maximize, target! can be very useful as a guiding force, as long as the metric you're using is good. I don't have a proof for it, but in general, you'll probably want the metric to be admissible.","category":"page"},{"location":"benchmarks.html#Benchmarks","page":"Benchmarks","title":"Benchmarks","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Since Julia developers can sometimes go crazy for performance and because PropCheck.jl already had a bunch of optimizations to (or try to, as we'll see) make it go fast, let's compare it to Supposition.jl to see how the two stack up against each other. Since both packages have been written by the same author, I think I'm in the clear and won't step on anyones feet :)","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"All benchmarks were run on the same machine, with the same Julia version:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> versioninfo()\nJulia Version 1.12.0-DEV.89\nCommit 35cb8a556b (2024-02-27 06:12 UTC)\nPlatform Info:\n OS: Linux (x86_64-pc-linux-gnu)\n CPU: 24 × AMD Ryzen 9 7900X 12-Core Processor\n WORD_SIZE: 64\n LLVM: libLLVM-16.0.6 (ORCJIT, znver4)\nThreads: 23 default, 1 interactive, 11 GC (on 24 virtual cores)\nEnvironment:\n JULIA_PKG_USE_CLI_GIT = true","category":"page"},{"location":"benchmarks.html#Generation","page":"Benchmarks","title":"Generation","text":"","category":"section"},{"location":"benchmarks.html#Integers","page":"Benchmarks","title":"Integers","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"The task is simple - generating a single Vector{Int} with 1_000_000 elements, through the respective interface of each package.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"First, PropCheck.jl:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> using BenchmarkTools\n\njulia> using PropCheck\n\njulia> intgen = PropCheck.vector(PropCheck.iconst(1_000_000), itype(Int));\n\njulia> @benchmark root(PropCheck.generate(intgen))\nBenchmarkTools.Trial: 1 sample with 1 evaluation.\n Single result which took 5.780 s (30.71% GC) to evaluate,\n with a memory estimate of 9.17 GiB, over 27285108 allocations.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"And now, Supposition:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> using BenchmarkTools\n\njulia> using Supposition\n\njulia> intgen = Data.Vectors(Data.Integers{Int}(); min_size=1_000_000, max_size=1_000_000);\n\njulia> @benchmark example($intgen)\nBenchmarkTools.Trial: 646 samples with 1 evaluation.\n Range (min … max): 5.556 ms … 24.662 ms ┊ GC (min … max): 0.00% … 72.10%\n Time (median): 6.344 ms ┊ GC (median): 4.18%\n Time (mean ± σ): 7.734 ms ± 4.033 ms ┊ GC (mean ± σ): 19.81% ± 19.08%\n\n █▇▅▄▅▅▅▂\n █████████▆▅▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▅█▇▆█▆▄▁▁▁▄▅▁▆▆▆▆▆▄▄▆ ▇\n 5.56 ms Histogram: log(frequency) by time 22.7 ms <\n\n Memory estimate: 25.04 MiB, allocs estimate: 34.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"GC percentage is about the same, but the used memory and total number of allocations are VASTLY in favor of Supposition.jl, by about a factor of ~1000 timewise and a factor 300 memorywise.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"To put this into perspective, here's a benchmark of just 1_000_000 Int randomly generated:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> @benchmark rand(Int, 1_000_000)\nBenchmarkTools.Trial: 10000 samples with 1 evaluation.\n Range (min … max): 182.570 μs … 11.340 ms ┊ GC (min … max): 0.00% … 96.50%\n Time (median): 311.934 μs ┊ GC (median): 0.00%\n Time (mean ± σ): 391.653 μs ± 244.852 μs ┊ GC (mean ± σ): 11.15% ± 15.24%\n\n ▆██▆▆▆▅▄▃▂▁▁▁▁ ▁▁▂▂▂▁▁ ▂\n ▄▆▇████████████████████▇▇▇▇▆▅▇█████████▇▇▇█▇▇▆▅▅▄▄▄▄▃▄▅▄▃▄▄▄▅ █\n 183 μs Histogram: log(frequency) by time 1.33 ms <\n\n Memory estimate: 7.63 MiB, allocs estimate: 3.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"So Supposition.jl is within 300x of just generating some random numbers, suggesting there's still room for improvement.","category":"page"},{"location":"benchmarks.html#Floats","page":"Benchmarks","title":"Floats","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"This is basically the same task as with Int, just producing 1_000_000 Float64 instead.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"We'll start with PropCheck.jl again:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> floatgen = PropCheck.vector(PropCheck.iconst(1_000_000), PropCheck.ifloatinfnan(Float64));\n\njulia> @benchmark root(PropCheck.generate(floatgen))\nBenchmarkTools.Trial: 2 samples with 1 evaluation.\n Range (min … max): 4.524 s … 4.677 s ┊ GC (min … max): 24.68% … 25.56%\n Time (median): 4.600 s ┊ GC (median): 25.13%\n Time (mean ± σ): 4.600 s ± 108.561 ms ┊ GC (mean ± σ): 25.13% ± 0.63%\n\n █ █\n █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█ ▁\n 4.52 s Histogram: frequency by time 4.68 s <\n\n Memory estimate: 4.64 GiB, allocs estimate: 18364056.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"And again, with Supposition.jl:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> floatgen = Data.Vectors(Data.Floats{Float64}(); min_size=1_000_000, max_size=1_000_000);\n\njulia> @benchmark example($floatgen)\nBenchmarkTools.Trial: 736 samples with 1 evaluation.\n Range (min … max): 5.547 ms … 22.696 ms ┊ GC (min … max): 0.00% … 75.37%\n Time (median): 6.720 ms ┊ GC (median): 5.02%\n Time (mean ± σ): 6.793 ms ± 993.344 μs ┊ GC (mean ± σ): 6.67% ± 4.56%\n\n ▂▂▂▆▁▅▃▄▃▂▃█▄▂▂▅▂▇▄ ▆ ▁▃▃▂▁▃▁▁ ▁\n ▃▁▂▂▁▁▃▃▃▂▄▄▇▅██████████████████████████████▇▅▇▅██▇▄▃▃▃▃▂▂▃ ▅\n 5.55 ms Histogram: frequency by time 7.88 ms <\n\n Memory estimate: 25.04 MiB, allocs estimate: 34.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Once again, Supposition.jl beats PropCheck.jl by a factor of 500+ in time and a factor of 100 in memory.","category":"page"},{"location":"benchmarks.html#Strings","page":"Benchmarks","title":"Strings","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Both Supposition.jl and PropCheck.jl can generate the full spectrum of possible String, by going through all assigned unicode codepoints using specialized generation methods. Let's compare them, starting again with PropCheck.jl:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"# the default uses all valid `Char`\njulia> strgen = PropCheck.str(PropCheck.iconst(1_000_000));\n\njulia> @benchmark root(PropCheck.generate(strgen))\nBenchmarkTools.Trial: 9 samples with 1 evaluation.\n Range (min … max): 458.354 ms … 631.947 ms ┊ GC (min … max): 24.95% … 46.11%\n Time (median): 572.739 ms ┊ GC (median): 39.24%\n Time (mean ± σ): 559.611 ms ± 55.519 ms ┊ GC (mean ± σ): 38.24% ± 6.11%\n\n █ █ █ █ ██ █ █ █\n █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁█▁█▁▁▁▁▁▁▁▁▁▁▁▁▁██▁▁▁▁▁█▁▁▁▁▁▁▁▁▁█▁▁█ ▁\n 458 ms Histogram: frequency by time 632 ms <\n\n Memory estimate: 1.01 GiB, allocs estimate: 4999798.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"PropCheck.jl manages to go below 1s runtime for the first time! It still doesn't manage to use less than 1GiB of memory though. Supposition.jl on the other hand..","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> strgen = Data.Text(Data.Characters(); min_len=1_000_000, max_len=1_000_000);\n\njulia> @benchmark example($strgen)\nBenchmarkTools.Trial: 163 samples with 1 evaluation.\n Range (min … max): 26.756 ms … 62.461 ms ┊ GC (min … max): 0.00% … 48.46%\n Time (median): 28.386 ms ┊ GC (median): 1.95%\n Time (mean ± σ): 30.679 ms ± 6.679 ms ┊ GC (mean ± σ): 8.86% ± 12.41%\n\n ▄██▅▃\n █████▆█▆▃▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▃▁▄▂▃▂▂▁▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄ ▂\n 26.8 ms Histogram: frequency by time 56.9 ms <\n\n Memory estimate: 30.22 MiB, allocs estimate: 66.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"..completely obliterates PropCheck.jl yet again, being only barely slower than generating one million Int or Float64. To put this into perspective, bare randstring is faster by a factor of only ~3:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> using Random\n\njulia> @benchmark randstring(typemin(Char):\"\\xf7\\xbf\\xbf\\xbf\"[1], 1_000_000)\nBenchmarkTools.Trial: 675 samples with 1 evaluation.\n Range (min … max): 6.920 ms … 9.274 ms ┊ GC (min … max): 0.00% … 0.00%\n Time (median): 7.055 ms ┊ GC (median): 0.00%\n Time (mean ± σ): 7.221 ms ± 366.592 μs ┊ GC (mean ± σ): 2.22% ± 3.98%\n\n ▅█▅▃▁\n ▄█████▇▇▅▃▃▃▃▃▃▂▃▂▁▃▂▂▂▃▃▄▃▃▄▂▃▃▃▃▂▃▃▂▂▂▂▂▂▃▂▂▂▂▂▂▂▁▁▁▂▁▁▂▂ ▃\n 6.92 ms Histogram: frequency by time 8.5 ms <\n\n Memory estimate: 7.60 MiB, allocs estimate: 4.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Considering the amount of state that is being kept track of here, I'd say this is not too shabby.","category":"page"},{"location":"benchmarks.html#Map","page":"Benchmarks","title":"Map","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Next, function mapping - which is one of the most basic tools to transform an input into something else. Our mapped function is the humble \"make even\" function, x -> 2x. With PropCheck.jl:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> evengen = PropCheck.vector(PropCheck.iconst(1_000_000), PropCheck.map(x -> 2x, PropCheck.itype(Int)));\n\njulia> @benchmark root(PropCheck.generate(evengen))\nBenchmarkTools.Trial: 1 sample with 1 evaluation.\n Single result which took 7.554 s (26.22% GC) to evaluate,\n with a memory estimate of 9.32 GiB, over 32284641 allocations.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"and Supposition.jl:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> evengen = Data.Vectors(map(x -> 2x, Data.Integers{Int}()); min_size=1_000_000, max_size=1_000_000);\n\njulia> @benchmark example($evengen, 1)\nBenchmarkTools.Trial: 724 samples with 1 evaluation.\n Range (min … max): 5.444 ms … 34.544 ms ┊ GC (min … max): 0.00% … 82.94%\n Time (median): 5.900 ms ┊ GC (median): 3.80%\n Time (mean ± σ): 6.905 ms ± 3.775 ms ┊ GC (mean ± σ): 16.51% ± 16.42%\n\n ▅█▅▄▁\n █████▄▅▅▅▄▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▅▆▅▆▅▄▅▁▅▁▅▅▆▆▅ ▇\n 5.44 ms Histogram: log(frequency) by time 21.7 ms <\n\n Memory estimate: 25.04 MiB, allocs estimate: 36.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"And once again, Supposition.jl is victorious on all accounts.","category":"page"},{"location":"benchmarks.html#Filtering","page":"Benchmarks","title":"Filtering","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Benchmarking filter is a bit special now - Supposition.jl tries to protect you from too-long sampling sessions, which PropCheck.jl just doesn't even try. As a result, if we naively try to filter for even numbers with PropCheck, we get a monstrosity:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> evengen = PropCheck.vector(PropCheck.iconst(1_000_000), PropCheck.filter(iseven, PropCheck.itype(Int)));\n\njulia> @benchmark root(PropCheck.generate(evengen))\nBenchmarkTools.Trial: 1 sample with 1 evaluation.\n Single result which took 37.428 s (24.58% GC) to evaluate,\n with a memory estimate of 50.67 GiB, over 220777721 allocations.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"37s and 50GiB memory used is a very tall order (especially for just a single vector!), and should rightly be kindly asked to leave the venue. Supposition.jl on the other hand stops you in your tracks:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> evengen = Data.Vectors(filter(iseven, Data.Integers{Int}()); min_size=1_000_000, max_size=1_000_000);\n\njulia> @benchmark example($evengen)\nERROR: Tried sampling 100000 times, without getting a result. Perhaps you're filtering out too many examples?","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"and asks you what you're even doing. After all, make 1_000_000 coin flips and you're vanishingly unlikely to actually get a full vector with 1_000_000 elements that are all even (somewhere on the order of 1e-301030, to be precise).","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"So to test this properly, we're going to make sure that the filtering step is not the bottleneck, by first using our trusty x -> 2x again and then \"filtering\" for only even numbers. This adds the additional filtering step, but doesn't let it fail, so the probability of getting an even number doesn't come into play and we can purely focus on the relative overhead to just map.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"With PropCheck.jl, that leads to:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> evengen = PropCheck.vector(PropCheck.iconst(1_000_000), PropCheck.filter(iseven, PropCheck.map(x -> 2x, PropCheck.itype(Int))));\n\njulia> @benchmark root(PropCheck.generate(evengen))\nBenchmarkTools.Trial: 1 sample with 1 evaluation.\n Single result which took 8.756 s (21.85% GC) to evaluate,\n with a memory estimate of 9.63 GiB, over 45284301 allocations.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Almost 9s - what a monster, and that's just for a single example! As for Supposition.jl..","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> evengen = Data.Vectors(filter(iseven, map(x -> 2x, Data.Integers{Int}())); min_size=1_000_000, max_size=1_000_000);\n\njulia> @benchmark example($evengen)\nBenchmarkTools.Trial: 712 samples with 1 evaluation.\n Range (min … max): 5.594 ms … 29.914 ms ┊ GC (min … max): 5.10% … 68.75%\n Time (median): 6.672 ms ┊ GC (median): 4.67%\n Time (mean ± σ): 7.019 ms ± 2.362 ms ┊ GC (mean ± σ): 9.82% ± 9.84%\n\n ▃█▄▅▅▃\n ███████▅▃▂▂▂▂▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▂▁▂▂▂▂ ▃\n 5.59 ms Histogram: frequency by time 21.2 ms <\n\n Memory estimate: 25.04 MiB, allocs estimate: 34.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Another factor 1000 timewise, and factor 200 memory wise!","category":"page"},{"location":"benchmarks.html#Shrinking","page":"Benchmarks","title":"Shrinking","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Generating values is only half the effort though; what about shrinking values to find a counterexample?","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"We're again going to use vectors of things as inputs, though we're going to use a slight modification. Shrinking is already pretty complicated, so we're going to look at much shorter inputs (only 1000 elements long), as well as hold their size constant. This way, only the elements of each collection will shrink, and neither framework can get \"lucky\" by only having to shrink short collections.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"In order to prevent clobbering the output with unnecessary text, both PropCheck.jl and Supposition.jl are silenced through redirect_* and/or any switches they may provide themselves. This way, the resulting measurement should mostly be related to the shrinking itself, rather than any printing badness creeping in.","category":"page"},{"location":"benchmarks.html#Integers-2","page":"Benchmarks","title":"Integers","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Without further ado, here's PropCheck.jl:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> intgen = PropCheck.vector(PropCheck.iconst(1000), itype(Int));\n\njulia> @time check(isempty, intgen; show_initial=false)\n[ Info: Found counterexample for 'isempty', beginning shrinking...\nInternal error: during type inference of\niterate(Base.Iterators.ProductIterator{Tuple{Base.Generator{Array{Int64, 1}, Base.Fix1{typeof(PropCheck.unfold), Base.ComposedFunction{Type{PropCheck.Shuffle{T} where T}, typeof(PropCheck.shrink)}}}, Vararg{Array{PropCheck.Tree{Int64}, 1}, 999}}})\nEncountered stack overflow.\nThis might be caused by recursion over very long tuples or argument lists.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"You'll notice that I'm using @time here instead of @benchmark. The reason for this is pragmatic - I don't want to wait all day on PropCheck.jl. In this case though, the worry was completely unfounded, as the compiler can't handle the huge amount of nested functions this ends up generating. This result is, unfortunately, consistent for the following experiments. As such, I'll only show Supposition.jl.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Supposition.jl not only delivers a result, but does so in record time:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> intgen = Data.Vectors(Data.Integers{Int}(); min_size=1000, max_size=1000);\n\njulia> @time @check db=false isempty(intgen);\n┌ Error: Property doesn't hold!\n│ Description = \"isempty\"\n│ Example = ([-9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808 … -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808],)\n└ @ Supposition ~/Documents/projects/Supposition.jl/src/testset.jl:280\nTest Summary: | Fail Total\nisempty | 1 1\n 0.404841 seconds (555.13 k allocations: 657.688 MiB, 5.12% gc time, 4.09% compilation time: 2% of which was recompilation)","category":"page"},{"location":"benchmarks.html#Floats-2","page":"Benchmarks","title":"Floats","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Supposition.jl:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> floatgen = Data.Vectors(Data.Floats{Float64}(); min_size=1000, max_size=1000);\n\njulia> @time @check db=false isempty(floatgen);\n┌ Error: Property doesn't hold!\n│ Description = \"isempty\"\n│ Example = ([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 … 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],)\n└ @ Supposition ~/Documents/projects/Supposition.jl/src/testset.jl:280\nTest Summary: | Fail Total\nisempty | 1 1\n 0.394480 seconds (535.17 k allocations: 656.722 MiB, 5.65% gc time, 1.20% compilation time: 5% of which was recompilation)","category":"page"},{"location":"benchmarks.html#Strings-2","page":"Benchmarks","title":"Strings","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Supposition.jl:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> strgen = Data.Text(Data.Characters(); min_len=1000, max_len=1000);\n\njulia> @time @check db=false isempty(strgen);\n┌ Error: Property doesn't hold!\n│ Description = \"isempty\"\n│ Example = (\"\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\",)\n└ @ Supposition ~/Documents/projects/Supposition.jl/src/testset.jl:280\nTest Summary: | Fail Total\nisempty | 1 1\n 0.564029 seconds (548.62 k allocations: 592.570 MiB, 3.27% gc time, 0.77% compilation time: 4% of which was recompilation)","category":"page"},{"location":"benchmarks.html#Map-2","page":"Benchmarks","title":"Map","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Supposition.jl:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> mapgen = Data.Vectors(map(x -> 2x, Data.Integers{Int}()); min_size=1000, max_size=1000);\n\njulia> @time @check db=false isempty(mapgen);\n┌ Error: Property doesn't hold!\n│ Description = \"isempty\"\n│ Example = ([0, 0, 0, 0, 0, 0, 0, 0, 0, 0 … 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],)\n└ @ Supposition ~/Documents/projects/Supposition.jl/src/testset.jl:280\nTest Summary: | Fail Total\nisempty | 1 1\n 0.408549 seconds (557.02 k allocations: 657.772 MiB, 5.49% gc time, 4.99% compilation time: <1% of which was recompilation)","category":"page"},{"location":"benchmarks.html#Filter","page":"Benchmarks","title":"Filter","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Supposition.jl:","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"julia> oddgen = Data.Vectors(filter(isodd, map(x -> 2x+1, Data.Integers{Int}())); min_size=1000, max_size=1000);\n\njulia> @time @check db=false isempty(oddgen);\n┌ Error: Property doesn't hold!\n│ Description = \"isempty\"\n│ Example = ([1, 1, 1, 1, 1, 1, 1, 1, 1, 1 … 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],)\n└ @ Supposition ~/Documents/projects/Supposition.jl/src/testset.jl:280\nTest Summary: | Fail Total\nisempty | 1 1\n 0.412517 seconds (562.41 k allocations: 658.063 MiB, 5.59% gc time, 4.44% compilation time: 2% of which was recompilation)","category":"page"},{"location":"benchmarks.html#Conclusion","page":"Benchmarks","title":"Conclusion","text":"","category":"section"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"If you've read down to here, I don't think I even have to write it out - Supposition.jl is fast! I feel pretty confident saying that it's unlikely to be the bottleneck of a testsuite. All of that without even explicitly looking for places to optimize the package yet. Of course, this doesn't even touch cranking up the number of samples Supposition.jl tries, or any form of memoization on the property you could quite easily add. So there is potential for going faster in the future.","category":"page"},{"location":"benchmarks.html","page":"Benchmarks","title":"Benchmarks","text":"Go and incorporate fuzzing into your testsuite ;)","category":"page"},{"location":"interfaces.html#Userfacing-API","page":"Userfacing API","title":"Userfacing API","text":"","category":"section"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"Supposition.jl provides a few main interfaces to hook into with your code, as well as use during general usage of Supposition.jl. These are pretty robust and very minimal.","category":"page"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"The interfaces mentioned on this page are intended for user-extension & usage, in the manner described. Overloading the functions in a different way or assuming more of an interface than is guaranteed is not supported.","category":"page"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"For the abstract-type based interfaces ExampleDB and Possibility, you can use the API provided by RequiredInterfaces.jl to check for basic compliance, if you want to provide a custom implementation.","category":"page"},{"location":"interfaces.html#Macro-based-API","page":"Userfacing API","title":"Macro-based API","text":"","category":"section"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"These macros are the main entryway most people should use, for both entry-level and advanced usage. @check is responsible for interfacing with the internals of Supposition.jl, orchestrating the generation of examples & reporting back to the testing framework.","category":"page"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"@composed is the one-stop-shop for composing a new generator from a number of existing ones.","category":"page"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"Supposition.@check\nSupposition.@composed","category":"page"},{"location":"interfaces.html#Supposition.@check","page":"Userfacing API","title":"Supposition.@check","text":"@check [key=val]... function...\n\nThe main way to declare & run a property based test. Called like so:\n\njulia> using Supposition, Supposition.Data\n\njulia> Supposition.@check [options...] function foo(a = Data.Text(Data.Characters(); max_len=10))\n length(a) > 8\n end\n\nSupported options, passed as key=value:\n\nrng::Random.AbstractRNG: Pass an RNG to use. Defaults to Random.Xoshiro(rand(Random.RandomDevice(), UInt)).\nmax_examples::Int: The maximum number of generated examples that are passed to the property.\nbroken::Bool: Mark a property that should pass but doesn't as broken, so that failures are not counted.\nrecord::Bool: Whether the result of the invocation should be recorded with any parent testsets.\ndb: Either a Boolean (true uses a fallback database, false stops recording examples) or an ExampleDB.\nconfig: A CheckConfig object that will be used as a default for all previous options. Options that are passed explicitly to @check will override whatever is provided through config.\n\nThe arguments to the given function are expected to be generator strategies. The names they are bound to are the names the generated object will have in the test. These arguments will be shown should the property fail!\n\nExtended help\n\nReusing existing properties\n\nIf you already have a predicate defined, you can also use the calling syntax in @check. Here, the generator is passed purely positionally to the given function; no argument name is necessary.\n\njulia> using Supposition, Supposition.Data\n\njulia> isuint8(x) = x isa UInt8\n\njulia> intgen = Data.Integers{UInt8}()\n\njulia> Supposition.@check isuint8(intgen)\n\nPassing a custom RNG\n\nIt is possible to optionally give a custom RNG object that will be used for random data generation. If none is given, Xoshiro(rand(Random.RandomDevice(), UInt)) is used instead.\n\njulia> using Supposition, Supposition.Data, Random\n\n# use a custom Xoshiro instance\njulia> Supposition.@check rng=Xoshiro(1234) function foo(a = Data.Text(Data.Characters(); max_len=10))\n length(a) > 8\n end\n\nwarning: Hardware RNG\nBe aware that you cannot pass a hardware RNG to @check directly. If you want to randomize based on hardware entropy, seed a copyable RNG like Xoshiro from your hardware RNG and pass that to @check instead. The RNG needs to be copyable for reproducibility.\n\nAdditional Syntax\n\nIn addition to passing a whole function like above, the following syntax are also supported:\n\ntext = Data.Text(Data.AsciiCharacters(); max_len=10)\n\n# If no name is needed, use an anonymous function\n@check (a = text) -> a*a\n@check (a = text,) -> \"foo: \"*a\n@check (a = text, num = Data.Integers(0,10)) -> a^num\n\n# ..or give the anonymous function a name too - works with all three of the above\n@check build_sentence(a = text, num = Data.Floats{Float16}()) -> \"The $a is $num!\"\nbuild_sentence(\"foo\", 0.5) # returns \"The foo is 0.5!\"\n\nwarning: Replayability\nWhile you can pass an anonymous function to @check, be aware that doing so may hinder replayability of found test cases when surrounding invocations of @check are moved. Only named functions are resistant to this.\n\n\n\n\n\n","category":"macro"},{"location":"interfaces.html#Supposition.@composed","page":"Userfacing API","title":"Supposition.@composed","text":"@composed\n\nA way to compose multiple Possibility into one, by applying a function.\n\nThe return type is inferred as a best-effort!\n\nUsed like so:\n\njulia> using Supposition, Supposition.Data\n\njulia> text = Data.Text(Data.AsciiCharacters(); max_len=10)\n\njulia> gen = Supposition.@composed function foo(a = text, num=Data.Integers(0, 10))\n lpad(num, 2) * \": \" * a\n end\n\njulia> example(gen)\n\" 8: giR2YL\\rl\"\n\nIn addition to passing a whole function like above, the following syntax are also supported:\n\n# If no name is needed, use an anonymous function\ndouble_up = @composed (a = text) -> a*a\nprepend_foo = @composed (a = text,) -> \"foo: \"*a\nexpo_str = @composed (a = text, num = Data.Integers(0,10)) -> a^num\n\n# ..or give the anonymous function a name too - works with all three of the above\nsentence = @composed build_sentence(a = text, num = Data.Floats{Float16}()) -> \"The $a is $num!\"\nbuild_sentence(\"foo\", 0.5) # returns \"The foo is 0.5!\"\n\n# or compose a new generator out of an existing function\nmy_func(str, number) = number * \"? \" * str\nask_number = @composed my_func(text, num)\n\n\n\n\n\n","category":"macro"},{"location":"interfaces.html#API-for-controlling-fuzzing","page":"Userfacing API","title":"API for controlling fuzzing","text":"","category":"section"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"These functions are intended for usage while testing, having various effects on the shrinking/fuzzing process. They are not intended to be part of a codebase permanently/in production.","category":"page"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"The trailing exclamation mark serves as a reminder that this will, under the hood, modify the currently running testcase.","category":"page"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"Supposition.target!(::Float64)\nSupposition.assume!(::Bool)\nSupposition.produce!(::Data.Possibility)\nSupposition.reject!\nSupposition.err_less\nSupposition.DEFAULT_CONFIG","category":"page"},{"location":"interfaces.html#Supposition.target!-Tuple{Float64}","page":"Userfacing API","title":"Supposition.target!","text":"target!(score)\n\nUpdate the currently running testcase to track the given score as its target.\n\nscore must be convertible to a Float64.\n\nwarning: Multiple Updates\nThis score can only be set once! Repeated calls will be ignored.\n\nwarning: Callability\nThis can only be called while a testcase is currently being examined or an example for a Possibility is being actively generated. It is ok to call this inside of @composed or @check, as well as any functions only intended to be called from one of those places.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Supposition.assume!-Tuple{Bool}","page":"Userfacing API","title":"Supposition.assume!","text":"assume!(precondition::Bool)\n\nIf this precondition is not met, abort the test and mark the currently running testcase as invalid.\n\nwarning: Callability\nThis can only be called while a testcase is currently being examined or an example for a Possibility is being actively generated. It is ok to call this inside of @composed or @check, as well as any functions only intended to be called from one of those places.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Supposition.Data.produce!-Tuple{Supposition.Data.Possibility}","page":"Userfacing API","title":"Supposition.Data.produce!","text":"produce!(p::Possibility{T}) -> T\n\nProduces a value from the given Possibility, recording the required choices in the currently active TestCase.\n\nwarning: Callability\nThis can only be called while a testcase is currently being examined or an example for a Possibility is being actively generated. It is ok to call this inside of @composed or @check, as well as any functions only intended to be called from one of those places.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Supposition.reject!","page":"Userfacing API","title":"Supposition.reject!","text":"reject!()\n\nReject the current testcase as invalid, meaning the generated example should not be considered as producing a valid counterexample.\n\nwarning: Callability\nThis can only be called while a testcase is currently being examined or an example for a Possibility is being actively generated. It is ok to call this inside of @composed or @check, as well as any functions only intended to be called from one of those places.\n\n\n\n\n\n","category":"function"},{"location":"interfaces.html#Supposition.err_less","page":"Userfacing API","title":"Supposition.err_less","text":"err_less(e1::E, e2::E) where E\n\nA comparison function for exceptions, used when encountering an error in a property. Returns true if e1 is considered to be \"easier\" or \"simpler\" than e2. Only definable when both e1 and e2 have the same type.\n\nThis is optional to implement, but may be beneficial for shrinking counterexamples leading to an error with rich metadata, in which case err_less will be used to compare errors of the same type from different counterexamples. In particular, this function will likely be helpful for errors with metadata that is far removed from the input that caused the error itself, but would nevertheless be helpful when investigating the failure.\n\nnote: Coincidental Errors\nThere may also be situations where defining err_less won't help to find a smaller counterexample if the cause of the error is unrelated to the choices taken during generation. For instance, this is the case when there is no network connection and a Sockets.DNSError is thrown during the test, or there is a network connection but the host your program is trying to connect to does not have an entry in DNS.\n\n\n\n\n\n","category":"function"},{"location":"interfaces.html#Supposition.DEFAULT_CONFIG","page":"Userfacing API","title":"Supposition.DEFAULT_CONFIG","text":"DEFAULT_CONFIG\n\nA ScopedValue holding the CheckConfig that will be used by default & as a fallback.\n\nCurrently uses these values:\n\nrng: Random.Xoshiro(rand(Random.RandomDevice(), UInt))\nmax_examples: 10_000\nrecord: true\nverbose: false\nbroken: false\ndb: UnsetDB()\nbuffer_size: 100_000\n\n@check will use a new instance of Random.Xoshiro by itself.\n\n\n\n\n\n","category":"constant"},{"location":"interfaces.html#Available-Possibility","page":"Userfacing API","title":"Available Possibility","text":"","category":"section"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"The Data module contains most everyday objects you're going to use when writing property based tests with Supposition.jl. For example, the basic generators for integers, strings, floating point values etc. are defined here. Everything listed in this section is considered supported under semver.","category":"page"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"Modules = [Data]\nOrder = [:function, :type]","category":"page"},{"location":"interfaces.html#Functions","page":"Userfacing API","title":"Functions","text":"","category":"section"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"Modules = [Data]\nOrder = [:function]\nFilter = t -> begin\n t != Supposition.Data.produce!\nend","category":"page"},{"location":"interfaces.html#Base.:|-Tuple{Supposition.Data.Possibility, Supposition.Data.Possibility}","page":"Userfacing API","title":"Base.:|","text":"|(::Possibility{T}, ::Possibility{S}) where {T,S} -> OneOf{Union{T,S}}\n\nCombine two Possibility into one, sampling uniformly from either.\n\nIf either of the two arguments is a OneOf, the resulting object acts as if all original non-OneOf had be given to OneOf instead. That is, e.g. OneOf(a, b) | c will act like OneOf(a,b,c).\n\nSee also OneOf.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Base.filter-Tuple{Any, Supposition.Data.Possibility}","page":"Userfacing API","title":"Base.filter","text":"filter(f, pos::Possibility)\n\nFilter the output of produce! on pos by applying the predicate f.\n\nnote: No stalling\nIn order not to stall generation of values, this will not try to produce a value from pos forever, but reject the testcase after some attempts.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Base.map-Tuple{Any, Supposition.Data.Possibility}","page":"Userfacing API","title":"Base.map","text":"map(f, pos::Possibility)\n\nApply f to the result of calling produce! on pos (lazy mapping).\n\nEquivalent to calling Map(pos, f).\n\nSee also Map.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Supposition.Data.BitIntegers-Tuple{}","page":"Userfacing API","title":"Supposition.Data.BitIntegers","text":"BitIntegers() <: Possibility{Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}}\n\nA Possibility for generating all possible bitintegers with fixed size.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Supposition.Data.bind-Tuple{Any, Supposition.Data.Possibility}","page":"Userfacing API","title":"Supposition.Data.bind","text":"bind(f, pos::Possibility)\n\nMaps the output of produce! on pos through f, and calls produce! on the result again. f is expected to take a value and return a Possibility.\n\nEquivalent to calling Bind(pos, f).\n\nSee also Bind.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Supposition.Data.postype-Tuple{P} where P<:Supposition.Data.Possibility","page":"Userfacing API","title":"Supposition.Data.postype","text":"postype(::P) where P <: Possibility\n\nGives the type of objects this Possibility object will generate.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Supposition.Data.postype-Union{Tuple{Type{_P}}, Tuple{_P}, Tuple{T}} where {T, _P<:Supposition.Data.Possibility{T}}","page":"Userfacing API","title":"Supposition.Data.postype","text":"postype(::Type{P<:Possibility})\n\nGives the type of objects this Possibility type will generate.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Supposition.Data.recursive-Tuple{Any, Supposition.Data.Possibility}","page":"Userfacing API","title":"Supposition.Data.recursive","text":"recursive(f, pos::Possibility; max_layers=5)\n\nRecursively expand pos into deeper nested Possibility by repeatedly passing pos itself to f. f returns a new Possibility, which is then passed into f again until the maximum depth is achieved.\n\nEquivalent to calling Recursive(pos, f).\n\nSee also Recursive.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Types","page":"Userfacing API","title":"Types","text":"","category":"section"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"Modules = [Data]\nOrder = [:type]\nFilter = t -> begin\n t != Supposition.Data.Possibility\nend","category":"page"},{"location":"interfaces.html#Supposition.Data.AsciiCharacters","page":"Userfacing API","title":"Supposition.Data.AsciiCharacters","text":"AsciiCharacters() <: Possibility{Char}\n\nA Possibility of producing arbitrary Char instances that are isascii. More efficient than filtering Characters.\n\njulia> using Supposition\n\njulia> ascii = Data.AsciiCharacters()\n\njulia> example(ascii, 5)\n5-element Vector{Char}:\n '8': ASCII/Unicode U+0038 (category Nd: Number, decimal digit)\n 'i': ASCII/Unicode U+0069 (category Ll: Letter, lowercase)\n 'R': ASCII/Unicode U+0052 (category Lu: Letter, uppercase)\n '\\f': ASCII/Unicode U+000C (category Cc: Other, control)\n '>': ASCII/Unicode U+003E (category Sm: Symbol, math)\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Bind","page":"Userfacing API","title":"Supposition.Data.Bind","text":"Bind(source::Possibility, f)\n\nBinds f to source, i.e., on produce!(::Bind, ::TestCase) this calls produce! on source, the result of which is passed to f, the output of which will be used as input to produce! again.\n\nIn other words, f takes a value produce!d by source and gives back a Possibility that is then immediately produce!d from.\n\nEquivalent to bind(f, source).\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Booleans","page":"Userfacing API","title":"Supposition.Data.Booleans","text":"Booleans() <: Possibility{Bool}\n\nA Possibility for sampling boolean values.\n\njulia> using Supposition\n\njulia> bools = Data.Booleans()\n\njulia> example(bools, 4)\n4-element Vector{Bool}:\n 0\n 1\n 0\n 1\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Characters","page":"Userfacing API","title":"Supposition.Data.Characters","text":"Characters(;valid::Bool = false) <: Possibility{Char}\n\nA Possibility of producing arbitrary Char instances.\n\nwarning: Unicode\nThis will produce! ANY possible Char by default, not just valid unicode codepoints! To only produce valid unicode codepoints, pass valid=true as a keyword argument.\n\njulia> using Supposition\n\njulia> chars = Data.Characters()\n\njulia> example(chars, 5)\n5-element Vector{Char}:\n '⠺': Unicode U+283A (category So: Symbol, other)\n '𰳍': Unicode U+30CCD (category Lo: Letter, other)\n '\\U6ec9c': Unicode U+6EC9C (category Cn: Other, not assigned)\n '\\U1a05c5': Unicode U+1A05C5 (category In: Invalid, too high)\n '𓂫': Unicode U+130AB (category Lo: Letter, other)\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Dicts","page":"Userfacing API","title":"Supposition.Data.Dicts","text":"Dicts(keys::Possibility, values::Possibility; min_size=0, max_size=10_000)\n\nA Possibility for generating Dict objects. The keys are drawn from keys, while the values are drawn from values. min_size/max_size control the number of objects placed into the resulting Dict, respectively.\n\njulia> dicts = Data.Dicts(Data.Integers{UInt8}(), Data.Integers{Int8}(); max_size=3);\n\njulia> example(dicts)\nDict{UInt8, Int8} with 2 entries:\n 0x54 => -29\n 0x1f => -28\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Floats","page":"Userfacing API","title":"Supposition.Data.Floats","text":"Floats{T <: Union{Float16,Float32,Float64}}(;infs=true, nans=true) <: Possibility{T}\n\nA Possibility for sampling floating point values.\n\nThe keyword infs controls whether infinities can be generated. nans controls whether any NaN (signaling & quiet) will be generated.\n\nwarning: Inf, Nan\nThis possibility will generate any valid instance, including positive and negative infinities, signaling and quiet NaNs and every possible float.\n\njulia> using Supposition\n\njulia> floats = Data.Floats{Float16}()\n\njulia> example(floats, 5)\n5-element Vector{Float16}:\n -8.3e-6\n 1.459e4\n 3.277\n NaN\n -0.0001688\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Floats-Tuple{}","page":"Userfacing API","title":"Supposition.Data.Floats","text":"Floats(;nans=true, infs=true) <: Possibility{Union{Float64,Float32,Float16}}\n\nA catch-all for generating instances of all three IEEE floating point types.\n\n\n\n\n\n","category":"method"},{"location":"interfaces.html#Supposition.Data.Integers","page":"Userfacing API","title":"Supposition.Data.Integers","text":"Integers(minimum::T, maximum::T) <: Possibility{T <: Integer}\nIntegers{T}() <: Possibility{T <: Integer}\n\nA Possibility representing drawing integers from [minimum, maximum]. The second constructors draws from the entirety of T.\n\nProduced values are of type T.\n\njulia> using Supposition\n\njulia> is = Data.Integers{Int}()\n\njulia> example(is, 5)\n5-element Vector{Int64}:\n -5854403925308846160\n 4430062772779972974\n -9995351034504801\n 2894734754389242339\n -6640496903289665416\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Just","page":"Userfacing API","title":"Supposition.Data.Just","text":"Just(value::T) <: Possibility{T}\n\nA Possibility that always produces value.\n\nwarning: Mutable Data\nThe source object given to this Just is not copied when produce! is called. Be careful with mutable data!\n\njulia> using Supposition\n\njulia> three = Data.Just(3)\n\njulia> example(three, 3)\n3-element Vector{Int64}:\n 3\n 3\n 3\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Map","page":"Userfacing API","title":"Supposition.Data.Map","text":"Map(source::Possibility, f) <: Possibility\n\nA Possibility representing mapping values from source through f.\n\nEquivalent to calling map(f, source).\n\nThe pre-calculated return type of Map is a best effort and may be wider than necessary.\n\njulia> using Supposition\n\njulia> makeeven(x) = (x ÷ 2) * 2\n\njulia> pos = map(makeeven, Data.Integers{Int8}())\n\njulia> all(iseven, example(pos, 10_000))\ntrue\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.OneOf","page":"Userfacing API","title":"Supposition.Data.OneOf","text":"OneOf(pos::Possibility...) <: Possibility\n\nA Possibility able to generate any of the examples one of the given Possibility can produce. The given Possibility are sampled from uniformly.\n\nAt least one Possibility needs to be given to OneOf.\n\npostype(::OneOf) is inferred as a best effort, and may be wider than necessary.\n\nOneOf can also be constructed through use of a | b on Possibility. Constructed in this way, if either a or b is already a OneOf, the resulting OneOf acts as if it had been given the original Possibility objects in the first place. That is, OneOf(a, b) | c acts like OneOf(a, b, c).\n\nSee also WeightedNumbers and WeightedSample.\n\njulia> of = Data.OneOf(Data.Integers{Int8}(), Data.Integers{UInt8}());\n\njulia> Data.postype(of)\nUnion{Int8, UInt8}\n\njulia> ex = map(of) do i\n (i, typeof(i))\n end;\n\njulia> example(ex)\n(-83, Int8)\n\njulia> example(ex)\n(0x9f, UInt8)\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Pairs","page":"Userfacing API","title":"Supposition.Data.Pairs","text":"Pairs(first::Possibility{T}, second::Possibility{S}) where {T,S} <: Possibility{Pair{T,S}}\n\nA Possibility for producing a => b pairs. a is produced by first, b is produced by second.\n\njulia> p = Data.Pairs(Data.Integers{UInt8}(), Data.Floats{Float64}());\n\njulia> example(p, 4)\n4-element Vector{Pair{UInt8, Float64}}:\n 0x41 => 4.1183566661848205e-230\n 0x48 => -2.2653631095108555e-119\n 0x2a => -6.564396855333643e224\n 0xec => 1.9330751262581671e-53\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Recursive","page":"Userfacing API","title":"Supposition.Data.Recursive","text":"Recursive(base::Possibility, extend; max_layers::Int=5) <: Possibility{T}\n\nA Possibility for generating recursive data structures. base is the basecase of the recursion. extend is a function returning a new Possibility when given a Possibility, called to recursively expand a tree starting from base. The returned Possibility is fed back into extend again, expanding the recursion by one layer.\n\nmax_layers designates the maximum layers Recursive should keep track of. This must be at least 1, so that at least the base case can always be generated. Note that this implies extend will be used at most max_layers-1 times, since the base case of the recursion will not be wrapped.\n\nEquivalent to calling recursive(extend, base).\n\nExamples\n\njulia> base = Data.Integers{UInt8}()\n\njulia> wrap(pos) = Data.Vectors(pos; min_size=2, max_size=3)\n\njulia> rec = Data.recursive(wrap, base; max_layers=3);\n\njulia> Data.postype(rec) # the result is formatted here for legibility\nUnion{UInt8,\n Vector{UInt8},\n Vector{Union{UInt8, Vector{UInt8}}}\n}\n\njulia> example(rec)\n0x31\n\njulia> example(rec)\n2-element Vector{Union{UInt8, Vector{UInt8}}}:\n UInt8[0xa9, 0xb4]\n 0x9b\n\njulia> example(rec)\n2-element Vector{UInt8}:\n 0xbd\n 0x25\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.SampledFrom","page":"Userfacing API","title":"Supposition.Data.SampledFrom","text":"SampledFrom(collection) <: Possibility{eltype(collection)}\n\nA Possibility for sampling uniformly from collection.\n\ncollection, as well as its eachindex, is assumed to be indexable.\n\nwarning: Mutable Data\nThe source objects from the collection given to this SampledFrom is not copied when produce! is called. Be careful with mutable data!\n\ntip: Sampling from `String`\nTo sample from a String, you can collect the string first to get a Vector{Char}. This is necessary because Strings use the variable-length UTF-8 encoding, which isn't arbitrarily indexable in constant time.\n\njulia> using Supposition\n\njulia> sampler = Data.SampledFrom([1, 1, 1, 2])\n\njulia> example(sampler, 4)\n4-element Vector{Int64}:\n 1\n 1\n 2\n 1\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Satisfying","page":"Userfacing API","title":"Supposition.Data.Satisfying","text":"Satisfying(source::Possibility, pred) <: Possibility\n\nA Possibility representing values from source fulfilling pred.\n\nEquivalent to calling filter(f, source).\n\njulia> using Supposition\n\njulia> pos = filter(iseven, Data.Integers{Int8}())\n\njulia> all(iseven, example(pos, 10_000))\ntrue\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Text","page":"Userfacing API","title":"Supposition.Data.Text","text":"Text(alphabet::Possibility{Char}; min_len=0, max_len=10_000) <: Possibility{String}\n\nA Possibility for producing Strings containing Chars of a given alphabet.\n\njulia> using Supposition\n\njulia> text = Data.Text(Data.AsciiCharacters(); max_len=15)\n\njulia> example(text, 5)\n5-element Vector{String}:\n \"U\\x127lxf\"\n \"hm\\x172SJ-(\"\n \"h`\\x03\\0\\x01[[il\"\n \"\\x0ep4\"\n \"9+Hk3 ii\\x1eT\"\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.Vectors","page":"Userfacing API","title":"Supposition.Data.Vectors","text":"Vectors(elements::Possibility{T}; min_size=0, max_size=10_000) <: Possibility{Vector{T}}\n\nA Possibility representing drawing vectors with length l in min_size <= l <= max_size, holding elements of type T.\n\nmin_size and max_size must be positive numbers, with min_size <= max_size.\n\njulia> using Supposition\n\njulia> vs = Data.Vectors(Data.Floats{Float16}(); max_size=5)\n\njulia> example(vs, 3)\n3-element Vector{Vector{Float16}}:\n [9.64e-5, 9.03e3, 0.04172, -0.0003352]\n [9.793e-5, -2.893, 62.62, 0.0001961]\n [-0.007023, NaN, 3.805, 0.1943]\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.WeightedNumbers","page":"Userfacing API","title":"Supposition.Data.WeightedNumbers","text":"WeightedNumbers(weights::Vector{Float64}) <: Possibility{Int}\n\nSample the numbers from 1:length(weights), each with a weight of weights[i].\n\nThe weights may be any number > 0.0.\n\nSee also OneOf.\n\njulia> using Supposition\n\njulia> bn = Data.WeightedNumbers([1.0, 1.0, 3.0]);\n\njulia> example(Data.Vectors(bn; min_size=3, max_size=15), 5)\n5-element Vector{Vector{Int64}}:\n [3, 2, 3, 3, 2, 3, 3]\n [1, 1, 1, 2, 1, 3, 1, 3]\n [2, 3, 3, 3, 3, 3, 3, 1, 1, 3, 3, 3]\n [3, 3, 2, 3, 3]\n [1, 3, 3, 3, 2, 2]\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Supposition.Data.WeightedSample","page":"Userfacing API","title":"Supposition.Data.WeightedSample","text":"WeightedSample{T}(colllection, weights::Vector{Float64}) <: Possibility{T}\n\nDraw an element from the indexable collection, biasing the drawing process by assigning each index i of col the weight at weights[i].\n\nlength of col and weights must be equal and eachindex(col) must be indexable.\n\nSee also OneOf.\n\ntip: Sampling from `String`\nTo sample from a String, you can collect the string first to get a Vector{Char}. This is necessary because Strings use the variable-length UTF-8 encoding, which isn't arbitrarily indexable in constant time.\n\njulia> bs = Data.WeightedSample([\"foo\", \"bar\", \"baz\"], [3.0, 1.0, 1.0]);\n\njulia> example(bs, 10)\n10-element Vector{String}:\n \"foo\"\n \"foo\"\n \"bar\"\n \"baz\"\n \"baz\"\n \"foo\"\n \"bar\"\n \"foo\"\n \"foo\"\n \"bar\"\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#Type-based-hooks","page":"Userfacing API","title":"Type-based hooks","text":"","category":"section"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"These are hooks for users to provide custom implementations of certain parts of Supposition.jl. Follow their contracts precisely if you implement your own.","category":"page"},{"location":"interfaces.html#Possibility{T}","page":"Userfacing API","title":"Possibility{T}","text":"","category":"section"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"Supposition.Data.Possibility","category":"page"},{"location":"interfaces.html#Supposition.Data.Possibility","page":"Userfacing API","title":"Supposition.Data.Possibility","text":"Possibility{T}\n\nAbstract supertype for all generators. The T type parameter describes the kinds of objects generated by this integrated shrinker.\n\nRequired methods:\n\nproduce!(::TestCase, ::P) where P <: Possibility\n\nFallback definitions:\n\npostype(::Possibility{T}) -> Type{T}\nexample(::Possibility{T}) -> T\n\n\n\n\n\n","category":"type"},{"location":"interfaces.html#ExampleDB","page":"Userfacing API","title":"ExampleDB","text":"","category":"section"},{"location":"interfaces.html","page":"Userfacing API","title":"Userfacing API","text":"Supposition.ExampleDB","category":"page"},{"location":"interfaces.html#Supposition.ExampleDB","page":"Userfacing API","title":"Supposition.ExampleDB","text":"ExampleDB\n\nAn abstract type representing a database of previous counterexamples.\n\nRequired methods:\n\nrecords(::ExampleDB): Returns an iterable of all currently recorded counterexamples.\nrecord!(::ExampleDB, key, value): Record the counterexample value under the key key.\nretrieve(::ExampleDB, key)::Option: Retrieve the previously recorded counterexample stored under key. Return nothing if no counterexample was stored under that key.\n\n\n\n\n\n","category":"type"},{"location":"intro.html#Introduction","page":"Introduction to PBT","title":"Introduction","text":"","category":"section"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"What follows is a short introduction to what Property Based Testing (PBT) is from my POV. This may not be exhaustive - if you want a more formal or deeper dive into this topic, I can greatly recommend this article by one of the authors of Hypothesis, the property based testing framework Supposition.jl is based on. For more formal methods, check out the blog of Hillel Wayne.","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"If you're fine with the (short) introduction I'm giving, but want some sort of motivation about WHY you should care, here's a quote from him, itself referencing other people:","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"In 2010 Carmen Reinhart and Kenneth Rogoff published Growth in a Time of Debt. It’s arguably one of the most influential economics papers of the decade, convincing the IMF to push austerity measures in the European debt crisis. It was a very, very big deal.In 2013 they shared their code with another team, who quickly found a bug. Once corrected, the results disappeared.Greece took on austerity because of a software bug. That’s pretty fucked up.","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"This is just a soundbite, the full article How Do We Trust Science Code has a number of other great examples for why the problem of correct software is not just something esoteric for software engineers, but is of practical interest to scientific coders too.","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"Now that I have your attention, let's get started:","category":"page"},{"location":"intro.html#What-is-Property-Based-Testing?","page":"Introduction to PBT","title":"What is Property Based Testing?","text":"","category":"section"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"Property Based Testing is the idea of checking the correctness of a function, algorithm or calculation against a number of desired properties that function should observe. Consider this function:","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"function foo(a::Int, b::Int)\n a < 5 && return false\n b < 5 && return false\n return true\nend","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"From reading the source code as a user, we can see that","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"a and b must be of type Int\nif either a or b is smaller than 5, the function returns false, otherwise it returns true.","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"So the property that should hold for this function is that if we supply two Int arguments, the function will always tell us whether they are both at least 5. We might define this property for testing purposes like so:","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"function foo_prop()\n a = rand(Int)\n b = rand(Int)\n if a < 5 || b < 5\n return foo(a,b) == false\n else\n return foo(a,b) == true\n end\nend","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"Every time we run foo_prop, we generate a random input for foo and check whether its output behaves as expected. Written like this, it has a few major drawbacks:","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"Being somewhat certain that we cover the function completely quickly becomes infeasible\nWe have no control over the numbers being generated\nWe can't reuse the way we generate these numbers; expanding a testsuite like this leads to a lot of boilerplate and repetition","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"On its own, just foo_prop is already property based testing - we take some expected input and check it against the expected output/behavior. However, on 64-bit systems, Int has a value in the interval [-9223372036854775808, 9223372036854775807], which is one of 2^64 different values. Considering that our function takes two of those, our input space has 2^2 times 64 distinct pairs of elements! Looping through all of them would take much too long. Worse, we may then need to record the result for each of them to prove later that we actually checked it. With more complex data types, this only grows worse as more different types and combinations of them are involved.","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"This is where a related approach called fuzzing comes in - instead of checking ALL values and giving a 100% guarantee that it works as expected, we only check a sampled subset of all possible values and therefore only receive a probabilistic result. However, this comes with the distinct advantage of being much, much faster than checking all possible values. We trade accuracy for performance (much like we do with floating point values). If our sampling is good enough & representative of the actual data we'd usually expect, this can be a very good indicator for correctness on our input. The difficulty comes from the second point above - controlling the values we put in to a satisfying degree, as well as, once a failure is found, reducing it to something we humans can more easily use to pinpoint the uncovered bug, through a process called \"shrinking\". You can find the introductory explanations for how this works in the context of Supposition.jl in the Basic Usage section of the examples.","category":"page"},{"location":"intro.html#Julia-specific-nuances","page":"Introduction to PBT","title":"Julia specific nuances","text":"","category":"section"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"Consider this (seemingly!) very trivial function:","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"function add(a,b)\n a + b\nend","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"Obviously, this function does nothing more than forward its arguments to +. From reading the source code above, we'd expect this to always behave the same as addition - and we'd probably be right! In julia though, a few subtleties come into play:","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"We don't know the type of the input arguments\nWe don't know how many different values each argument can take on\nWe don't know whether + is implemented on whatever we get in the first place\nIf it is, we don't know its implementation and we don't know whether it's correct/as we expect","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"So in reality, purely from reading the source code, we know nothing more about add other than \"passes its argument to +\". This sort of genericity is both a blessing and a curse, in that it allows anyone that has + defined on two types to call our function, while also making it devilishly difficult for us as developers to anticipate all possible behaviors that can occur.","category":"page"},{"location":"intro.html","page":"Introduction to PBT","title":"Introduction to PBT","text":"With property based testing, we should in theory be able to define a set of properties we'd like to hold on our code, for any object that can be passed into our add function. Users of our code who define a new type should be able to take those properties and check the behavior of their type & implementation in our function against the expected properties and find out (at least probabilistically) whether they've implemented the required functions correctly.","category":"page"},{"location":"Examples/composition.html#Composing-generators","page":"Composing Generators","title":"Composing generators","text":"","category":"section"},{"location":"Examples/composition.html#@composed","page":"Composing Generators","title":"@composed","text":"","category":"section"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"While Supposition.jl provides basic generators for a number of objects from Base, quite a lot of Julia code relies on the use of custom structs. At the innermost level, all Julia structs are composed of one or more of these basic types, like Int, String, Vector etc. Of course, we want to be able to generate & correctly shrink these custom structs as well, so how can this be done? Enter @composed, which can do exactly that. Here's how it's used:","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"using Supposition\n\nconst intgen = Data.Integers{Int}()\n\nmakeeven(x) = (x÷0x2)*0x2\n\neven_complex = @composed function complex_even(a=intgen, b=intgen)\n a = makeeven(a)\n b = makeeven(b)\n a + b*im\nend\nexample(even_complex, 5)","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"In essence, @composed takes a function that is given some generators, and ultimately returns a generator that runs the function on those given generators. As a full-fledged Possibility, you can of course do everything you'd expect to do with other Possibility objects from Supposition.jl, including using them as input to other @composed! This makes them a powerful tool for composing custom generators.","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"@check function all_complex_even(c=even_complex)\n iseven(real(c)) && iseven(imag(c))\nend\nnothing # hide","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"warning: Type stability\nThe inferred type of objects created by a generator from @composed is a best effort and may be wider than expected. E.g. if the input generators are non-const globals, it can easily happen that type inference falls back to Any. The same goes for other type instabilities and the usual best-practices surrounding type stability.","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"In addition, @composed defines the function given to it as well as a regular function, which means that you can call & reuse it however you like:","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"complex_even(1.0,2.0)","category":"page"},{"location":"Examples/composition.html#Filtering,-mapping,-and-other-combinators","page":"Composing Generators","title":"Filtering, mapping, and other combinators","text":"","category":"section"},{"location":"Examples/composition.html#filter","page":"Composing Generators","title":"filter","text":"","category":"section"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"Of course, manually marking, mapping or filtering inside of @composed is sometimes a bit too much. For these cases, all Possibility support filter and map, returning a new Data.Satisfying or Data.Map Possibility respectively:","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"using Supposition\n\nintgen = Data.Integers{UInt8}()\n\nf = filter(iseven, intgen)\n\nexample(f, 10)","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"Note that filtering is, in almost all cases, strictly worse than constructing the desired objects directly. For example, if the filtering predicate rejects too many examples from the input space, it can easily happen that no suitable examples can be found:","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"g = filter(>(typemax(UInt8)), intgen)\ntry # hide\nexample(g, 10)\ncatch e # hide\nBase.display_error(e) # hide\nend # hide\nnothing # hide","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"It is best to only filter when you're certain that the part of the state space you're filtering out is not substantial.","category":"page"},{"location":"Examples/composition.html#map","page":"Composing Generators","title":"map","text":"","category":"section"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"In order to make it easier to directly construct conforming instances, you can use map, transforming the output of one Possibility into a different object:","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"using Supposition\n\nintgen = Data.Integers{UInt8}()\nmakeeven(x) = (x÷0x2)*0x2\nm = map(makeeven, intgen)\n\nexample(m, 10)","category":"page"},{"location":"Examples/composition.html","page":"Composing Generators","title":"Composing Generators","text":"warning: Type stability\nThe inferred type of objects created by a generator from map is a best effort and may be wider than expected. Ensure your function f is easily inferrable to have good chances for mapping it to be inferable as well.","category":"page"},{"location":"faq.html#FAQ","page":"FAQ","title":"FAQ","text":"","category":"section"},{"location":"faq.html#Why-write-a-new-package?-What-about-PropCheck.jl?","page":"FAQ","title":"Why write a new package? What about PropCheck.jl?","text":"","category":"section"},{"location":"faq.html","page":"FAQ","title":"FAQ","text":"PropCheck.jl is based on the Haskell library Hedgehog, while Supposition.jl is based on Hypothesis. For a detailed look into the differences between these (as well as QuickCheck), I've written up a small comparison on my blog. Some understanding of property based testing is required, but the TL;DR is that the approaches taken by these two frameworks are fundamentally different.","category":"page"},{"location":"faq.html","page":"FAQ","title":"FAQ","text":"Originally, I was planning on only investigating what Hypothesis did differently from Hedgehog and incorporating the results of that investigation into PropCheck.jl, but once I learned how different the approaches truly are, I quickly realized that marrying the two would be more work and likely less fruitful than just porting Hypothesis directly. My resolve in that regard had only grown stronger after porting MiniThesis to Julia, which this package is ultimately also based on. So far, I have found the core of the package to be extremely solid, and I don't expect that to change.","category":"page"},{"location":"faq.html","page":"FAQ","title":"FAQ","text":"Some of the upsides I've noticed while using Supposition.jl so far is that it's much, MUCH easier to wrap your head around than PropCheck.jl ever was. Even after working on & thinking about Hedgehog & PropCheck.jl for more than 4 years, I still get lost in its internals. I don't see that happening with Supposition.jl; I can confidently say that I am (for the moment) able to keep the entire package in my head. Another big upside is that, as far as I can tell, Supposition.jl is much faster than PropCheck.jl, even after the latter received extensive type stability analysis and performance improvements out of necessity. I haven't done the same for Supposition.jl so far - be sure to check out the Benchmarks section for a direct comparison. Of course, only part of this is due to the underlying approach of Hypothesis vs. Hedgehog. Sticking to a much more functional & function-based implementation with PropCheck.jl is sure to hold the package back, and perhaps the situation would be different with a more data-oriented approach.","category":"page"},{"location":"faq.html#What-can-Supposition.jl-do-that-PropCheck.jl-can't?","page":"FAQ","title":"What can Supposition.jl do that PropCheck.jl can't?","text":"","category":"section"},{"location":"faq.html","page":"FAQ","title":"FAQ","text":"While there is a big overlap in capabilities between the two, there are a number of things that Supposition.jl can very easily do that would require a major rework of the internals of PropCheck.jl. Here's a small (incomplete) list:","category":"page"},{"location":"faq.html","page":"FAQ","title":"FAQ","text":"Shrink examples that lead to an error\nEasily use temporal & stateful property tests\nReproducibly replay previously found counterexamples (with caveats regarding external state)\nGenerate values based on the values that were put into a test, and have those values in turn shrink just as well as the values that were put in while preserving the invariants they were generated under.","category":"page"},{"location":"faq.html#What-feature-X-of-PropCheck.jl-corresponds-to-feature-Y-of-Supposition.jl?","page":"FAQ","title":"What feature X of PropCheck.jl corresponds to feature Y of Supposition.jl?","text":"","category":"section"},{"location":"faq.html","page":"FAQ","title":"FAQ","text":"The following is a short overview/feature comparison between PropCheck.jl and Supposition.jl. It may not be a perfect match for all functionality - be sure to check the documentation of each respective feature to learn about their minute differences!","category":"page"},{"location":"faq.html","page":"FAQ","title":"FAQ","text":"Feature PropCheck.jl Supposition.jl\nChecking Interface check(, ) @check prop() or @check function prop(arg=, ...) # ...use args... end\nmap map(, ) map(, )\nfilter filter(, ) filter(, )\ncomposition through combination interleave(integ::AbstractIntegrated...) @composed function comp(a=, ...) # ...use args... end\nVectors PropCheck.vector(len::AbstractIntegrated, objs::AbstractIntegrated) `Data.Vectors(objs::Data.Possibility; minsize=0, maxsize=...)\nTuples PropCheck.tuple(len::AbstractIntegrated, objs::AbstractIntegrated) Currently unsupported, but could be added in a PR\nIntegers PropCheck.inegint/PropCheck.iposint Data.Integers(min, max)\nFloating point PropCheck.ifloat(T) and its variants Data.Floats{T}(; infs=, nans=)\nStrings PropCheck.str(len::AbstractIntegrated, alphabet::AbstractIntegrated) Data.Text(::Possibility{Char}; min_len=0, max_len=...)\nStateful generation IntegratedOnce Unsupported due to deterministic replaying of finite generators being tricky\n IntegratedFiniteIterator Unsupported due to deterministic replaying of finite generators being tricky\n IntegratedLengthBounded Unsupported due to deterministic replaying of finite generators being tricky\n IntegratedChain Unsupported due to deterministic replaying of finite generators being tricky\n PropCheck.iunique Unsupported due to deterministic replaying of finite generators being tricky\nGeneration of constant data PropCheck.iconst(x) Data.Just(x)\nGeneration from Collections IntegratedRange(x)/PropCheck.isample Data.SampledFrom(x)\nGeneration of shrinkable constants IntegratedVal(x) Unsupported until custom shrinking functions are added, see #25\nType-based generation PropCheck.itype Unsupported for now, see #21 for more information (it's coming though! And smarter than PropCheck.jl too ;) ).","category":"page"},{"location":"faq.html#Can-I-use-Supposition.jl-to-test-an-MIT/proprietary/other-licensed-project?","page":"FAQ","title":"Can I use Supposition.jl to test an MIT/proprietary/other licensed project?","text":"","category":"section"},{"location":"faq.html","page":"FAQ","title":"FAQ","text":"Yes!","category":"page"},{"location":"faq.html","page":"FAQ","title":"FAQ","text":"Supposition.jl is licensed under the EUPLv1.2, which means that modifications to Supposition.jl also need to be licensed under the EUPLv1.2 (and various other obligations). However, simply using Supposition.jl in a testsuite of any project has no influence on the license of that project (or even the testsuite), because the EUPLv1.2 is not a \"viral\" copyleft license. There is no risk of having to license your MIT/proprietary/other licensed project under the EUPLv1.2, because under European law (which the EUPLv1.2 defaults to, even if you're outside of the EU as either licensor or licensee) linking computer programs for the purposes of interoperability is exempt from copyright, which is the cause of \"virality\" in other licenses.","category":"page"},{"location":"faq.html","page":"FAQ","title":"FAQ","text":"For more information about using Supposition.jl in a commercial setting, see EUPL and Proprietary / Commercial use, written by Patrice-Emmanuel Schmitz, who has written extensive analysis of the EUPL under EU law. For more information about the EUPL in general see European Union Public Licence Guidelines. For more information about copyright in the EU in general, see Directive 2009/24/EC.","category":"page"},{"location":"api.html#Documentation-Reference","page":"API Reference","title":"Documentation Reference","text":"","category":"section"},{"location":"api.html","page":"API Reference","title":"API Reference","text":"This section contains a complete reference of everything Supposition.jl contains, on one page. This is not a devdocs section, but a reference, for quick lookups of what something does, without having to hunt for the exact definition in the source code. A proper devdocs section with a high level introduction will be added at a later date.","category":"page"},{"location":"api.html","page":"API Reference","title":"API Reference","text":"warning: Stability\nThe entries written on this page are automatically generated and DO NOT represent the currently supported API surface. Feel free to use anything you can find here, but be aware that just because it's listed here, does not mean it's covered under semver (though it may be - check Userfacing API if you're unsure).","category":"page"},{"location":"api.html#Data-reference","page":"API Reference","title":"Data reference","text":"","category":"section"},{"location":"api.html","page":"API Reference","title":"API Reference","text":"The Data module contains most everyday objects you're going to use when writing property based tests with Supposition.jl. For example, the basic generators for integers, strings, floating point values etc. are defined here. Everything listed in this section is considered supported under semver.","category":"page"},{"location":"api.html","page":"API Reference","title":"API Reference","text":"Modules = [Data]\nOrder = [:function, :type]","category":"page"},{"location":"api.html#Functions","page":"API Reference","title":"Functions","text":"","category":"section"},{"location":"api.html","page":"API Reference","title":"API Reference","text":"Modules = [Data]\nOrder = [:function]","category":"page"},{"location":"api.html#Base.:|-Tuple{Supposition.Data.Possibility, Supposition.Data.Possibility}-api","page":"API Reference","title":"Base.:|","text":"|(::Possibility{T}, ::Possibility{S}) where {T,S} -> OneOf{Union{T,S}}\n\nCombine two Possibility into one, sampling uniformly from either.\n\nIf either of the two arguments is a OneOf, the resulting object acts as if all original non-OneOf had be given to OneOf instead. That is, e.g. OneOf(a, b) | c will act like OneOf(a,b,c).\n\nSee also OneOf.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Base.filter-Tuple{Any, Supposition.Data.Possibility}-api","page":"API Reference","title":"Base.filter","text":"filter(f, pos::Possibility)\n\nFilter the output of produce! on pos by applying the predicate f.\n\nnote: No stalling\nIn order not to stall generation of values, this will not try to produce a value from pos forever, but reject the testcase after some attempts.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Base.map-Tuple{Any, Supposition.Data.Possibility}-api","page":"API Reference","title":"Base.map","text":"map(f, pos::Possibility)\n\nApply f to the result of calling produce! on pos (lazy mapping).\n\nEquivalent to calling Map(pos, f).\n\nSee also Map.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.Data.BitIntegers-Tuple{}-api","page":"API Reference","title":"Supposition.Data.BitIntegers","text":"BitIntegers() <: Possibility{Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}}\n\nA Possibility for generating all possible bitintegers with fixed size.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.Data.bind-Tuple{Any, Supposition.Data.Possibility}-api","page":"API Reference","title":"Supposition.Data.bind","text":"bind(f, pos::Possibility)\n\nMaps the output of produce! on pos through f, and calls produce! on the result again. f is expected to take a value and return a Possibility.\n\nEquivalent to calling Bind(pos, f).\n\nSee also Bind.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.Data.postype-Tuple{P} where P<:Supposition.Data.Possibility-api","page":"API Reference","title":"Supposition.Data.postype","text":"postype(::P) where P <: Possibility\n\nGives the type of objects this Possibility object will generate.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.Data.postype-Union{Tuple{Type{_P}}, Tuple{_P}, Tuple{T}} where {T, _P<:Supposition.Data.Possibility{T}}-api","page":"API Reference","title":"Supposition.Data.postype","text":"postype(::Type{P<:Possibility})\n\nGives the type of objects this Possibility type will generate.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.Data.produce!-api","page":"API Reference","title":"Supposition.Data.produce!","text":"produce!(tc::TestCase, pos::Possibility{T}) -> T\n\nProduces a value from the given Possibility, recording the required choices in the TestCase tc.\n\nThis needs to be implemented for custom Possibility objects, passing the given tc to any inner requirements directly.\n\nSee also Supposition.produce!\n\ntip: Examples\nYou should not call this function when you have a Possibility and want to inspect what an object produced by that Possibility looks like - use example for that instead.\n\n\n\n\n\n","category":"function"},{"location":"api.html#Supposition.Data.recursive-Tuple{Any, Supposition.Data.Possibility}-api","page":"API Reference","title":"Supposition.Data.recursive","text":"recursive(f, pos::Possibility; max_layers=5)\n\nRecursively expand pos into deeper nested Possibility by repeatedly passing pos itself to f. f returns a new Possibility, which is then passed into f again until the maximum depth is achieved.\n\nEquivalent to calling Recursive(pos, f).\n\nSee also Recursive.\n\n\n\n\n\n","category":"method"},{"location":"api.html","page":"API Reference","title":"API Reference","text":"Modules = [Data]\nOrder = [:type]\nFilter = t -> begin\n t != Supposition.Data.Possibility\nend","category":"page"},{"location":"api.html#Supposition.Data.AsciiCharacters-api","page":"API Reference","title":"Supposition.Data.AsciiCharacters","text":"AsciiCharacters() <: Possibility{Char}\n\nA Possibility of producing arbitrary Char instances that are isascii. More efficient than filtering Characters.\n\njulia> using Supposition\n\njulia> ascii = Data.AsciiCharacters()\n\njulia> example(ascii, 5)\n5-element Vector{Char}:\n '8': ASCII/Unicode U+0038 (category Nd: Number, decimal digit)\n 'i': ASCII/Unicode U+0069 (category Ll: Letter, lowercase)\n 'R': ASCII/Unicode U+0052 (category Lu: Letter, uppercase)\n '\\f': ASCII/Unicode U+000C (category Cc: Other, control)\n '>': ASCII/Unicode U+003E (category Sm: Symbol, math)\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Bind-api","page":"API Reference","title":"Supposition.Data.Bind","text":"Bind(source::Possibility, f)\n\nBinds f to source, i.e., on produce!(::Bind, ::TestCase) this calls produce! on source, the result of which is passed to f, the output of which will be used as input to produce! again.\n\nIn other words, f takes a value produce!d by source and gives back a Possibility that is then immediately produce!d from.\n\nEquivalent to bind(f, source).\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Booleans-api","page":"API Reference","title":"Supposition.Data.Booleans","text":"Booleans() <: Possibility{Bool}\n\nA Possibility for sampling boolean values.\n\njulia> using Supposition\n\njulia> bools = Data.Booleans()\n\njulia> example(bools, 4)\n4-element Vector{Bool}:\n 0\n 1\n 0\n 1\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Characters-api","page":"API Reference","title":"Supposition.Data.Characters","text":"Characters(;valid::Bool = false) <: Possibility{Char}\n\nA Possibility of producing arbitrary Char instances.\n\nwarning: Unicode\nThis will produce! ANY possible Char by default, not just valid unicode codepoints! To only produce valid unicode codepoints, pass valid=true as a keyword argument.\n\njulia> using Supposition\n\njulia> chars = Data.Characters()\n\njulia> example(chars, 5)\n5-element Vector{Char}:\n '⠺': Unicode U+283A (category So: Symbol, other)\n '𰳍': Unicode U+30CCD (category Lo: Letter, other)\n '\\U6ec9c': Unicode U+6EC9C (category Cn: Other, not assigned)\n '\\U1a05c5': Unicode U+1A05C5 (category In: Invalid, too high)\n '𓂫': Unicode U+130AB (category Lo: Letter, other)\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Dicts-api","page":"API Reference","title":"Supposition.Data.Dicts","text":"Dicts(keys::Possibility, values::Possibility; min_size=0, max_size=10_000)\n\nA Possibility for generating Dict objects. The keys are drawn from keys, while the values are drawn from values. min_size/max_size control the number of objects placed into the resulting Dict, respectively.\n\njulia> dicts = Data.Dicts(Data.Integers{UInt8}(), Data.Integers{Int8}(); max_size=3);\n\njulia> example(dicts)\nDict{UInt8, Int8} with 2 entries:\n 0x54 => -29\n 0x1f => -28\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Floats-Tuple{}-api","page":"API Reference","title":"Supposition.Data.Floats","text":"Floats(;nans=true, infs=true) <: Possibility{Union{Float64,Float32,Float16}}\n\nA catch-all for generating instances of all three IEEE floating point types.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.Data.Floats-api","page":"API Reference","title":"Supposition.Data.Floats","text":"Floats{T <: Union{Float16,Float32,Float64}}(;infs=true, nans=true) <: Possibility{T}\n\nA Possibility for sampling floating point values.\n\nThe keyword infs controls whether infinities can be generated. nans controls whether any NaN (signaling & quiet) will be generated.\n\nwarning: Inf, Nan\nThis possibility will generate any valid instance, including positive and negative infinities, signaling and quiet NaNs and every possible float.\n\njulia> using Supposition\n\njulia> floats = Data.Floats{Float16}()\n\njulia> example(floats, 5)\n5-element Vector{Float16}:\n -8.3e-6\n 1.459e4\n 3.277\n NaN\n -0.0001688\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Integers-api","page":"API Reference","title":"Supposition.Data.Integers","text":"Integers(minimum::T, maximum::T) <: Possibility{T <: Integer}\nIntegers{T}() <: Possibility{T <: Integer}\n\nA Possibility representing drawing integers from [minimum, maximum]. The second constructors draws from the entirety of T.\n\nProduced values are of type T.\n\njulia> using Supposition\n\njulia> is = Data.Integers{Int}()\n\njulia> example(is, 5)\n5-element Vector{Int64}:\n -5854403925308846160\n 4430062772779972974\n -9995351034504801\n 2894734754389242339\n -6640496903289665416\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Just-api","page":"API Reference","title":"Supposition.Data.Just","text":"Just(value::T) <: Possibility{T}\n\nA Possibility that always produces value.\n\nwarning: Mutable Data\nThe source object given to this Just is not copied when produce! is called. Be careful with mutable data!\n\njulia> using Supposition\n\njulia> three = Data.Just(3)\n\njulia> example(three, 3)\n3-element Vector{Int64}:\n 3\n 3\n 3\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Map-api","page":"API Reference","title":"Supposition.Data.Map","text":"Map(source::Possibility, f) <: Possibility\n\nA Possibility representing mapping values from source through f.\n\nEquivalent to calling map(f, source).\n\nThe pre-calculated return type of Map is a best effort and may be wider than necessary.\n\njulia> using Supposition\n\njulia> makeeven(x) = (x ÷ 2) * 2\n\njulia> pos = map(makeeven, Data.Integers{Int8}())\n\njulia> all(iseven, example(pos, 10_000))\ntrue\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.OneOf-api","page":"API Reference","title":"Supposition.Data.OneOf","text":"OneOf(pos::Possibility...) <: Possibility\n\nA Possibility able to generate any of the examples one of the given Possibility can produce. The given Possibility are sampled from uniformly.\n\nAt least one Possibility needs to be given to OneOf.\n\npostype(::OneOf) is inferred as a best effort, and may be wider than necessary.\n\nOneOf can also be constructed through use of a | b on Possibility. Constructed in this way, if either a or b is already a OneOf, the resulting OneOf acts as if it had been given the original Possibility objects in the first place. That is, OneOf(a, b) | c acts like OneOf(a, b, c).\n\nSee also WeightedNumbers and WeightedSample.\n\njulia> of = Data.OneOf(Data.Integers{Int8}(), Data.Integers{UInt8}());\n\njulia> Data.postype(of)\nUnion{Int8, UInt8}\n\njulia> ex = map(of) do i\n (i, typeof(i))\n end;\n\njulia> example(ex)\n(-83, Int8)\n\njulia> example(ex)\n(0x9f, UInt8)\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Pairs-api","page":"API Reference","title":"Supposition.Data.Pairs","text":"Pairs(first::Possibility{T}, second::Possibility{S}) where {T,S} <: Possibility{Pair{T,S}}\n\nA Possibility for producing a => b pairs. a is produced by first, b is produced by second.\n\njulia> p = Data.Pairs(Data.Integers{UInt8}(), Data.Floats{Float64}());\n\njulia> example(p, 4)\n4-element Vector{Pair{UInt8, Float64}}:\n 0x41 => 4.1183566661848205e-230\n 0x48 => -2.2653631095108555e-119\n 0x2a => -6.564396855333643e224\n 0xec => 1.9330751262581671e-53\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Recursive-api","page":"API Reference","title":"Supposition.Data.Recursive","text":"Recursive(base::Possibility, extend; max_layers::Int=5) <: Possibility{T}\n\nA Possibility for generating recursive data structures. base is the basecase of the recursion. extend is a function returning a new Possibility when given a Possibility, called to recursively expand a tree starting from base. The returned Possibility is fed back into extend again, expanding the recursion by one layer.\n\nmax_layers designates the maximum layers Recursive should keep track of. This must be at least 1, so that at least the base case can always be generated. Note that this implies extend will be used at most max_layers-1 times, since the base case of the recursion will not be wrapped.\n\nEquivalent to calling recursive(extend, base).\n\nExamples\n\njulia> base = Data.Integers{UInt8}()\n\njulia> wrap(pos) = Data.Vectors(pos; min_size=2, max_size=3)\n\njulia> rec = Data.recursive(wrap, base; max_layers=3);\n\njulia> Data.postype(rec) # the result is formatted here for legibility\nUnion{UInt8,\n Vector{UInt8},\n Vector{Union{UInt8, Vector{UInt8}}}\n}\n\njulia> example(rec)\n0x31\n\njulia> example(rec)\n2-element Vector{Union{UInt8, Vector{UInt8}}}:\n UInt8[0xa9, 0xb4]\n 0x9b\n\njulia> example(rec)\n2-element Vector{UInt8}:\n 0xbd\n 0x25\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.SampledFrom-api","page":"API Reference","title":"Supposition.Data.SampledFrom","text":"SampledFrom(collection) <: Possibility{eltype(collection)}\n\nA Possibility for sampling uniformly from collection.\n\ncollection, as well as its eachindex, is assumed to be indexable.\n\nwarning: Mutable Data\nThe source objects from the collection given to this SampledFrom is not copied when produce! is called. Be careful with mutable data!\n\ntip: Sampling from `String`\nTo sample from a String, you can collect the string first to get a Vector{Char}. This is necessary because Strings use the variable-length UTF-8 encoding, which isn't arbitrarily indexable in constant time.\n\njulia> using Supposition\n\njulia> sampler = Data.SampledFrom([1, 1, 1, 2])\n\njulia> example(sampler, 4)\n4-element Vector{Int64}:\n 1\n 1\n 2\n 1\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Satisfying-api","page":"API Reference","title":"Supposition.Data.Satisfying","text":"Satisfying(source::Possibility, pred) <: Possibility\n\nA Possibility representing values from source fulfilling pred.\n\nEquivalent to calling filter(f, source).\n\njulia> using Supposition\n\njulia> pos = filter(iseven, Data.Integers{Int8}())\n\njulia> all(iseven, example(pos, 10_000))\ntrue\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Text-api","page":"API Reference","title":"Supposition.Data.Text","text":"Text(alphabet::Possibility{Char}; min_len=0, max_len=10_000) <: Possibility{String}\n\nA Possibility for producing Strings containing Chars of a given alphabet.\n\njulia> using Supposition\n\njulia> text = Data.Text(Data.AsciiCharacters(); max_len=15)\n\njulia> example(text, 5)\n5-element Vector{String}:\n \"U\\x127lxf\"\n \"hm\\x172SJ-(\"\n \"h`\\x03\\0\\x01[[il\"\n \"\\x0ep4\"\n \"9+Hk3 ii\\x1eT\"\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.Vectors-api","page":"API Reference","title":"Supposition.Data.Vectors","text":"Vectors(elements::Possibility{T}; min_size=0, max_size=10_000) <: Possibility{Vector{T}}\n\nA Possibility representing drawing vectors with length l in min_size <= l <= max_size, holding elements of type T.\n\nmin_size and max_size must be positive numbers, with min_size <= max_size.\n\njulia> using Supposition\n\njulia> vs = Data.Vectors(Data.Floats{Float16}(); max_size=5)\n\njulia> example(vs, 3)\n3-element Vector{Vector{Float16}}:\n [9.64e-5, 9.03e3, 0.04172, -0.0003352]\n [9.793e-5, -2.893, 62.62, 0.0001961]\n [-0.007023, NaN, 3.805, 0.1943]\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.WeightedNumbers-api","page":"API Reference","title":"Supposition.Data.WeightedNumbers","text":"WeightedNumbers(weights::Vector{Float64}) <: Possibility{Int}\n\nSample the numbers from 1:length(weights), each with a weight of weights[i].\n\nThe weights may be any number > 0.0.\n\nSee also OneOf.\n\njulia> using Supposition\n\njulia> bn = Data.WeightedNumbers([1.0, 1.0, 3.0]);\n\njulia> example(Data.Vectors(bn; min_size=3, max_size=15), 5)\n5-element Vector{Vector{Int64}}:\n [3, 2, 3, 3, 2, 3, 3]\n [1, 1, 1, 2, 1, 3, 1, 3]\n [2, 3, 3, 3, 3, 3, 3, 1, 1, 3, 3, 3]\n [3, 3, 2, 3, 3]\n [1, 3, 3, 3, 2, 2]\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Data.WeightedSample-api","page":"API Reference","title":"Supposition.Data.WeightedSample","text":"WeightedSample{T}(colllection, weights::Vector{Float64}) <: Possibility{T}\n\nDraw an element from the indexable collection, biasing the drawing process by assigning each index i of col the weight at weights[i].\n\nlength of col and weights must be equal and eachindex(col) must be indexable.\n\nSee also OneOf.\n\ntip: Sampling from `String`\nTo sample from a String, you can collect the string first to get a Vector{Char}. This is necessary because Strings use the variable-length UTF-8 encoding, which isn't arbitrarily indexable in constant time.\n\njulia> bs = Data.WeightedSample([\"foo\", \"bar\", \"baz\"], [3.0, 1.0, 1.0]);\n\njulia> example(bs, 10)\n10-element Vector{String}:\n \"foo\"\n \"foo\"\n \"bar\"\n \"baz\"\n \"baz\"\n \"foo\"\n \"bar\"\n \"foo\"\n \"foo\"\n \"bar\"\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition-reference","page":"API Reference","title":"Supposition reference","text":"","category":"section"},{"location":"api.html","page":"API Reference","title":"API Reference","text":"Modules = [Supposition]\nOrder = [:macro, :function, :type, :constant]","category":"page"},{"location":"api.html","page":"API Reference","title":"API Reference","text":"Modules = [Supposition]\nOrder = [:macro, :function, :type, :constant]\nFilter = t -> begin\n t != Supposition.ExampleDB\nend","category":"page"},{"location":"api.html#Supposition.@check-Tuple-api","page":"API Reference","title":"Supposition.@check","text":"@check [key=val]... function...\n\nThe main way to declare & run a property based test. Called like so:\n\njulia> using Supposition, Supposition.Data\n\njulia> Supposition.@check [options...] function foo(a = Data.Text(Data.Characters(); max_len=10))\n length(a) > 8\n end\n\nSupported options, passed as key=value:\n\nrng::Random.AbstractRNG: Pass an RNG to use. Defaults to Random.Xoshiro(rand(Random.RandomDevice(), UInt)).\nmax_examples::Int: The maximum number of generated examples that are passed to the property.\nbroken::Bool: Mark a property that should pass but doesn't as broken, so that failures are not counted.\nrecord::Bool: Whether the result of the invocation should be recorded with any parent testsets.\ndb: Either a Boolean (true uses a fallback database, false stops recording examples) or an ExampleDB.\nconfig: A CheckConfig object that will be used as a default for all previous options. Options that are passed explicitly to @check will override whatever is provided through config.\n\nThe arguments to the given function are expected to be generator strategies. The names they are bound to are the names the generated object will have in the test. These arguments will be shown should the property fail!\n\nExtended help\n\nReusing existing properties\n\nIf you already have a predicate defined, you can also use the calling syntax in @check. Here, the generator is passed purely positionally to the given function; no argument name is necessary.\n\njulia> using Supposition, Supposition.Data\n\njulia> isuint8(x) = x isa UInt8\n\njulia> intgen = Data.Integers{UInt8}()\n\njulia> Supposition.@check isuint8(intgen)\n\nPassing a custom RNG\n\nIt is possible to optionally give a custom RNG object that will be used for random data generation. If none is given, Xoshiro(rand(Random.RandomDevice(), UInt)) is used instead.\n\njulia> using Supposition, Supposition.Data, Random\n\n# use a custom Xoshiro instance\njulia> Supposition.@check rng=Xoshiro(1234) function foo(a = Data.Text(Data.Characters(); max_len=10))\n length(a) > 8\n end\n\nwarning: Hardware RNG\nBe aware that you cannot pass a hardware RNG to @check directly. If you want to randomize based on hardware entropy, seed a copyable RNG like Xoshiro from your hardware RNG and pass that to @check instead. The RNG needs to be copyable for reproducibility.\n\nAdditional Syntax\n\nIn addition to passing a whole function like above, the following syntax are also supported:\n\ntext = Data.Text(Data.AsciiCharacters(); max_len=10)\n\n# If no name is needed, use an anonymous function\n@check (a = text) -> a*a\n@check (a = text,) -> \"foo: \"*a\n@check (a = text, num = Data.Integers(0,10)) -> a^num\n\n# ..or give the anonymous function a name too - works with all three of the above\n@check build_sentence(a = text, num = Data.Floats{Float16}()) -> \"The $a is $num!\"\nbuild_sentence(\"foo\", 0.5) # returns \"The foo is 0.5!\"\n\nwarning: Replayability\nWhile you can pass an anonymous function to @check, be aware that doing so may hinder replayability of found test cases when surrounding invocations of @check are moved. Only named functions are resistant to this.\n\n\n\n\n\n","category":"macro"},{"location":"api.html#Supposition.@composed-Tuple{Expr}-api","page":"API Reference","title":"Supposition.@composed","text":"@composed\n\nA way to compose multiple Possibility into one, by applying a function.\n\nThe return type is inferred as a best-effort!\n\nUsed like so:\n\njulia> using Supposition, Supposition.Data\n\njulia> text = Data.Text(Data.AsciiCharacters(); max_len=10)\n\njulia> gen = Supposition.@composed function foo(a = text, num=Data.Integers(0, 10))\n lpad(num, 2) * \": \" * a\n end\n\njulia> example(gen)\n\" 8: giR2YL\\rl\"\n\nIn addition to passing a whole function like above, the following syntax are also supported:\n\n# If no name is needed, use an anonymous function\ndouble_up = @composed (a = text) -> a*a\nprepend_foo = @composed (a = text,) -> \"foo: \"*a\nexpo_str = @composed (a = text, num = Data.Integers(0,10)) -> a^num\n\n# ..or give the anonymous function a name too - works with all three of the above\nsentence = @composed build_sentence(a = text, num = Data.Floats{Float16}()) -> \"The $a is $num!\"\nbuild_sentence(\"foo\", 0.5) # returns \"The foo is 0.5!\"\n\n# or compose a new generator out of an existing function\nmy_func(str, number) = number * \"? \" * str\nask_number = @composed my_func(text, num)\n\n\n\n\n\n","category":"macro"},{"location":"api.html#Supposition.Data.produce!-Tuple{Supposition.Data.Possibility}-api","page":"API Reference","title":"Supposition.Data.produce!","text":"produce!(p::Possibility{T}) -> T\n\nProduces a value from the given Possibility, recording the required choices in the currently active TestCase.\n\nwarning: Callability\nThis can only be called while a testcase is currently being examined or an example for a Possibility is being actively generated. It is ok to call this inside of @composed or @check, as well as any functions only intended to be called from one of those places.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.adjust-Tuple{Supposition.TestState, Supposition.Attempt}-api","page":"API Reference","title":"Supposition.adjust","text":"adjust(ts::TestState, attempt)\n\nAdjust ts by testing for the choices given by attempt.\n\nReturns whether attempt was by some measure better than the previously best attempt.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.assemble-Union{Tuple{T}, Tuple{I}, Tuple{Type{T}, I, I, I}} where {I, T<:Union{Float16, Float32, Float64}}-api","page":"API Reference","title":"Supposition.assemble","text":"assemble(::T, sign::I, expo::I, frac::I) where {I, T <: Union{Float16, Float32, Float64}} -> T\n\nAssembles sign, expo and frac arguments into the floating point number of type T it represents. sizeof(T) must match sizeof(I).\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.assume!-Tuple{Bool}-api","page":"API Reference","title":"Supposition.assume!","text":"assume!(precondition::Bool)\n\nIf this precondition is not met, abort the test and mark the currently running testcase as invalid.\n\nwarning: Callability\nThis can only be called while a testcase is currently being examined or an example for a Possibility is being actively generated. It is ok to call this inside of @composed or @check, as well as any functions only intended to be called from one of those places.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.assume!-Tuple{Supposition.TestCase, Bool}-api","page":"API Reference","title":"Supposition.assume!","text":"assume!(::TestCase, precondition::Bool)\n\nReject this TestCase if precondition is false.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.choice!-Tuple{Supposition.TestCase, UInt64}-api","page":"API Reference","title":"Supposition.choice!","text":"choice!(tc::TestCase, n)\n\nForce a number of choices to occur, taking from the existing prefix first. If the prefix is exhausted, draw from [zero(n), n] instead.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.consider-Tuple{Supposition.TestState, Supposition.Attempt}-api","page":"API Reference","title":"Supposition.consider","text":"consider(ts::TestState, attempt::Attempt) -> Bool\n\nReturns whether the given choices are a conceivable example for the testcase given by ts.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.err_choices-Tuple{Supposition.TestState}-api","page":"API Reference","title":"Supposition.err_choices","text":"err_choices\n\nReturn the choices that led to the recorded error, if any. If none, return Nothing.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.err_less-api","page":"API Reference","title":"Supposition.err_less","text":"err_less(e1::E, e2::E) where E\n\nA comparison function for exceptions, used when encountering an error in a property. Returns true if e1 is considered to be \"easier\" or \"simpler\" than e2. Only definable when both e1 and e2 have the same type.\n\nThis is optional to implement, but may be beneficial for shrinking counterexamples leading to an error with rich metadata, in which case err_less will be used to compare errors of the same type from different counterexamples. In particular, this function will likely be helpful for errors with metadata that is far removed from the input that caused the error itself, but would nevertheless be helpful when investigating the failure.\n\nnote: Coincidental Errors\nThere may also be situations where defining err_less won't help to find a smaller counterexample if the cause of the error is unrelated to the choices taken during generation. For instance, this is the case when there is no network connection and a Sockets.DNSError is thrown during the test, or there is a network connection but the host your program is trying to connect to does not have an entry in DNS.\n\n\n\n\n\n","category":"function"},{"location":"api.html#Supposition.event!-api","page":"API Reference","title":"Supposition.event!","text":"event!(obj)\nevent!(label::AbstractString, obj)\n\nRecord obj as an event in the current testcase that occured while running your property. If no label is given, a default one will be chosen.\n\n\n\n\n\n","category":"function"},{"location":"api.html#Supposition.example-Tuple{Supposition.Data.Possibility}-api","page":"API Reference","title":"Supposition.example","text":"example(pos::Possibility; tries=100_000, generation::Int)\n\nGenerate an example for the given Possibility.\n\nexample tries to have pos produce an example tries times and throws an error if pos doesn't produce one in that timeframe. generation indicates how \"late\" in a usual run of @check the example might have been generated.\n\nUsage:\n\njulia> using Supposition, Supposition.Data\n\njulia> example(Data.Integers(0, 10))\n7\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.example-Union{Tuple{T}, Tuple{Supposition.Data.Possibility{T}, Integer}} where T-api","page":"API Reference","title":"Supposition.example","text":"example(gen::Possibility, n::Integer; tries=100_000)\n\nGenerate n examples for the given Possibility. Each example is given tries attempts to generate. If any fail, the entire process is aborted.\n\njulia> using Supposition, Supposition.Data\n\njulia> is = Data.Integers(0, 10);\n\njulia> example(is, 10)\n10-element Vector{Int64}:\n 9\n 1\n 4\n 4\n 7\n 4\n 6\n 10\n 1\n 8\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.find_user_error_frame-api","page":"API Reference","title":"Supposition.find_user_error_frame","text":"find_user_error_frame(err, trace)\n\nTry to heuristically guess where an error was actually coming from.\n\nFor example, ErrorException is (generally) thrown from the error function, which would always report the same location if we'd naively take the first frame of the trace. This tries to be a bit smarter (but still fairly conservative) and return something other than the first frame for a small number of known error-throwing functions.\n\n\n\n\n\n","category":"function"},{"location":"api.html#Supposition.find_user_stack_depth-Tuple{Any}-api","page":"API Reference","title":"Supposition.find_user_stack_depth","text":"find_user_stack_depth\n\nReturn a heuristic guess for how many functions deep in user code an error was thrown. Falls back to the full length of the stacktrace.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.for_choices-Tuple{Vector{UInt64}, Random.AbstractRNG, UInt64, Int64}-api","page":"API Reference","title":"Supposition.for_choices","text":"for_choices(prefix; rng=Random.default_rng())\n\nCreate a TestCase for a given set of known choices.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.forced_choice!-Tuple{Supposition.TestCase, UInt64}-api","page":"API Reference","title":"Supposition.forced_choice!","text":"forced_choice(tc::TestCase, n::UInt64)\n\nInsert a definite choice in the choice sequence.\n\nNote that all integrity checks happen here!\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.generate!-Tuple{Supposition.TestState}-api","page":"API Reference","title":"Supposition.generate!","text":"generate(ts::TestState)\n\nTry to generate an example that falsifies the property given to ts.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.num_testcases-Tuple{Supposition.SuppositionReport}-api","page":"API Reference","title":"Supposition.num_testcases","text":"num_testcases(sr::SuppositionReport)\n\nReturns the number of valid TestCases that were attempted during this run.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.reject!-Tuple{}-api","page":"API Reference","title":"Supposition.reject!","text":"reject!()\n\nReject the current testcase as invalid, meaning the generated example should not be considered as producing a valid counterexample.\n\nwarning: Callability\nThis can only be called while a testcase is currently being examined or an example for a Possibility is being actively generated. It is ok to call this inside of @composed or @check, as well as any functions only intended to be called from one of those places.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.reject-Tuple{Supposition.TestCase}-api","page":"API Reference","title":"Supposition.reject","text":"reject(::TestCase)\n\nMark this test case as invalid.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.run-Tuple{Supposition.TestState}-api","page":"API Reference","title":"Supposition.run","text":"run(ts::TestState)\n\nRun the checking algorithm on ts, generating values until we should stop, targeting the score we want to target on and finally shrinking the result.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.should_keep_generating-Tuple{Supposition.TestState}-api","page":"API Reference","title":"Supposition.should_keep_generating","text":"should_keep_generating(ts::TestState)\n\nWhether ts should keep generating new test cases, or whether ts is finished.\n\ntrue returned here means that the given property is not trivial, there is no result yet and we have room for more examples.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.shrink_redistribute-Tuple{Supposition.TestState, Supposition.Attempt, UInt64}-api","page":"API Reference","title":"Supposition.shrink_redistribute","text":"shrink_redistribute(ts::TestState, attempt::Attempt, k::UInt)\n\nTry to shrink attempt by redistributing value between two elements length k apart.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.shrink_reduce-Tuple{Supposition.TestState, Supposition.Attempt}-api","page":"API Reference","title":"Supposition.shrink_reduce","text":"shrink_reduce(::TestState, attempt::Attempt)\n\nTry to shrink attempt by making the elements smaller.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.shrink_remove-Tuple{Supposition.TestState, Supposition.Attempt, UInt64}-api","page":"API Reference","title":"Supposition.shrink_remove","text":"shrink_remove(ts::TestState, attempt::Attempt, k::UInt)\n\nTry to shrink attempt by removing k elements at a time\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.shrink_sort-Tuple{Supposition.TestState, Supposition.Attempt, UInt64}-api","page":"API Reference","title":"Supposition.shrink_sort","text":"shrink_sort(::TestState, attempt::Attempt, k::UInt)\n\nTry to shrink attempt by sorting k contiguous elements at a time.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.shrink_swap-Tuple{Supposition.TestState, Supposition.Attempt, UInt64}-api","page":"API Reference","title":"Supposition.shrink_swap","text":"shrink_swap(::TestState, attempt::Attempt, k::UInt)\n\nTry to shrink attempt by swapping two elements length k apart.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.shrink_zeros-Tuple{Supposition.TestState, Supposition.Attempt, UInt64}-api","page":"API Reference","title":"Supposition.shrink_zeros","text":"shrink_zeros(::TestSTate, attempt::Attempt, k::UInt)\n\nTry to shrink attempt by setting k elements at a time to zero.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.target!-Tuple{Float64}-api","page":"API Reference","title":"Supposition.target!","text":"target!(score)\n\nUpdate the currently running testcase to track the given score as its target.\n\nscore must be convertible to a Float64.\n\nwarning: Multiple Updates\nThis score can only be set once! Repeated calls will be ignored.\n\nwarning: Callability\nThis can only be called while a testcase is currently being examined or an example for a Possibility is being actively generated. It is ok to call this inside of @composed or @check, as well as any functions only intended to be called from one of those places.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.target!-Tuple{Supposition.TestCase, Float64}-api","page":"API Reference","title":"Supposition.target!","text":"target!(tc::TestCase, score::Float64)\n\nUpdate tc to use score as the score this TestCase achieves during optimization.\n\nwarning: Multiple Updates\nThis score can only be set once! Repeated calls will be ignored.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.target!-Tuple{Supposition.TestState}-api","page":"API Reference","title":"Supposition.target!","text":"target!(ts::TestState)\n\nIf ts has a target to go towards set, this will try to climb towards that target by adjusting the choice sequence until ts shouldn't generate anymore.\n\nIf ts is currently tracking an error it encountered, it will try to minimize the stacktrace there instead.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.tear-Tuple{T} where T<:Union{Float16, Float32, Float64}-api","page":"API Reference","title":"Supposition.tear","text":"tear(x::T) where T <: Union{Float16, Float32, Float64} -> Tuple{I, I, I}\n\nReturns the sign, exponent and fractional parts of a floating point number. The returned tuple consists of three unsigned integer types I of the same bitwidth as T.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.test_function-Tuple{Supposition.TestState, Supposition.TestCase}-api","page":"API Reference","title":"Supposition.test_function","text":"test_function(ts::TestState, tc::TestCase)\n\nTest the function given to ts on the test case tc.\n\nReturns a NTuple{Bool, 2} indicating whether tc is interesting and whether it is \"better\" than the previously best recorded example in ts.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.weighted!-Tuple{Supposition.TestCase, Float64}-api","page":"API Reference","title":"Supposition.weighted!","text":"weighted(tc::TestCase, p::Float64)\n\nReturn true with probability p, false otherwise.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.windows-Tuple{Any, Any, Any}-api","page":"API Reference","title":"Supposition.windows","text":"windows(array, a, b)\n\nSplit array into three windows, with split points at a and b. The split points belong to the middle window.\n\n\n\n\n\n","category":"method"},{"location":"api.html#Supposition.CheckConfig-api","page":"API Reference","title":"Supposition.CheckConfig","text":"CheckConfig(;options...)\n\nA struct holding the initial configuration for an invocation of @check.\n\nOptions:\n\nrng: The initial RNG object given to @check. Defaults to a copyable Random.AbstractRNG.\nmax_examples: The maximum number of examples allowed to be drawn with this config. -1 means infinite drawing (careful!). Defaults to 10_000.\nrecord: Whether the result should be recorded in the parent testset, if there is one. Defaults to true.\nverbose: Whether the printing should be verbose, i.e. print even if it's a Pass. Defaults to false.\nbroken: Whether the invocation is expected to fail. Defaults to false.\ndb: An ExampleDB for recording failure cases for later replaying. Defaults to default_directory_db().\nbuffer_size: The default maximum buffer size to use for a test case. Defaults to 100_000.\n\nwarning: Buffer Size\nAt any one point, there may be more than one active buffer being worked on. You can try to increase this value when you encounter a lot of Overrun. Do not set this too large, or you're very likely to run out of memory; the default results in ~800kB worth of choices being possible, which should be plenty for most fuzzing tasks. It's generally unlikely that failures only occur with very large values here, and not with smaller ones.\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Composed-api","page":"API Reference","title":"Supposition.Composed","text":"Composed{S,T} <: Possibility{T}\n\nA Possibility composed from multiple different Possibility through @composed. A tiny bit more fancy/convenient compared to map if multiple Possibility are required to be mapped over at the same time.\n\nShould not be instantiated manually; keep the object returned by @composed around instead.\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.DirectoryDB-api","page":"API Reference","title":"Supposition.DirectoryDB","text":"DirectoryDB <: ExampleDB\n\nAn ExampleDB that records examples as files in a directory.\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Error-api","page":"API Reference","title":"Supposition.Error","text":"Error\n\nA result indicating that an error was encountered while generating or shrinking.\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Fail-api","page":"API Reference","title":"Supposition.Fail","text":"Fail\n\nA result indicating that a counterexample was found.\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.NoRecordDB-api","page":"API Reference","title":"Supposition.NoRecordDB","text":"NoRecordDB <: ExambleDB\n\nAn ExampleDB that doesn't record anything, and won't retrieve anything.\n\nnote: Doing nothing\nIf you're wondering why this exists, I can recommend \"If you're just going to sit there doing nothing, at least do nothing correctly\" by the ever insightful Raymond Chen!\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Pass-api","page":"API Reference","title":"Supposition.Pass","text":"Pass\n\nA result indicating that no counterexample was found.\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Result-api","page":"API Reference","title":"Supposition.Result","text":"Result\n\nAn abstract type representing the ultimate result a TestState ended up at.\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.SuppositionReport-api","page":"API Reference","title":"Supposition.SuppositionReport","text":"SuppositionReport <: AbstractTestSet\n\nAn AbstractTestSet, for recording the final result of @check in the context of @testset\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.TestCase-api","page":"API Reference","title":"Supposition.TestCase","text":"TestCase\n\nA struct representing a single (ongoing) test case.\n\nprefix: A fixed set of choices that must be made first.\nrng: The RNG this testcase ultimately uses to draw from. This is used to seed the task-local RNG object before generating begins.\ngeneration: The \"generation\" this TestCase was made in. Can be used for determining how far along in the generation process we are (higher is further).\nmax_generation: The maximum \"generation\" this TestCase could have been made in. Does not necessarily exist.\nmax_size: The maximum number of choices this TestCase is allowed to make.\nchoices: The binary choices made so far.\ntargeting_score: The score this TestCase attempts to maximize.\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.TestState-api","page":"API Reference","title":"Supposition.TestState","text":"TestState\n\nconfig: The configuration this TestState is running with\nis_interesting: The user given property to investigate\nrng: The currently used RNG\nvalid_test_cases: The count of (so far) valid encountered testcases\ncalls: The number of times is_interesting was called in total\nresult: The choice sequence leading to a non-throwing counterexample\nbest_scoring: The best scoring result that was encountered during targeting\ntarget_err: The error this test has previously encountered and the smallest choice sequence leading to it\nerror_cache: A cache of errors encountered during shrinking that were not of the same type as the first found one, or are from a different location\ntest_is_trivial: Whether is_interesting is trivial, i.e. led to no choices being required\nprevious_example: The previously recorded attempt (if any).\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.UnsetDB-api","page":"API Reference","title":"Supposition.UnsetDB","text":"UnsetDB\n\nAn ExampleDB that is only used by the default CheckConfig, to mark as \"no config has been set\". If this is the database given in a config to @check and no other explicit database has been given, @check will choose the default_directory_db() instead.\n\nCannot be used during testing.\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.CURRENT_TESTCASE-api","page":"API Reference","title":"Supposition.CURRENT_TESTCASE","text":"CURRENT_TESTCASE\n\nA ScopedValue containing the currently active test case. Intended for use in user-facing functions like target! or assume! that need access to the current testcase, but shouldn't require it as an argument to make the API more user friendly.\n\nNot intended for user-side access, thus considered internal and not supported under semver.\n\n\n\n\n\n","category":"constant"},{"location":"api.html#Supposition.DEFAULT_CONFIG-api","page":"API Reference","title":"Supposition.DEFAULT_CONFIG","text":"DEFAULT_CONFIG\n\nA ScopedValue holding the CheckConfig that will be used by default & as a fallback.\n\nCurrently uses these values:\n\nrng: Random.Xoshiro(rand(Random.RandomDevice(), UInt))\nmax_examples: 10_000\nrecord: true\nverbose: false\nbroken: false\ndb: UnsetDB()\nbuffer_size: 100_000\n\n@check will use a new instance of Random.Xoshiro by itself.\n\n\n\n\n\n","category":"constant"},{"location":"api.html#Supposition.MESSAGE_BASED_ERROR-api","page":"API Reference","title":"Supposition.MESSAGE_BASED_ERROR","text":"MESSAGE_BASED_ERROR\n\nA Union of some some in Base that are known to contain only the field :msg.\n\nIf you're using one of these errors and require specialized shrinking on them, define a custom exception type and throw that instead of overriding err_less. The definition of err_less for these types is written for the most generality, not perfect accuracy.\n\nwarning: Unstable\nThis heavily relies on internals of Base, and may break & change in future versions. THIS IS NOT SUPPORTED API.\n\n\n\n\n\n","category":"type"},{"location":"api.html#Supposition.Option-api","page":"API Reference","title":"Supposition.Option","text":"Option{T}\n\nA utility alias for Union{Some{T}, Nothing}.\n\n\n\n\n\n","category":"type"},{"location":"Examples/stateful.html#Stateful-Testing","page":"Stateful Testing","title":"Stateful Testing","text":"","category":"section"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"So far, we've only seem examples of very simple & trivial properties, doing little more than showcasing syntax. However, what if we're operating on some more complicated datastructure and want to check whether the operations we can perform on it uphold the invariants we expect? This too can, in a for now basic form, be done with Supposition.jl.","category":"page"},{"location":"Examples/stateful.html#Juggling-Jugs","page":"Stateful Testing","title":"Juggling Jugs","text":"","category":"section"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Consider this example from the movie Die Hard With A Vengeance:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"\n\n\n
","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"The problem John McClane & Zeus Carver have to solve is the well known 3L & 5L variation on the water pouring puzzle. You have two jugs, one that can hold 3L of liquid and one that can hold 5L. The task is to measure out precisely 4L of liquid, using nothing but those two jugs. Let's model the problem and have Supposition.jl solve it for us:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"struct Jugs\n small::Int\n large::Int\nend\nJugs() = Jugs(0,0)","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"We start out with a struct holding our two jugs; one Int for the small jug and one Int for the large jug. Next, we need the operations we can perform on these jugs. These are","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Filling a jug to the brim\nNo partial filling! That's not accurate enough.\nEmptying a jug\nNo partial emptying! That's not accurate enough.\nPouring one jug into the other\nAny leftover liquid stays in the jug we poured from - don't spill anything!","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Defining them as functions returning a new Jugs, we get:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"# filling\nfill_small(j::Jugs) = Jugs(3, j.large)\nfill_large(j::Jugs) = Jugs(j.small, 5)\n\n# emptying\nempty_small(j::Jugs) = Jugs(0, j.large)\nempty_large(j::Jugs) = Jugs(j.small, 0)\n\n# pouring\nfunction pour_small_into_large(j::Jugs)\n nlarge = min(5, j.large + j.small)\n nsmall = j.small - (nlarge - j.large)\n Jugs(nsmall, nlarge)\nend\n\nfunction pour_large_into_small(j::Jugs)\n nsmall = min(3, j.small + j.large)\n nlarge = j.large - (nsmall - j.small)\n Jugs(nsmall, nlarge)\nend\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"From the top, we have filling either jug (note that we can only fill the small jug up to 3L, and the large up to 5L), emptying either jug, and finally pouring one into the other, taking care not to spill anything (i.e., any leftovers stay in the jug we poured out of).","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"We can very easily now generate a sequence of operations:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"using Supposition\n\nraw_ops = (fill_small, fill_large, empty_small, empty_large, pour_small_into_large, pour_large_into_small)\ngen_ops = Data.Vectors(Data.SampledFrom(raw_ops))\ngen_ops = Data.Vectors(Data.SampledFrom(raw_ops); min_size=5, max_size=10) # hide\nexample(gen_ops)","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Generating a sequence of operations is simply generating a vector from all possible ones! This is the input to our property. We declare that for all sequences of operations we can do with a Jug, all invariants we expect must hold true.","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Speaking of invariants, we need three of them that must be preserved at all times:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"The small jug must ALWAYS have a fill level between 0 and 3 (inclusive).\nThe large jug must ALWAYS have a fill level between 0 and 5 (inclusive).\nThe large just must NEVER have a fill level of exactly 4.","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"The last invariant may look a bit odd, but remember that Supposition.jl is trying to find a falsifying example. The first two invariants are sanity checks to make sure that our pouring functions are well behaved; the last invariant is the solution we want to find, by combining the operations above in an arbitrary order. Let's translate these into functions as well:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"small_jug_invariant(j::Jugs) = 0 <= j.small <= 3\nlarge_jug_invariant(j::Jugs) = 0 <= j.large <= 5\nlevel_invariant(j::Jugs) = j.large != 4\ninvariants = (small_jug_invariant, large_jug_invariant, level_invariant)\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"And now, to finally combine all of these:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"gen_ops = Data.Vectors(Data.SampledFrom(raw_ops)) # hide\n# do a little dance so that this expected failure doesn't kill doc building # hide\ntry # hide\n@check function solve_die_hard(ops = gen_ops)\n jugs = Jugs()\n\n for op in ops\n # apply the rule\n jugs = op(jugs)\n\n # check invariants\n for f in invariants\n f(jugs) || return false\n end\n end\n\n return true\nend\ncatch # hide\nend # hide\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"This pattern is very extensible, and a good candidate for the next UX overhaul (getting a reported failure for the target we actually want to find is quite bad UX). Nevertheless, it already works right now!","category":"page"},{"location":"Examples/stateful.html#Balancing-a-heap","page":"Stateful Testing","title":"Balancing a heap","text":"","category":"section"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"The previous example showed how we can check these kinds of operations based invariants on an immutable struct. There is no reason why we can't do the same with a mutable struct (or at least, a struct containing a mutable object) though, so let's look at another example: ensuring a heap observes its heap property. As a quick reminder, the heap property for a binary heap is that each child of a node is <= than that node, resulting in what's called a \"Max-Heap\" (due to the maximum being at the root). Similarly, if the property for children is >=, we get a \"Min-Heap\". Here, we're going to implement a Min-Heap.","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"First, we need to define our datastructure:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"struct Heap{T}\n data::Vector{T}\nend\nHeap{T}() where T = Heap{T}(T[])","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"as well as the usual operations (isempty, push!, pop!) on that heap:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"isempty: Whether the heap has elements\npush!: Put an element onto the heap\npop!: Retrieve the smallest element of the heap (i.e., remove the root)","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Written in code, this might look like this:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Base.isempty(heap::Heap) = isempty(heap.data)\n\nfunction Base.push!(heap::Heap{T}, value::T) where T\n data = heap.data\n push!(data, value)\n index = lastindex(data)\n while index > firstindex(data)\n parent = index >> 1\n if data[parent] > data[index]\n data[parent], data[index] = data[index], data[parent]\n index = parent\n else\n break\n end\n end\n heap\nend\n\nBase.pop!(heap::Heap) = popfirst!(heap.data)","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"In this implementation, we're simply using an array as the backing store for our heap. The first element is the root, followed by the left subtree, followed by the right subtree. As implemented, pop! will return the correct element if the heap is currently balanced, but because pop! doesn't rebalance the heap after removing the root, pop! may leave it in an invalid state. A subsequent pop! may then remove an element that is not the smallest currently stored.","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"We can very easily test this manually:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"using Supposition\n\nintvec = Data.Vectors(Data.Integers{UInt8}())\n\ntry # hide\n@check function test_pop_in_sorted_order(ls=intvec)\n h = Heap{eltype(ls)}()\n\n # push all items\n for l in ls\n push!(h, l)\n end\n\n # pop! all items\n r = eltype(ls)[]\n while !isempty(h)\n push!(r, pop!(h))\n end\n\n # the pop!ed items should be sorted\n r == sort(ls)\nend\ncatch # hide\nend # hide\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"And as expected, the minimal counterexample is [0x0, 0x1, 0x0]. We first pop! 0x0, followed by 0x1 while it should be 0x0 again, and only then 0x1, resulting in [0x0, 0x0, 0x1] instead of [0x0, 0x1, 0x0].","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Replacing this with a (presumably) correct implementation looks like this:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"function fixed_pop!(h::Heap)\n isempty(h) && throw(ArgumentError(\"Heap is empty!\"))\n data = h.data\n isone(length(data)) && return popfirst!(data)\n result = first(data)\n data[1] = pop!(data)\n index = 0\n while (index * 2 + 1) < length(data)\n children = [ index*2+1, index*2+2 ]\n children = [ i for i in children if i < length(data) ]\n @assert !isempty(children)\n sort!(children; by=x -> data[x+1])\n broke = false\n for c in children\n if data[index+1] > data[c+1]\n data[index+1], data[c+1] = data[c+1], data[index+1]\n index = c\n broke = true\n break\n end\n end\n !broke && break\n end\n return result\nend","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Me telling you that this is correct though should only be taken as well-intentioned, but not necessarily as true. There might be more bugs that have sneaked in after all, that aren't caught by our naive \"pop in order and check that it's sorted\" test. There could be a nasty bug waiting for us that only happens when various push! and pop! are interwoven in just the right way. Using stateful testing techniques and the insight that we can generate sequences of operations on our Heap with Supposition.jl too! We're first going to try with the existing, known broken pop!:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"gen_push = map(Data.Integers{UInt}()) do i\n (push!, i)\nend\ngen_pop = Data.Just((pop!, nothing))\ngen_ops = Data.Vectors(Data.OneOf(gen_push, gen_pop); max_size=10_000)\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"We either push! an element, or we pop! from the heap. Using (pop!, nothing) here will make it a bit easier to actually define our test. Note how the second element acts as the eventual argument to pop!.","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"There's also an additional complication - because we don't have the guarantee anymore that the Heap contains elements, we have to guard the use of pop! behind a precondition check. In case the heap is empty, we can just consume the operation and treat it as a no-op, continuing with the next operation:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"# let's dance again, Documenter.jl! # hide\ntry # hide\n@check function test_heap(ops = gen_ops)\n heap = Heap{UInt}()\n\n for (op, val) in ops\n if op === push!\n # we can always push\n heap = op(heap, val)\n else\n # check our precondition!\n isempty(heap) && continue\n\n # the popped minimum must always == the minimum\n # of the backing array, so retrieve the minimum\n # through alternative internals\n correct = minimum(heap.data)\n val = op(heap)\n\n # there's only one invariant this time around\n # and it only needs checking in this branch:\n val != correct && return false\n end\n end\n\n # by default, we pass the test!\n # this happens if our `ops` is empty or all operations\n # worked successfully\n return true\nend\ncatch # hide\nend # hide\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Once again, we find our familiar example UInt[0x0, 0x1, 0x0], though this time in the form of operations done on the heap:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"ops = Union{Tuple{typeof(pop!), Nothing}, Tuple{typeof(push!), UInt64}}[\n (push!, 0x0000000000000001),\n (push!, 0x0000000000000000),\n (push!, 0x0000000000000000),\n (pop!, nothing),\n (pop!, nothing)\n]","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"We push three elements (0x1, 0x0 and 0x0) and when popping two, the second doesn't match the expected minimum anymore!","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Now let's try the same property with our (hopefully correct) fixed_pop!:","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"gen_fixed_pop = Data.Just((fixed_pop!, nothing))\ngen_fixed_ops = Data.Vectors(Data.OneOf(gen_push, gen_fixed_pop); max_size=10_000)\n# Documenter shenanigans require me to repeat this. # hide\nfunction test_heap(ops) # hide\n heap = Heap{UInt}() # hide\n # hide\n for (op, val) in ops # hide\n if op === push! # hide\n # we can always push # hide\n heap = op(heap, val) # hide\n else # hide\n # check our precondition! # hide\n isempty(heap) && continue # hide\n # hide\n # the popped minimum must always == the minimum # hide\n # of the backing array, so retrieve the minimum # hide\n # through alternative internals # hide\n correct = minimum(heap.data) # hide\n val = op(heap) # hide\n # hide\n # there's only one invariant this time around # hide\n # and it only needs checking in this branch: # hide\n val != correct && return false # hide\n end # hide\n end # hide\n # hide\n # by default, we pass the test! # hide\n return true # hide\nend # hide\n\n@check test_heap(gen_fixed_ops)\nnothing # hide","category":"page"},{"location":"Examples/stateful.html","page":"Stateful Testing","title":"Stateful Testing","text":"Now this is much more thorough testing!","category":"page"},{"location":"resources.html#PBT-Resources","page":"PBT Resources","title":"PBT Resources","text":"","category":"section"},{"location":"resources.html","page":"PBT Resources","title":"PBT Resources","text":"This page contains a collection of PBT tutorials and other useful resources for learning PBT techniques. Most, if not all, should be directly translatable to Supposition.jl in one form or another. If you find a new tutorial or resource that helped you test your code with Supposition.jl in some manner, please don't hesitate to open a PR adding the resource here!","category":"page"},{"location":"resources.html","page":"PBT Resources","title":"PBT Resources","text":"The purpose of Hypothesis by David R. MacIver\n[...], the larger purpose of Hypothesis is to drag the world kicking and screaming into a new and terrifying age of high quality software.\nHypothesis testing with Oracle functions by Hillel Wayne\nA blogpost about using existing (but slower/partially incorrect) implementations to make sure a refactored or new implementation still conforms to all expected contracts of the old implementation.\nSolving the Water Jug Problem from Die Hard 3 with TLA+ and Hypothesis by Nicholas Chammas\nA blogpost about helping out John McClane (Bruce Willis) and Zeus Carver (Samuel L. Jackson) ~defuse a bomb~ solve fun children's games.\nThis blogpost has been translated to Supposition.jl! Check it out in the examples.\nRule Based Stateful Testing by David R. MacIver\nA blogpost from the main developer behind Hypothesis, showing how to test stateful systems with Hypothesis.\nThis blogpost has been translated to Supposition.jl! Check it out in the examples.\nNote: Not all features of Hypothesis have been ported to Supposition.jl, in particular the UX for stateful testing is very bare bones. The linked example contains a very manual implementation of the features utilized by Hypothesis for much the same thing, but should be easily adaptable for all kinds of stateful tests.\nProprty Testing Stateful Code in Rust by Raphael Gashignard\nA blogpost about fuzzing internal datastructures of nushell using PBT and the Rust library proptest.\nAutomate Your Way to Better Code: Advanced Property Testing (with Oskar Wickström) by Kris Jenkins from Developer Voices\nMy Job as a programmer is to be lazy in the smart way - I see that many unit tests, and I just want to automate the problem away. Well that's the promise of property testing - write a bit of code that describes the shape of your software, and it will go away and create 10_000 unit tests to see if you're right, if it actually does work that way. [..] we're also going to address my biggest disappointment so far with property testing: which is that it only seems to work in theory. It's great for textbook examples, I'm sold on the principle, but I've struggled to make it work on my more gnarly real world code.\nThis is an absolutely delightful listen! A nice definition of what property based testing is, as well as a lot of discussion on how to start out with property based testing and continue with the approach onto more difficult pastures. Don't let yourself be intimidated by the length - take your time with this one, it's well worth it!\nThe Magic of Property Testing by Kris Jenkins from Developer Voices\nThis is a followup to \"Automate Your Way to Better Code\", showcasing an example of property based testing in PureScript. The fuzzing framework used here is a port of QuickCheck, but the general flow should be translatable to Supposition.jl. One feature being showcased (generation of objects through reflection) is not yet available in Supposition.jl; see this discussion for the state of things. Nevertheless, even without that, the generation capabilities of random data in Supposition.jl are just as powerful.","category":"page"},{"location":"Examples/recursive.html#Recursive-Generation","page":"Recursive Generation","title":"Recursive Generation","text":"","category":"section"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"In some situations, it is required to generate objects that can nest recursively. For example, JSON is an often used data exchange format that consists of various layers of dictionaries (with string keys) and one dimensional arrays, as well as strings, numbers, booleans an Nothing.","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"In Supposition.jl, we can generate these kinds of recursively nested objects using the Data.Recursive Possibility. For this, we need a generator of a basecase, as well as a function that wraps one generated example in a new layer by returning a new Possibility.","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"We can construct the Possibility that generates the basecase like so:","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"using Supposition\n\nstrings = Data.Text(Data.AsciiCharacters())\nstrings = Data.Text(Data.AsciiCharacters(); max_len=10) # hide\nbools = Data.Booleans()\nnone = Data.Just(nothing)\nnumbers = Data.Floats{Float64}()\nbasecase = strings | numbers | bools | none","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"which gives us a Data.OneOf, a Possibility that can generate any one of the objects generated by the given Possibility.","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"For wrapping into new layers, we need a function that wraps our basecase Possibility and gives us a new Possibility generating the wrapped objects. For the JSON example, this means we can wrap an object either in a Vector, or a Dict, where the latter has String keys.","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"note: Wrapping order\nRecursive expects a function that takes a Possibility for generating the children of the wrapper object, which you should pass into a generator. The generator for the wrapper can be any arbitrary Possibility.","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"Defining that function like so:","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"function jsonwrap(child)\n vecs = Data.Vectors(child)\n dicts = Data.Dicts(strings, child)\n vecs = Data.Vectors(child; max_size=5) # hide\n dicts = Data.Dicts(strings, child; max_size=5) # hide\n vecs | dicts\nend","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"allows us to construct the Possibility for generating nested JSON-like objects:","category":"page"},{"location":"Examples/recursive.html","page":"Recursive Generation","title":"Recursive Generation","text":"json = Data.Recursive(basecase, jsonwrap; max_layers=3)\nexample(json)\n# a little bit of trickery, to show a nice example # hide\nprintln( # hide\n\"Dict{String, Union{Nothing, Bool, Float64, Dict{String, Union{Nothing, Bool, Float64, String}}, String, Vector{Union{Nothing, Bool, Float64, String}}}} with 5 entries:\" * # hide\n\"\\n \\\"!\\\" => -1.58772e111\" * # hide\n\"\\n \\\"\\\\e^Y\\\\x1cq\\\\bEj8\\\" => -4.31286e-135\" * # hide\n\"\\n \\\"^\\\" => Union{Nothing, Bool, Float64, String}[false]\" * # hide\n\"\\n \\\"\\\\x0f \\\\t;lgC\\\\e\\\\x15\\\" => nothing\" * # hide\n\"\\n \\\"Y266uYkn6\\\" => -5.68895e-145\" # hide\n) # hide\nnothing # hide","category":"page"},{"location":"index.html#Supposition.jl-Documentation","page":"Main Page","title":"Supposition.jl Documentation","text":"","category":"section"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"This is the documentation for Supposition.jl, a property based testing framework inspired by Hypothesis.","category":"page"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"It features choice-sequence based generation & shrinking of examples, which can smartly shrink initial failures to smaller example while preserving the invariants the original input was generated under. It's also easy to combine generators into new generators.","category":"page"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"Check out the Examples in the sidebar to get an introduction to property based testing and to learn how to write your own tests!","category":"page"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"Here's also a sitemap for the rest of the documentation:","category":"page"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"Pages = [ \"index.md\", \"intro.md\", \"faq.md\", \"interfaces.md\", \"api.md\" ]\nDepth = 3","category":"page"},{"location":"index.html#Goals","page":"Main Page","title":"Goals","text":"","category":"section"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"Good performance\nA test framework should not be the bottleneck of the testsuite.\nComposability\nIt should not be required to modify an existing codebase to accomodate Supposition.jl\nHowever, for exploratory fuzzing it may be advantageous to insert small markers into a codebase\nReusability\nIt should be easily possible to reuse large parts of existing definitions (functions/structs) to build custom generators\nRepeatability\nIt should be possible to replay previous (failing) examples and reproduce the sequence of steps taken exactly. The only cases where this isn't going to work is if your code relies on external state, such as querying a hardware RNG for random data or similar objects that are not under the control of the testing framework itself (such as the capacity of your harddrive, for example).\nDiscoverability (of API boundaries)\nSupposition.jl should be easy to use to find the actual API boundaries of a given function, if that is not yet known or not sufficiently specified in the docstring of a function. It should be enough to know the argument types to start fuzzing a function (at least in the simplest sense of \"does it error\").\nEase of Use\nIt should be relatively straightforward to write custom generators.","category":"page"},{"location":"index.html#Limitations","page":"Main Page","title":"Limitations","text":"","category":"section"},{"location":"index.html","page":"Main Page","title":"Main Page","text":"Due to its nature as a fuzzing framework and the (usually) huge associated statespace, Supposition.jl cannot give a formal proof of correctness. It's only an indicator (but a pretty good one).","category":"page"}]
}