Maximum call stack size exceeded with Test.fuzz2

[harrysarson]

SSCCE:

module Example exposing (suite)

import Expect 
import Fuzz
import Test exposing (..)

suite : Test
suite =
    fuzz2 Fuzz.float (Fuzz.floatRange -1.0e100 1.0e100) "should not run out of stack" <|
        \a b ->
            Expect.fail "always fail"

Expected

Test to fail

Actual

elm-test 0.19.0-beta8
---------------------

Running 1 test. To reproduce these results, run: elm-test --fuzz 100 --seed 48737553166794

↓ Example
✗ should not run out of stack

    This test failed because it threw an exception: "RangeError: Maximum call stack size exceeded"

Notes

• This only occurs when a test fails.
• Changing floatRange to float fixes this.
• Possibly related “Maximum call stack size exceeded” on comparison of large data structures #22

[mgold]

It's more likely #18, which was fixed in 1.1.0. Can you see if you're running that version?

[harrysarson]

elm.json says that this example is run using elm-test 1.1.0.

[mgold]

Okay. I would say it has something to do with your tiny float range but the RNG handles it fine:

> Random.float -1.0e100 1.0e100
Generator <function> : Random.Generator Float
> gen = Random.float -1.0e100 1.0e100
Generator <function> : Random.Generator Float
> seed = Random.initialSeed 314159
Seed 135935125 1013904223 : Random.Seed
> Random.step gen seed
(-2.623217990942024e+99,Seed 4011486095 1013904223)
    : ( Float, Random.Seed )

It might be trying to shrink a really small number and it fails?

[harrysarson]

I added Debug.log calls:

module Example exposing (suite)

import Expect exposing (Expectation)
import Fuzz exposing (Fuzzer, int, list, string)
import Test exposing (..)

suite : Test
suite =
    fuzz2 Fuzz.float (Fuzz.floatRange -1.0e100 1.0e100) "should not run out of stack" <|
        \a b ->
           Expect.fail <| Debug.log "Fail with: a, b = " (String.fromFloat a ++ ", " ++ String.fromFloat b)

The output is now:

elm-test 0.19.0-beta8
---------------------

Running 1 test. To reproduce these results, run: elm-test --fuzz 100 --seed 2313684668028

Fail with: a, b = : "2535342688.333782, 4.036496730212384e+99"
Fail with: a, b = : "0, 4.036496730212384e+99"
Fail with: a, b = : "0.000001, 4.036496730212384e+99"
Fail with: a, b = : "0.000001, 0"
Fail with: a, b = : "0.000001, 0.000001"
Fail with: a, b = : "20.761459776876663, -2.4929455325177343e+99"
Fail with: a, b = : "0, -2.4929455325177343e+99"
Fail with: a, b = : "0.000001, -2.4929455325177343e+99"
Fail with: a, b = : "0.000001, 2.4929455325177343e+99"
Fail with: a, b = : "0.000001, 0"
Fail with: a, b = : "0.000001, 0.000001"
Fail with: a, b = : "-39.76976598642466, -1e+100"
Fail with: a, b = : "39.76976598642466, -1e+100"
Fail with: a, b = : "0, -1e+100"
Fail with: a, b = : "0.000001, -1e+100"
↓ Example
✗ should not run out of stack

    This test failed because it threw an exception: "RangeError: Maximum call stack size exceeded"



TEST RUN FAILED

Duration: 270 ms
Passed:   0
Failed:   1

I have run this a couple of times and the last log call is always Fail with: a, b = : "0.000001, ??" (?? varies).

[harrysarson]

Additionally: elm-test --fuzz 3 works fine (it fails properly) whereas elm-test --fuzz 4 throws the call stack error.

[catz]

Tried to refresh my knowledge on decoders and got the same stack error for this exercise https://github.com/zwilias/elm-demystify-decoders/blob/master/src/Exercise07.elm

I accidentally typed succeed "sure" instead of succeed "sure." and received. This is with default fuzz settings or with --fuzz 4

Running 1 test. To reproduce these results, run: elm-test --fuzz 4 --seed 405727356696022 /Users/catz/Projects/elm2019/elm-demystify-decoders/tests/Exercise07/Tests.elm

↓ Exercise07.Tests
↓ Exercise 07
✗ Decode random JS value

    This test failed because it threw an exception: "RangeError: Maximum call stack size exceeded"

With --fuzz 3 everything works correctly

Running 1 test. To reproduce these results, run: elm-test --fuzz 3 --seed 248159612312337 /Users/catz/Projects/elm2019/elm-demystify-decoders/tests/Exercise07/Tests.elm

↓ Exercise07.Tests
↓ Exercise 07
✗ Decode random JS value

Given <internals>

    Ok "sure"
    ╷
    │ Expect.equal
    ╵
    Ok "sure."

[Janiczek]

Any ideas? I'm getting this with lists-of-random-length:
Random.int 0 4 |> Random.andThen (\length -> Random.map List_ (Random.list length lazyExprGenerator))
(as opposed to Random.int 0 3 |> ... which doesn't do that, even with --fuzz 99999)

[mgold]

Can you (or someone) make a clone-able example to debug?

[Janiczek]

https://ellie-app.com/4JSdnNjJxdBa1
This should hopefully be minimal enough.
Vary the Random.int 0 5 number to trigger different results. ie. with Random.int 0 2 it doesn't crash.

[Janiczek]

By Debug.loging the expr inside the test, I've found that when we crash, nothing gets logged
➡️
I assume the crash happens in the Generator

Also, we can replace the shrinker with Shrink.noShrink with no change in behaviour (the crash still happens)

[Janiczek]

Trimming it down a bit further: https://ellie-app.com/4JSRDc2sstqa1
(Generating Plus2 works, generating Plus3 crashes.)

So, maybe it's Random.lazy?

[Janiczek]

Stepping through, it's most likely a combination of Random.andThen + Random.lazy.
Eg. this is the third stepping through the lazyExprGenerator's callback.

I don't know what would solve this though. Some kind of trampolining?

[mgold]

Can the bug be isolated from elm-test to only elm/random?

Also, we can replace the shrinker with Shrink.noShrink with no change in behaviour

That makes sense since the test always passes (provided the value is generated successfully).

[Janiczek]

@mgold Yes, it can: https://ellie-app.com/4Kv76dc7Hkpa1

[Janiczek]

That means I'll create a ticket in elm/random then 🤷‍♂️

[Janiczek]

elm/random#9

[armatures]

Great investigation on this! I'm encountering it in a couple projects. Any recommendations for avoiding it (It sounds like mapN functions are the source of the problem)?

Is this the reference to a generator that would need a hard depth limit? That's a total shot in the dark: I was just hunting for a generator like the one referenced in the elm/random issue.

[mgold]

No, that's only used to shrink values on failures, and the tree is lazy. I think it's a problem with fuzzers inherently. You need to be careful with Random.andThen.

[Janiczek]

This is no longer an issue on current master (soon to be 2.0.0), since fuzzers now don't map 1-to-1 to random generators. Let's close this after 2.0.0 gets released