Simulation results sensitive to whether save/make dependencies

[dachengx]

For reproducibility, all codes here are running under 0d3fb95.

When running TestFullChain by hand, I got two different results of raw_records under different codes.

fuse/tests/test_FullChain.py

Line 5 in 0d3fb95

class TestFullChain(unittest.TestCase):

Two groups of codes are:

t = TestFullChain()
t.setUpClass()
t.test_PMTResponseAndDAQ()  # makes raw_records

and

t = TestFullChain()
t.setUpClass()
t.test_ElectronDrift()  # makes drifted_electrons
t.test_ElectronExtraction()  # makes extracted_electrons
t.test_ElectronTiming()  # makes electron_time
t.test_PMTAfterPulses()  # makes pmt_afterpulses
t.test_PMTResponseAndDAQ()  # makes raw_records

The difference in the results can be seen from the metadata of raw_records:

"chunks": [
        {
            "chunk_i": 0,
            "end": 2651890863,
            "filename": "raw_records-rfvprleg6h-000000",
            "filesize": 15885209,
            "first_endtime": 52500390,
            "first_time": 52499380,
            "last_endtime": 2552200830,
            "last_time": 2552200460,
            "n": 95384,
            "nbytes": 23273696,
            "run_id": "TestRun_00000",
            "start": 51499782,
            "subruns": null
        }

and

"chunks": [
        {
            "chunk_i": 0,
            "end": 2651890863,
            "filename": "raw_records-rfvprleg6h-000000",
            "filesize": 15736119,
            "first_endtime": 52500370,
            "first_time": 52499360,
            "last_endtime": 2552200920,
            "last_time": 2552200850,
            "n": 94568,
            "nbytes": 23074592,
            "run_id": "TestRun_00000",
            "start": 51499782,
            "subruns": null
        }

The 2nd method has a smaller "n" .

Note: by setting deterministic_seed to True, the result should be deterministic.

fuse/fuse/plugins/pmt_and_daq/pmt_response_and_daq.py

Line 42 in 0d3fb95

deterministic_seed = straxen.URLConfig(

This might be a problem of DownChunkingPlugin, according to @HenningSE .

[dachengx]

I find the reason for this:

Actually, the results differ earlier from quanta.

In NestYields(which provides quanta) and S1PhotonPropagationBase(which provides propagated_s1_photons), fuse set seed to nest's random number generator: nest_rng.set_seed(self.short_seed).

https://github.com/NESTCollaboration/nest/blob/1e825f60ebb5b143216d84ef3647e3928ab9af17/src/RandomGen.cpp#L22

fuse/fuse/plugins/micro_physics/yields.py

Line 67 in e6e27fb

nest_rng.set_seed(self.short_seed)

fuse/fuse/plugins/detector_physics/s1_photon_propagation.py

Line 154 in e6e27fb

nest_rng.set_seed(self.short_seed)

It seems that in current workflow, nest_rng is a global generator, which is different from np.random.default_rng(https://numpy.org/doc/stable/reference/random/generator.html). You can also see this behavior in NEST https://github.com/NESTCollaboration/nest/blob/1e825f60ebb5b143216d84ef3647e3928ab9af17/src/RandomGen.cpp#L11.

In the above #108 (comment) two cases, the number of execution of nest_rng is different. These initializations happen at strax.context.get_components https://github.com/AxFoundation/strax/blob/2e4c48a1f9c05cc9daec2201ef7791159a3c6f0e/strax/context.py#L1153.

For the 1st group of codes, it runs nest_rng.set_seed(self.short_seed) twice for NestYields and S1PhotonPropagationBase.

t = TestFullChain()
t.setUpClass()
t.test_PMTResponseAndDAQ()  # nest_rng.set_seed for NestYields then S1PhotonPropagationBase

For the 2nd group of codes, it runs nest_rng.set_seed(self.short_seed) three times for NestYields, S1PhotonPropagationBase and S1PhotonPropagationBase:

t = TestFullChain()
t.setUpClass()
t.test_ElectronDrift()  # nest_rng.set_seed for NestYields
t.test_ElectronExtraction()
t.test_ElectronTiming()
t.test_PMTAfterPulses()  # nest_rng.set_seed for S1PhotonPropagationBase
t.test_PMTResponseAndDAQ()  # nest_rng.set_seed for S1PhotonPropagationBase

So the solution would be:

1. find a way to initialize a local nest_rng
2. or only put the nest-related random number generator into one plugin, like combine the NestYields and S1PhotonPropagationBase
3. set seed of nest_rng inside strax.Plugin.compute, but maybe we can not predict the order of execution of compute in strax's multi-thread processing

And in general(not directly related to this issue), if we use some time_range selection of strax, the state of random generator(even for np.random.default_rng) will be different for different time_range. So for a deterministic result, we should either find a way to strictly control the random state, or discourage the usage of time_range selection.

[HenningSE]

I am trying with 6fd5399 to set the nestpy random seed inside the compute function. As before, a seed is build in the setup function from the lineage and run_number. Then the seed is used in the compute function to set the random seed. Afterwards the seed is incremented so we get different random numbers next time the compute function is used. I'm not 100% sure this method works and we should discuss this next week.

[dachengx]

MISSION ACCOMPLISHED by
NESTCollaboration/nest#180
NESTCollaboration/nestpy#106
#112 (0751ae3)