s2a gives strange output

[savchenkoyana]

Dear maintainers,

Thank you for this repository!

I use NEMO tools for my analysis (e.g., s2a, manipulate, etc.). What I recently encountered is that s2a gives strange output for my nemo file. When I use something like this:

s2a my_file.nemo out_a times=0.0

The resulting file out_a looks like it was printed several times all over again: dashes with text, then correct output lines (full), the dashes, the output all over again.

I tried to debug it putting a std::cout after the line https://github.com/teuben/nemo/blob/master/usr/dehnen/falcON/src/public/exe/s2a.cc#L154
So I got as much std::cout prints as there are duplicates in s2a output.
I also face duplicated output when I use manipulate command.

Is this behavior expected? I think maybe the bug comes from the fact that my snapshot is very large. Is there a way to fix this?

Sincerely,
Yana

[teuben]

How large is the snapshot. Can you give me the history of the snapshot with the "hisf" command.

[teuben]

you could also use "snappprint my_file.nemo times=0" which has an output similar to s2a, to see if that fails too

[teuben]

mkplummer p10 10 
hackcode1 p10 p10.out fcells=2 
snapprint p10.out times=0
s2a p10.out times=0

that last command failed because s2a doesn't know how to skip diagnostic frames, see the command tsf how it looks inside. Not sure if your file has diagnastics.

[savchenkoyana]

• Snapshot size is 84 Gb
• The output of hisf command:

  hisf snap_mu0.0_s1.0_sigma1.0_r10.0_N20000_06_12/out_postprocessed.nemo

  is as follows:

  snapscale - snap_mu0.0_s1.0_sigma1.0_r10.0_N20000_06_12/out_postprocessed.nemo rscale=1000.0 VERSION=3.2b
  snapmask snap_mu0.0_s1.0_sigma1.0_r10.0_N20000_06_12/out.nemo - select=0:19999 VERSION=1.9a
  gyrfalcON snap_mu0.0_s1.0_sigma1.0_r10.0_N20000/IC_preprocessed.nemo snap_mu0.0_s1.0_sigma1.0_r10.0_N20000/out.nemo logstep=3000 eps=0.0003684031498640387 kmax=15 tstop=14 step=1.8693875619711035e-04 Grav=4.30091727067736e-06 VERSION=3.6

  Shortly: I create initial conditions IC_preprocessed.nemo with Agama (SMBH at a center + cluster at its orbit), evolve it to obtain out.nemo, scale to change units (kpc->pc) and mask to get rid of SMBH in snapshot data to finally get out_postprocessed.nemo
• both out.nemo and out_postprocessed.nemo have the same bug with s2a command
• snapprint gives the same bug (duplicates printing 2-3 times instead of 1)

[savchenkoyana]

I prepared a smaller snapshot overnight (I used the same procedure as previously but set higher tstep in gyrfalcON command). The new snapshot size is about 2Gb and it does not give duplicates in s2a...

[teuben]

Could you repeat this for another magic size number, namely 4.1Gb. At 4.0 GB we still are below unsigned 32bits overrun, but e.g. 4.5gb we are over that limit . I'm surprised snapprint also fails, bit that program is easier to debug

…

On Sun, Dec 22, 2024, 03:01 Yana ***@***.***> wrote: I prepared a smaller snapshot overnight (I used the same procedure as previously but set higher tstep in gyrfalcON command). The new snapshot size is about 2Gb and it does not give duplicates in s2a... — Reply to this email directly, view it on GitHub <#162 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZ4MGL5BT57QH5SZ2K2KWT2GZWVPAVCNFSM6AAAAABUAZ2YVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJYGM3DQMJVG4> . You are receiving this because you commented.Message ID: ***@***.***>

[teuben]

Another idea is to use snaptrim to extract one snapshot, and see if s2a then works

…

On Sun, Dec 22, 2024, 08:13 Peter Teuben ***@***.***> wrote: Could you repeat this for another magic size number, namely 4.1Gb. At 4.0 GB we still are below unsigned 32bits overrun, but e.g. 4.5gb we are over that limit . I'm surprised snapprint also fails, bit that program is easier to debug On Sun, Dec 22, 2024, 03:01 Yana ***@***.***> wrote: > I prepared a smaller snapshot overnight (I used the same procedure as > previously but set higher tstep in gyrfalcON command). The new snapshot > size is about 2Gb and it does not give duplicates in s2a... > > — > Reply to this email directly, view it on GitHub > <#162 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAZ4MGL5BT57QH5SZ2K2KWT2GZWVPAVCNFSM6AAAAABUAZ2YVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJYGM3DQMJVG4> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

[teuben]

I did a simple test which now is at 6Gb and s2a and snapprint still work

…

On Sun, Dec 22, 2024, 08:14 Peter Teuben ***@***.***> wrote: Another idea is to use snaptrim to extract one snapshot, and see if s2a then works On Sun, Dec 22, 2024, 08:13 Peter Teuben ***@***.***> wrote: > Could you repeat this for another magic size number, namely 4.1Gb. At 4.0 > GB we still are below unsigned 32bits overrun, but e.g. 4.5gb we are over > that limit . > > I'm surprised snapprint also fails, bit that program is easier to debug > > On Sun, Dec 22, 2024, 03:01 Yana ***@***.***> wrote: > >> I prepared a smaller snapshot overnight (I used the same procedure as >> previously but set higher tstep in gyrfalcON command). The new snapshot >> size is about 2Gb and it does not give duplicates in s2a... >> >> — >> Reply to this email directly, view it on GitHub >> <#162 (comment)>, or >> unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AAZ4MGL5BT57QH5SZ2K2KWT2GZWVPAVCNFSM6AAAAABUAZ2YVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJYGM3DQMJVG4> >> . >> You are receiving this because you commented.Message ID: >> ***@***.***> >> >

[teuben]

Does the tsf command work on the big data work. What is the output times, in my test I had a small step=0.000001, and noted that the time fuzz is too big for this , outputting multiple snapshots. But it won't crash the program

…

On Sun, Dec 22, 2024, 08:25 Peter Teuben ***@***.***> wrote: I did a simple test which now is at 6Gb and s2a and snapprint still work On Sun, Dec 22, 2024, 08:14 Peter Teuben ***@***.***> wrote: > Another idea is to use snaptrim to extract one snapshot, and see if s2a > then works > > On Sun, Dec 22, 2024, 08:13 Peter Teuben ***@***.***> wrote: > >> Could you repeat this for another magic size number, namely 4.1Gb. At >> 4.0 GB we still are below unsigned 32bits overrun, but e.g. 4.5gb we are >> over that limit . >> >> I'm surprised snapprint also fails, bit that program is easier to debug >> >> On Sun, Dec 22, 2024, 03:01 Yana ***@***.***> wrote: >> >>> I prepared a smaller snapshot overnight (I used the same procedure as >>> previously but set higher tstep in gyrfalcON command). The new >>> snapshot size is about 2Gb and it does not give duplicates in s2a... >>> >>> — >>> Reply to this email directly, view it on GitHub >>> <#162 (comment)>, >>> or unsubscribe >>> <https://github.com/notifications/unsubscribe-auth/AAZ4MGL5BT57QH5SZ2K2KWT2GZWVPAVCNFSM6AAAAABUAZ2YVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJYGM3DQMJVG4> >>> . >>> You are receiving this because you commented.Message ID: >>> ***@***.***> >>> >>

[savchenkoyana]

I did the simulation with size 7.4Gb, it is okay. So for simulations with 7Gb and 2Gb the output of s2a myfile.nemo out_a times=0 looks like this:

# ... (lines with comments)
# 'mass ' 'pos  ' 'vel  '
#
... (lines with data for particles)

But for the first simulation it is like this:

# ... (lines with comments)
# 'mass ' 'pos  ' 'vel  '
#
... (lines with data for particles)
# ... (lines with comments)
# 'mass ' 'pos  ' 'vel  '
#
... (lines with data for particles)
# ... (lines with comments)
# 'mass ' 'pos  ' 'vel  '
#
... (lines with data for particles)

So basically in all cases i have 20000 particles and expect to get ~20000 rows in s2a output, but with the large simulation file I get the same output multiple times (for example, s2a output could have ~60000 lines just like in example above).

I still can use it though by only loading the data between two group of comment lines.

[teuben]

I had the experience in NEMO that I also got multiple of 20,000 particles, which in turn depended on the TIMEFUZZ value that is hardcoded in many NEMO programs. s2a is from FALCON, which is has different methods. I fixed all of NEMO (we have a new version 4.5) that ensures one common TIMEFUZZ now defined in <stdinc.h> -

but this doesn't solve your s2a problem.

Did you try snaptrim? It has a keyword where you can set the timefuzz= parameter

[savchenkoyana]

Yes, I've just tried snaptrim, seems like it works, thanks!

May I ask a couple of questions?

• So just to be clear, should I use the default timefuzz=0.000001 for large snashots?
• Another question is: after I update NEMO to the current version 4.5.0, s2a won't be fixed, right? And snapprint too? Is there a way to know beforehand the name of other NEMO programs that still have the same peculiarities so that I'm super careful when using them?
• Does timefuzz problem only arise with super large snapshot or it could happen with normal snapshots when reading from disk is super slow?

[teuben]

My understanding of the problem is that it doesn't depend on the size of the snapshot, but on the times. They are too close to each other. I saw you are using fun units , where G is not 1, so perhaps your dynamical time is not of order unity, but much smaller?

…

On Mon, Dec 23, 2024, 01:00 Yana ***@***.***> wrote: Yes, I've just tried snaptrim, seems like it works, thanks! May I ask a couple of questions? - So just to be clear, should I use the default timefuzz=0.000001 for large snashots? - Another question is: after I update NEMO to the current version 4.5.0, s2a won't be fixed, right? And snapprint too? Is there a way to know beforehand the name of other NEMO programs that still have the same peculiarities so that I'm super careful when using them? - Does timefuzz problem only arise with super large snapshot or it could happen with normal snapshots when reading from disk is super slow? — Reply to this email directly, view it on GitHub <#162 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZ4MGOIMWI4XZ7AJQ32YYD2G6RHTAVCNFSM6AAAAABUAZ2YVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJYHE3DMMRZGM> . You are receiving this because you commented.Message ID: ***@***.***>

[teuben]

Slightly related is a bug in NEMO when an item is over 4GB (which doesn't happen with your snapshots) it seems to fail. This is mentioned in #17 and no doubt is an old relic of the code not converted to 64bit integers.