[ENH] Read Rockstar members

[cphyc]

PR Summary

This allow to read member data from Rockstar catalogues.

PR Checklist

• Adds a test for any bugs fixed. Adds tests for new features.

[neutrinoceros]

@yt-fido test this please

[neutrinoceros]

@cphyc I think adjustments are needed to pass the new test

[brittonsmith]

I'm really glad someone finally had the courage to implement this. Can you also add a section in the Rockstar frontend docs in doc/source/examining/loading_data.rst?

[brittonsmith]

I'm not sure where the conflicts came from, but the rest looked good to me. If they can be cleared up, I think this is ready to be merged.

[neutrinoceros]

@cphyc conflicts probably came from #4275
Happy to merge this once they're resolved

[neutrinoceros]

@yt-fido test this please

[neutrinoceros]

@yt-fido test this please

[neutrinoceros]

We have one failing test to deal with

[brittonsmith]

I cannot reproduce the failing test locally with a Python 3.10 install. Is there a way to get more information?

[cphyc]

I think this might be real. I need to investigate!

Weird; I have tried locally and cannot reproduce the error on 3.9, 3.10 or 3.11.

Here's the code I tried

import yt

ds = yt.load_sample("rockstar_halos/halos_0.0.bin")

for halo_id, Npart in zip(
    ds.r["halos", "particle_identifier"],
    ds.r["halos", "num_p"],
):
    print(halo_id)
    halo = ds.halo("halos", halo_id)
    assert halo is not None

    # Try accessing properties
    halo.position
    halo.velocity
    halo.mass

    # Make sure we can access the member particles
    assert len(halo.member_ids) == Npart

This should be the code of the failing test, unless I am mistaken.

[cphyc]

@yt-fido test this please.

[neutrinoceros]

@cphyc the test server is dead, see discussion on Slack.

[cphyc]

@yt-fido test this please

[cphyc]

Well, I am officially puzzled.

[matthewturk]

256 is suspiciously reminiscent of a field delimiter. I would guess that there is some location where a read is happening of 8 bytes on one and 4 bytes on the other, possibly due to some oddity in the way that dtypes are determined. I would not have thought this possible given the architectures but I'm also puzzled and grasping at things.

Perhaps it would be good to catch the error, and then print out the results of tell or something, to make sure we're at the same place.

[neutrinoceros]

@yt-fido test this please

[matthewturk]

Here's the final stdout:

Reading 25 particles for halo 295.0 dimensionless. position=404536
Reading 25 particles for halo 296.0 dimensionless. position=404736
Reading 2427 particles for halo 0.0 dimensionless. position=78664

And here's the relevant part of the assertion:

assert_equal(len(halo.member_ids), Npart)

Mismatched elements: 1 / 1 (100%)
Max absolute difference: 2171., units='dimensionless'
Max relative difference: 8.4805, units='dimensionless'
 x: array(2427)
 y: unyt_quantity(256., '(dimensionless)')

And then it fails with 2171 != 256. So it is getting a length of 2171 for the halo member ids, but it has read that NPart is 256.

NPart is read here:

            halos = np.fromfile(f, dtype=io._halo_dt, count=pcount)
            self._member_offset = f.tell()
            self._ids_halos = list(halos["particle_identifier"])
            self._Npart = halos["num_p"]

The location in the file for both is the same -- 78664 -- which leads me to believe that the problem isn't in the particle reading, but rather in the setting of NPart.

A few other comments about this -- 2427 is exactly 256 higher than 2171, and 2171 doesn't show up anywhere in the halo sizes. This suggests to me that there's some subtraction going on somewhere ... but maybe not.

[matthewturk]

can you promote this explicitly to 'i8'?

[cphyc]

@yt-fido test this please

[cphyc]

@yt-fido test this please

[cphyc]

@yt-fido test this please

[cphyc]

@yt-fido test this please

[neutrinoceros]

@yt-fido Test this please

[cphyc]

I made a bit of progress - it seems that the checksum of the data is not the same on fido as it is on my computer:

# locally
/home/…/rockstar_halos/halos_0.0.bin has sha256sum b'09accfb7ab6f162f29fcfb7c6393e207ba1be9d61cbdc230f17d39f0e90914d7'
/home/…/rockstar_halos/halos_0.1.bin has sha256sum b'42e64f974f7d72f201443d42463fd642f8e37c7c485eac54810e3be8adfd0a22'
# on yt-fido
/mnt/yt/rockstar_halos/halos_0.0.bin has sha256sum b'd4c395708e153ae9ec32873c1b1b13f52ee5e829b6cee34e2ab91c67fe8bdf20'
/mnt/yt/rockstar_halos/halos_0.1.bin has sha256sum b'42e64f974f7d72f201443d42463fd642f8e37c7c485eac54810e3be8adfd0a22'

I should note that the local data was downloaded through yt.load_sample, so it should reflect whatever is on https://yt-project.org/data/.
@Xarthisius do you have an idea what's happening?

[cphyc]

I am quite confident is that the test data isn't the same as the one online. @Xarthisius would it be possible to make sure that the dataset on jenkins is the same as the one obtained when using load_sample("rockstar_halos/halos_0.0.bin")?

I have tested locally and it works like a charm.

[cphyc]

@yt-fido Test this please

[neutrinoceros]

adding the "enhancement" label just to make Mergeable Bot happy