ros3 vs. s3fs

[bendichter]

Here I am comparing two different ways of directly accessing s3 buckets: the HDF5 ros3 driver vs. s3fs. It looks like we will gain substantial performance benefits from switching to s3fs.

In order to get this to work, I had to relax a few input validation mechanisms in both pynwb and hdmf. See changes here and here.

Tests were done on DANDI Hub.

First, I set up the environment:

!pip install git+https://github.com/NeurodataWithoutBorders/pynwb.git@s3fs
!pip install git+https://github.com/hdmf-dev/hdmf.git@s3fs
pip install ndx-events

nwb-events is needed because we don't have loading namespaces working for s3fs yet. I don't think this would be hard to implement, but it didn't seem necessary for this performance test.

import s3fs
import pynwb
import h5py
from dandi.dandiapi import DandiAPIClient
import ndx_events  # needed because we can't yet load namespaces dynamically from s3fs

Then I get the s3 location for a Neuropixel NWB file. s3fs requires the url in a slightly different form, but it's the same underlying data.

# neuropixels, Giocomo Lab (46 GB)
dandiset_id, filepath = "000053", "sub-npI1/sub-npI1_ses-20190415_behavior+ecephys.nwb"


with DandiAPIClient() as client:
    asset = client.get_dandiset(dandiset_id, "draft").get_asset_by_path(filepath)
    s3_url = asset.get_content_url(follow_redirects=1, strip_query=True)
    
print(s3_url)
bucket = "s3://dandiarchive/blobs" + s3_url.split("blobs")[1]
print(bucket)

Now let's get a baseline for ros3 read

%%time

io = pynwb.NWBHDF5IO(s3_url, mode="r", driver="ros3")
s3_nwbfile = io.read()

The first time I execute this it takes about 4 seconds. Interestingly, subsequent runs take about 0.15 seconds, which indicates that there must be some caching going on.

The analogous lines for s3fs:

%%time


fs = s3fs.S3FileSystem(anon=True)

f = fs.open(bucket, 'rb')
file = h5py.File(f)
io = pynwb.NWBHDF5IO(path='',mode='r',file=file)

s3fs_nwbfile = io.read()

take about 4.5 seconds. Consistently longer than ros3, but not by much.

Then I tried data read for ros3:

%%time

data = s3_nwbfile.acquisition["ElectricalSeries"].data[:100]

CPU times: user 2.2 s, sys: 284 ms, total: 2.48 s
Wall time: 8.14 s

compared to data read for s3fs:

%%time

data = s3fs_nwbfile.acquisition["ElectricalSeries"].data[:100]

CPU times: user 1.4 s, sys: 625 ms, total: 2.02 s
Wall time: 3.03 s

My findings are that with s3fs you pay a small time penalty on initialization over ros3, but then you can read data roughly 2.5-3x as fast (it varies by run). Also, it looks like the ros3 driver is caching header info so subsequent nwbfile initializations are really fast but s3fs is not. Note that this does not appear to be the case for the dataset values, only the header info. I don't think this is much of a benefit though because I don't expect users to be initializing files multiple times.

[bendichter]

Are there any important tests I missed?

[yarikoptic]

could it be done systematically over a set of "prototypical" nwb files in the archive from different dandisets? Our typical use cases could be extraction of metadata, and then for completeness - loading of the entire file/data.

FWIW, just for reference open issue/discussion of local caching for s3ds from versioned bucket like ours

[oruebel]

Would be interesting to do some more detailed profiling to see where time is being spent. In particular, it would be interesting to see more details on the network traffic (e.g., how many messages, size of messages etc.).

[bendichter]

With a little more, it looks like data read time in non-linear:. Here I am reading more rows of the dataset and measuring the time.

[yarikoptic]

hard to say about non-linearity with only those samples imho -- there is a more or less consistent initial overhead (>= 2.5/7.8 secs s3fs/ros3) and then I guess about 2.4/4.4 sec per 100,000 rows ;) overall s3fs is likely to be winning since just greedily gets bigger chunks (iirc 5MB blocksize default of fsspec) than ros3 (probably goes with smaller chunk size) thus making it speedier. I wish ROS3 was "smarter" in parallelizing the requests since it should know better on how much data is being requested ahead of time (so can parallel requests) whenever more basic s3fs can only know about blocksize client asks to read AFAIK.