Pre-discover all leaf files for dataset creation.

[delucchi-cmu]

There's some weird interaction between arrow, fsspec, and HTTP, whereby discovery of nested files doesn't behave well and the dataset appears empty, when calling parquet.dataset() with an HTTP directory.

This change recursively finds all files under the Norder* directories, discards more directories, and cleans up the URL string (e.g. Norder%3D0/... to Norder=0/...), so they match the relative paths contained in the _metadata file.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.84%. Comparing base (98205be) to head (82c152c).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #501      +/-   ##
==========================================
+ Coverage   94.82%   94.84%   +0.01%     
==========================================
  Files          28       28              
  Lines        1818     1823       +5     
==========================================
+ Hits         1724     1729       +5     
  Misses         94       94              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hombit]

Looks good!