You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've seen similar issues with ZarrDatasets.jl where data is stored on S3. raster[:, :] uses the concurrent I/O but collect(raster) somehow takes 20x the time, and network download speed is 10x lower.
It's because CF variable is not a disk array so it doesn't have all the methods overrides to make these things chunked and fast.
The underlying Variable is, but you have to go around CFArray manually to get the chunks as on Rasters main, or just manually wrap Variable with custom CF as in Rasters cf branch.
It would be better long-term to make AbstractVariable <: AbstractDiskArray but I'm sure people are tired of hearing me say that ;)
In Rasters we hit a bug where I had forgotten to re-wrap CFVariable in our internal
CFDiskArray
made specifically for fixing this CommonDataModel bug.This one small change made writing fiies 100x faster
https://github.com/rafaqz/Rasters.jl/pull/633/files
Broadcasting to/from a CFVariable means reading/writing for every single pixel
The text was updated successfully, but these errors were encountered: