Zarr implementation in Scala
Example usage:
import lasersonlab.zarr._ // bring all things zarr into scope
// Schema for a type that we will instatiate and then save as a Zarr "group" (a top-level directory with
// subdirectories for each "array" field
case class Foo(
ints: Array[`2`, Int], // 2-D array of ints
doubles: Array[`3`, Double] // 3-D array of doubles
)
// Instantiate a Foo
val foo =
Foo(
ints = Array( 1000 :: 1000 :: ⊥)( 1 to 1000000 ),
doubles = Array(100 :: 100 :: 100 :: ⊥)((1 to 1000000).map(_.toDouble))
)
// Save as a Zarr group: a directory containing "ints" and "doubles" subdirectories, each a Zarr array:
foo.save(Path("foo"))At this point, the directory foo will look like:
foo
├── [ 24] .zgroup
├── [ 128] doubles
│ ├── [ 368] .zarray
│ └── [ 58864] 0.0.0
└── [ 128] ints
├── [ 346] .zarray
└── [ 36267] 0.0
The ints and doubles arrays are each written out in one "chunk"-file; by default, chunks are comprised of as many "rows" as will fit in 32 MB (uncompressed).
This number can be tweaked by providing an implicit ChunkSize:
implicit val chunkSize: ChunkSize = 1 MB
val foo =
Foo(
ints = Array( 1000 :: 1000 :: ⊥)( 1 to 1000000 ),
doubles = Array(100 :: 100 :: 100 :: ⊥)((1 to 1000000).map(_.toDouble))
)
foo.save(Path("foo.1m"))yields:
zarr/foo.1m
├── [ 24] .zgroup
├── [ 352] doubles
│ ├── [ 367] .zarray
│ ├── [ 10858] 0.0.0
│ ├── [ 9814] 1.0.0
│ ├── [ 10311] 2.0.0
│ ├── [ 9552] 3.0.0
│ ├── [ 10013] 4.0.0
│ ├── [ 9349] 5.0.0
│ ├── [ 9358] 6.0.0
│ └── [ 7964] 7.0.0
└── [ 224] ints
├── [ 345] .zarray
├── [ 9436] 0.0
├── [ 9466] 1.0
├── [ 9490] 2.0
└── [ 8736] 3.0
Chunk-sizes can also be passed explicitly to the Array constructor:
Array(
1000 :: 1000 :: ⊥,
500 :: 100 :: ⊥
)(
1 to 1000000
)Random-accesses can be performed like:
foo.ints(123 :: 456 :: ⊥)
// 123457"Folds" also work:
foo.doubles.foldLeft(0.0)(_ + _)
// 5.000005E11cats.Foldable syntax works, but requires the .t accessor:
foo.ints.t.toList
// List(1, 2, 3, … 999999, 1000000)More sophisticated traversals and manipulations are still TODO.