Skip to content

Uncorrect compression (or decompression) for some lengths not divisible by 8 bytes #357

Open
@byphilipp

Description

@byphilipp

Describe the bug
The result of compression and decompression with
filters=[blosc2.Filter.SHUFFLE, blosc2.Filter.BYTEDELTA]
for some lengths not divisible by 8 bytes
is not equvivalent: the last byte is differs

To Reproduce

import blosc2
import numpy as np
dtype = np.float32
for N in range(100,270):
    x = np.random.rand(N).astype(dtype)
    kw=dict(
            clevel=5,
#            filters=[blosc2.Filter.SHUFFLE], # correct
#            filters=[blosc2.Filter.BYTEDELTA], # correct
#            filters=[blosc2.Filter.BYTEDELTA, blosc2.Filter.SHUFFLE], # correct
             filters=[blosc2.Filter.SHUFFLE, blosc2.Filter.BYTEDELTA], # uncorrect !
    )
    y = np.frombuffer( blosc2.decompress2 (blosc2.compress2(np.frombuffer(x, dtype=np.uint8).copy(order='C'), codec=blosc2.Codec.ZSTD, **kw)), dtype=dtype)
    #y = np.frombuffer( np.frombuffer(x, dtype=np.uint8), dtype=dtype)
    if y[-1]-x[-1] != 0:
        print(f"{N=} {y[-1]-x[-1]}")

output:

N=195 -0.1865813434123993
N=223 -0.20511126518249512
N=225 2.6814947806386197e+37
N=227 -0.2120416760444641
N=229 1.0758920518724525e+25
N=235 -0.8137646317481995
N=237 -0.5785623788833618
N=239 -445465753550848.0
N=241 344564.53125
N=247 -6.356997702928093e+27
N=249 6891780.0
N=251 -0.4392470419406891
N=253 -1.2648026543392063e+29
N=257 -101628403712.0
N=259 56037704.0
N=261 -0.7367768287658691
N=263 -0.5903393626213074
N=265 -0.5874314308166504
N=267 -0.1973332017660141
N=269 -0.1740696132183075

Expected behavior
Empty output

Desktop (please complete the following information):

  • OS: Ubuntu 22.04
  • Python 3.11
  • blosc2 - I try 3.0.0, 2.7.1, 2.6.2 and see this problem in all versions

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions