Unscrambling obfuscated pixel data in Carl Zeiss Meditec (CZM) ophthalmic datasets

[scaramallion]

I was recently asked to try and decipher some extremely non-conformant DICOM datasets nominally from Carl Zeiss Meditec (CZM) ophthalmic equipment for producing OCT angiography and related images, such as the Cirrus range. The datasets are non-conformant in multiple ways, but perhaps the most serious is that CZM appear to have deliberately obfuscated the pixel data to prevent others from using it.

I'm happy to say I've succeeded in reversing that obfuscation, although for a while there I really thought it wouldn't be possible. I'm posting the solution here for two reasons:

• To help other people who have come across these datasets and would like to be able to decode the pixel data
• The deliberate obfuscation of the pixel data by CZM really annoys me, because it goes completely against the interoperability spirit of DICOM, so the larger the audience the better

The pixel data is obfuscated in two ways:

• First the J2K data has sections transposed (for example, the first 253 bytes are moved to 3/5ths of the way into the data)
• Once the transpositions are done, every 7th byte starting at offset 0 is overwritten by XORing it with 0x5A

The sample datasets I received were horribly non-conformant, but I'm not sure if that was an artefact of how they were obtained or if that's actually how CZM equipment produces them (in which case, wow). The pixel data also has a JP2 header, which again is non-conformant to the DICOM Standard, so be aware of that and remove it if required (search for the 0xFF 0x4F start-of-codestream J2K marker, which should be at offset 85 of the unscrambled data).

So without further ado, here's how to unscramble the pixel data and write it to file as a sane JPEG 2000 file with JP2 header. Alternatively you could uncompress the unscrambled frame using Pillow or your preferred third-party J2K library.

# unscramble.py

import argparse
import math
from pathlib import Path

from pydicom import dcmread
from pydicom.encaps import generate_pixel_data_frame
from pydicom.uid import JPEG2000Lossless


def setup_argparse() -> argparse.Namespace:
    """Setup the command line arguments"""
    parser = argparse.ArgumentParser(
        description=(
            "Read a CZM DICOM dataset and extract and unscramble the pixel "
            "data which is then written to file as one or more JPEG 2000 "
            "images"
        ),
        usage="unscramble path",
    )

    req_opts = parser.add_argument_group("Parameters")
    req_opts.add_argument(
        "path",
        help="The path to the CZM DICOM dataset",
        type=str
    )

    return parser.parse_args()


def unscramble_czm(frame: bytes) -> bytearray:
    """Return an unscrambled image frame.

    Parameters
    ----------
    frame : bytes
        The scrambled CZM JPEG 2000 data frame as found in the DICOM dataset.

    Returns
    -------
    bytearray
        The unscrambled JPEG 2000 data.
    """
    # Fix the 0x5A XORing
    frame = bytearray(frame)
    for ii in range(0, len(frame), 7):
        frame[ii] = frame[ii] ^ 0x5A

    # Offset to the start of the JP2 header - empirically determined
    jp2_offset = math.floor(len(frame) / 5 * 3)

    # Double check that our empirically determined jp2_offset is correct
    offset = frame.find(b"\x00\x00\x00\x0C")
    if offset == -1:
        raise ValueError("No JP2 header found in the scrambled pixel data")

    if jp2_offset != offset:
        print(
            f"JP2 header found at offset {offset} rather than the expected "
            f"{jp2_offset}"
        )
        jp2_offset = offset

    d = bytearray()
    d.extend(frame[jp2_offset:jp2_offset + 253])
    d.extend(frame[993:1016])
    d.extend(frame[276:763])
    d.extend(frame[23:276])
    d.extend(frame[1016:jp2_offset])
    d.extend(frame[:23])
    d.extend(frame[763:993])
    d.extend(frame[jp2_offset + 253:])

    assert len(d) == len(frame)

    return d


if __name__ == "__main__":
    args = setup_argparse()

    p = Path(args.path).resolve(strict=True)

    # Read and check the dataset is CZM
    ds = dcmread(p)
    meta = ds.file_meta
    if meta.TransferSyntaxUID != JPEG2000Lossless:
        raise ValueError(
            "Only DICOM datasets with a 'Transfer Syntax UID' of JPEG 2000 "
            "(Lossless) are supported"
        )

    if not ds.Manufacturer.startswith("Carl Zeiss Meditec"):
        raise ValueError("Only CZM DICOM datasets are supported")

    if "PixelData" not in ds:
        raise ValueError("No 'Pixel Data' found in the DICOM dataset")

    # Iterate through the frames, unscramble and write to file
    if "NumberOfFrames" in ds:
        # Workaround horrible non-conformance in datasets :(
        if isinstance(ds.NumberOfFrames, str):
            nr_frames = ds.NumberOfFrames.split('\0')[0]
        else:
            nr_frames = ds.NumberOfFrames

        frames = generate_pixel_data_frame(ds.PixelData, int(nr_frames))
        for idx, frame in enumerate(frames):
            with open(f"{p.stem}_{idx:>03}.jp2", "wb") as f:
                f.write(unscramble_czm(frame))
    else:
        # CZM is non-conformant for single frames :(
        with open(f"{p.stem}.jp2", "wb") as f:
            f.write(unscramble_czm(ds.PixelData))

[elliottvg]

I've just spend 2 days doing exactly the same thing only to be sent this link by a coworker :D

[etuejcm]

Your work is really helpful!
I'm doing the same thing recently, but I cant find where the OCTA data is. Could you give some suggestions?
Thanks a lot!

[kevinpasquinelli]

@scaramallion: do you have a link to an example of the DICOM file in question?

[scaramallion]

Nothing I can share, sorry.

[rubenhx]

@scaramallion this is impressive, and very useful in my current project. For educational purposes, could you briefly jot down how you approached this issue? Thanks for doing this!

[scaramallion]

The 0x5A XORing was fairly easy to spot given its regularity - it messed up the J2K header in the datasets, so the fix was obvious.

Undoing the transpositions took a lot more work, I was able to identify the block at jp2_offset because it contained the start of the J2K codestream and the associated header. The rest was possible due to the format of J2K codestreams (which has the codestream split into packets for each image quality layer, with each packet having it's own header) as well as having access to quite a few different datasets.

Luckily a couple of the datasets had a transposition occurring partway through the header for a J2K codestream packet, so they were fairly easy to line up. Where that wasn't the case I could read the packet data and determine how far it should be to the next packet and use that information to determine if a transposition occurred somewhere in the middle, then brute force a possible solution to be tested to see if it could be decoded - J2K is pretty resistant to errors in a given packet so as long as you're making progress the decode should work and the image quality should improve in the result.

[rubenhx]

Thanks a lot @scaramallion. Again, very impressive that you were able to reverse 'scramble' this (deliberately not using the word 'engineering').