Skip to content

Step By Step ALE Driver Creation

Jump to bottom
Jacob Cain edited this page Sep 20, 2023 · 15 revisions

Step by Step ALE Driver Creation

How to write an ALE driver for an instrument that is already implemented in ISIS. For these instructions, bracketted text should be replaced by names specific to your data/instruments.

  1. ISIS Prerequisites

  2. Fork and clone the ALE repository.

  3. Setup an ALE environment

    • Go to the base directory in your cloned ALE repo
    • Create and activate a conda environment: 
      conda env create ale -f environment.yml
conda activate ale
    • Set variables and add ISIS binaries to PATH: 
      export ISISROOT=[myRepoLocation]/ISIS3
export PATH=$PATH:[myRepoLocation]/ISIS3/bin
    • Uninstall any preexisting versions of ALE and link in the development version: 
      pip uninstall ale #run 2 or 3x to make sure
python setup.py develop

  4. Open/initialize related kernels and/or cubes for reference

    • See Data Sources for where to find Kernels and Cubes
    • Copy a Cube (and maybe the Kernels as well) to a local data area for testing.
      • Ideally, use an uninitialized cub (not a lvl2 or lvl3).
        • Level 0 - has spice data attached
        • Level 1 - ingested with no processing
        • Levels 2 & 3 - projected
      • Open up the Cube in a text editor to make sure it is a Cube from the right instrument.
    • In the local data area, make 2 copies of the Cube for testing, one with ALE and one with ISIS 
      [cubename].ale.cub
[cubename].isis.cub
    • Before you start developing you ale driver, initialise the ISIS Cube with spiceinit. 
      spiceinit from= [cubename].isis.cub
    • ISIS's spiceinit uses the first driver that works, first looking for an ALE driver, then using an ISIS driver if all ALE drivers fail. If you need to initialize a cube with an ISIS driver later, temporarily put a line of code in the ALE driver that will cause it to fail.

  5. Add a file for your driver

    • Go to the ale/base/drivers drivers directory
    • If it doesn't already exist, create a python file with the name of the spacecraft followed by _drivers (e.g. hayabusa_drivers.py)
    • You may want to copy the structure from another _driver class to start out

  6. Write your class

    • See Class Signature below to name class and select mixins.
    • Add these common functions (and others as needed):
      • instrument_id
        • map the sensor name to the spacecraft
        • spacecraft name as shown in the NAIF Toolkit Docs
        • Look in the Kernel for a more specific ID that might also include the instrument.
      • sensor_model_version
        • Should almost always return 1

  7. Perfect your ALE driver so you can spiceinit a cube with it.

    • Run spiceinit on the ALE cube (spiceinit from= [cubename].ale.cub)
    • Look for errors from the ale driver ([spacecraftname]_driver.py). This should let you see what methods haven't been covered by mixins. You will need to write these methods yourself in the driver.
    • Based on the errors you receive, add necessary methods, test, and repeat until spiceinit succeeds with the ale driver and outputs a cube from source = ale (and not source = isis).
      • To set your ISIS conda environment to use your local ALE instance: 
        # Activate your ISIS conda environment
$ conda activate isis3

# Check which ALE dependency ISIS is using
$ conda list ale

# Remove only ALE and not its dependencies
$ conda remove --force ale

# Install dev build to ISIS's conda dir
$ cd <path-to-ale-repo>/ale/build
$ rm -rf CMakeCache.txt
$ cmake .. -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX/
$ make
$ make install
$ cd ..
$ python setup.py develop

# Rebuild ISIS to point to dev build
$ cd ../ISIS3/build
$ rm -rf CMakeCache.txt
$ cmake -DJP2KFLAG=ON -Gninja <path-to-isis-repo>/ISIS3/isis
$ ninja -j24

# Verify ALE dependency is dev build
$ conda list ale

  8. Compare Ale Cube to ISIS Cube

    • Generate Labels for the ISIS and ALE Cubes 
      catlab from= [cubename].isis.cub to= [cubename]_isis.lbl
catlab from= [cubename].ale.cub to= [cubename]_ale.lbl
    • Compare ALE Label to the ISIS Label to check the accuracy (using the ISIS label as truth data). 
      vimdiff [cubename].isis.lbl [cubename].ale.lbl
    • TODO: What values are particularly important to compare, and what are their difference tolerances?

  9. Slice the Kernels

    The Kernel Slicing notebook uses ckslicer, spkmerge, and toxfr from the NAIF Toolkit Utilities. It attempts to extract info from the .cub/kernels to automatically generate the arguments needed to slice the kernels. If the Kernel Slicing notebook fails, you can slice the kernels manually by running the utilities with the appropriate arguments (see the user guides on the NAIF Toolkit Utilities page).

    • Go to the notebooks subdirectory and start a jupyter notebook
    • Open the KernelSlice.ipynb notebook.
    • You will likely have to make minor changes to the Kernel Slicer's code, reflecting any unique functions in your driver, especially:
      • Accessing the cube_pvl data structure for properties named differently in your driver.
      • The first few lines after the imports, with the locations of:
        • Your test Cube
        • The isis_data directory from the ISIS remote data area
        • An output directory
      • Note: Requires NAIF program ckslicer (MacOS) [internal devs may need to contact IT to install the software on their machine]
    • ⚠️ After slicing the kernels, remove any absolute paths (especially paths including /Users/yourusername/):
      • From the kernel slicing Jupyter Notebook and its output
      • From the transfer kernels (it is safe to delete the metadata fields that include your absolute paths)

  10. Copy Test Data

    • In tests/pytests/data, make a directory named after your test cube.
    • See Test Data for what data you will need to copy into here.
    • Copy an ISD into tests/pytests/data/isds (See Generating ISDs)

  11. Test with PyTest

    • In ale/pytests/ make a python file called test_[spacecraftname]_drivers.py
    • To start out, you may want to look at the structure of another test.
    • Make a kernels test function and a load test function.
    • Make a class with a test for each function you added to your driver. They should assert the value that was set in that function.
      • if your driver function calls other functions (like scs2e or gdpool), you might want to patch a value in so the return is always the same.
    • Run your test, fix mismatched asserts, and fix failing tests. 
      pytest tests/pytests/test_[spacecraft]_drivers.py

Data Sources

Kernels

  • A combination of Documentation and Metadata
  • isisDataArea/isis_data/[spacecraftname]/kernels/ik
  • iak (instead of ik) for addendum kernels

PDS Image Atlas

Lunar Orbital Data Explorer

ISIS test Cubes

  • Cubes contain metadata and an image from a spacecraft.
  • isisDataArea/isis_testData/isis/src/[spacecraftname]/unitTestData

Generating Cubes

  • If the cube you want is absent, generate it with an ISIS program, documented here.
    (look for an [instrument]2isis program under the appropriate mission.)
  • Test Data can often be found here: 
    /isis_testData/isis/src/[spacecraft]/apps/[sensor]2isis/tsts/[varies]/input`

Camera CPPs implemented in ISIS

  • myRepoLocation/ISIS3/isis/src/[spacecraftname]/objs/[sensorname]/[sensorname].cpp

Labels and Fields

For more info on what a specific label or field might mean, check these sources:

Internet

Class Signature

The class signature for your driver should follow this format. See each letter below for guidance on how to name the class/what mixins to use.

# e.g, class HayabusaAmicaIsisLabelNaifSpiceDriver(Framer, IsisLabel, NaifSpice, RadialDistortion, Driver):
class [A][B][C][D]Driver ([E], [C], [D], [F], Driver):

A - Spacecraft Name

The name of the spacecraft and/or mission (e.g. Messenger, Hayabusa, Galileo, Odyssey).

B - Instrument Name

The name of the specific sensor on the spacecraft (e.g. AmicaCamera, NarrowAngleCamera, ThemisIr). Often shortened to an acronym or abbreviation (TO DO: When is it shortened and how do I find the short name?).

C - Label Type: IsisLabel or PDS3Label

  • Most common: IsisLabel
  • PDS3Label is rare
  • If dealing with an ISIS converted cube .lbl -> ISIS Label.
  • If you haven't done a x_to_isis conversion -> PDS3

D - Auxilliary Data Type: IsisSpice or NaifSpice

  • Most common: NaifSpice
  • IsisSpice is rare
  • If the spice data is already attached, IsisSpice can use the data already attached

E - Camera Type: Framer, LineScan, or (rare) PushFrame

Can be found in ISIS Implementation.

Possibly from the instrument Kernel: see if the CENTER is in the middle of the square (framer) or the first line (line scan). Or the kernel documentation may say directly.

F - Distortion Type

  • RadialDistortion and then NoDistortion are the most common
  • Usually can find in the IK or IAK in isis_data
  • Look in Kernel or IAK for ODK, ODTK, Distortion, Optical
  • TRANSX and TRANSY are distortion coefficients

G - Last Mixin is always Driver

Specific Methods

instrument_id

lookup_table = {A:B}

  • A - look in cube label (isis_testdata/isis/src/[spacecraft name]/unitTestData)
  • B - Google “NAIF SPICE Integer ID” and Ctrl + F the spacecraft name, OR look in the kernel for a more specific ID that might also include the instrument.

sensor_model_version

Usually just returns 1.

Test Data

  1. In the last cell of the KernelSlice.ipynb Jupyter Notebook, is the kernels command. As the last step of the Kernel Slicing, run this command to generate a list of the necessary Kernels.

  2. For any Binary Kernels ending in .bsp or .bc, the .xsp and .xc, Transfer Kernels generated in the Kernel Slicing output directory should take their place in the ALE test data for your instrument.

  3. For the rest of the Kernels listed (not ending in .bsp or .bc), copy them into the the ALE test data for your instrument.

  4. Check if your test data folder is missing kernels (like fk and sclk) and copy those kernels from the isis_data area into your test data folder.

  5. Copy/paste the naif0012.tls kernel from another ALE test data area.

  6. Finally, the isis label (that we made with catlab, ending in _isis.lbl) is needed as truth data. Copy it into the ALE test data directory

For more info on these kernels and what each is for, see the Intro to Kernels or read more about specific Kernels in the NAIF Toolkit Docs.

Generating ISDs

ℹ️ Note
The process for generating ISDs may change soon. A Jupyter Notebook is being written to help with this process. If the Jupyter Notebook is complete, please document how to use it here and remove this notice.

To create an ISD (Image Support Data), you will need the output from spiceinitting a Cube.

spiceinit FROM=[cubename].ale.cub > [spacecraft][sensor]_isd.json

If your terminal has a long enough scrollback history, you can simply copy the output from the terminal instead of piping it to a file.

Open this data in a text editor. Keep only the JSON code block following ISD: from the opening bracket to the corresponding closing bracket, excluding ISD: at the start and everything before it, and excluding Group = Kernels and everything after it at the end.

Sample output from spiceinit 
// ...
Trying <class 'ale.drivers.hayabusa_drivers.HayabusaAmicaIsisLabelNaifSpiceDriver'>
Success with:  <ale.drivers.hayabusa_drivers.HayabusaAmicaIsisLabelNaifSpiceDriver object at 0x169d29ed0>
ISD:
// Exclude everything before this
{
  "isis_camera_version": 1,
  "image_lines": 401,
  "image_samples": 401,
  "name_platform": "HAYABUSA",
  "name_sensor": "HAYABUSA_AMICA",
  "reference_height": {
    "maxheight": 1000,
    "minheight": -1000,
    "unit": "m"
// ...
    "velocities": [
      [
        3.591316801764071,
        27.34212273912132,
        12.079942252825946
      ]
    ],
    "reference_frame": 1
  }
}
// Exclude everthing after this
Group = Kernels
  NaifFrameCode             = -130102
  LeapSecond                = $base/kernels/lsk/naif0012.tls
  TargetAttitudeShape       = ($base/kernels/pck/pck00009.tpc,
// ...

You should end up with a file named [spacecraft][sensor]_isd.json that contains the only the JSON data for the isd.

Clone this wiki locally