Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/grid tiles #393

Merged
merged 83 commits into from
Oct 31, 2023
Merged

Feature/grid tiles #393

merged 83 commits into from
Oct 31, 2023

Commits on May 9, 2023

  1. Stashing - GridTiles

    milos.colic committed May 9, 2023
    Configuration menu
    Copy the full SHA
    e0fab1e View commit details
    Browse the repository at this point in the history

Commits on May 30, 2023

  1. Add CRS method to Index Systems.

    Add OSR SpatialRef method to IndexSystem.
    Add CustomIndexSystem with CRS support.
    Define RST_GridTiles.
    Add support for osrTransformCRS for MosaicGeometry.
    Add extent method to MosaicGeometry.
    Add isEmpty to MosaicRaster.
    Define MosaicRasterChip.
    Add support for getRasterForCell to MosaicRaster.
    Define RasterClipByVector operator.
    Define RasterTessellate operator.
    Define RST_GridTiles expression.
    Define SpatialRefUtils.
    milos.colic committed May 30, 2023
    Configuration menu
    Copy the full SHA
    f6cb725 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2023

  1. Major changes:

    - Add GDALBinaryFileFormat that supports manipulating rasters in memory.
    - Add GDALWarp operator that parses single line gdalwarp operation.
    - Introduce storage location concept for rasters, in-memory vs disk.
    - Simplify tiling logic through usage of gdalwarp.
    - Remove saveCheckpoint logic.
    - Add writeToPath and writeToBytes logic for rasters.
    - Update RST_ReTile to support both in-memory and disk.
    - Define RST_GridTiles that tessellate raster into grid based rasters.
    - Add VectorClipper that generates in-memory .shp clip file.
    Minor changes:
    - Add _tmp to gitignore.
    - Add logic for easy testing of display in local Jupyter.
    milos.colic committed Jun 4, 2023
    Configuration menu
    Copy the full SHA
    fc4f415 View commit details
    Browse the repository at this point in the history
  2. Update gitignore.

    milos.colic committed Jun 4, 2023
    Configuration menu
    Copy the full SHA
    f0eef62 View commit details
    Browse the repository at this point in the history
  3. Major changes:

    - Update GDALBinaryFileFormat to be able to read rasters both as paths and as byte arrays.
    - Update all RasterExprs to support rasters both as paths and as byte arrays.
    - Update tests to test for 2 types of rasters.
    Minor changes:
    - Remove large tif test file.
    - Remove python tif test.
    - Make ZIP resolution automatic.
    milos.colic committed Jun 4, 2023
    Configuration menu
    Copy the full SHA
    94effd2 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2023

  1. Major changes:

    - Consolidate GDALFileFormat and GDALBinaryFileFormat.scala to a single API.
    - Fix the way ZIPs are handled in memory.
    Minor changes:
    - Remove large tif test file.
    - Remove python tif test.
    - Make ZIP resolution automatic.
    milos.colic committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    abf3f24 View commit details
    Browse the repository at this point in the history
  2. Properly name the raster column in the schema.

    Add Mosaic UUID to raster metadata at readtime.
    Unify unlink and cleanup into cleanup.
    Update tests.
    milos.colic committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    964cfbc View commit details
    Browse the repository at this point in the history
  3. Minor fixes for UUID.

    milos.colic committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    f772460 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2023

  1. Merge branch 'main' into feature/grid_tiles

    # Conflicts:
    #	docs/source/usage/kepler.ipynb
    milos.colic committed Jun 11, 2023
    Configuration menu
    Copy the full SHA
    35e0fef View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2023

  1. Add BalancedSubdivision for dividing rasters into pieces that maintai…

    …n x/y ratio.
    
    Add GDALBuildVRT for executing build VRT programs.
    Add GDALTranslate for executing translate programs.
    Add GDALWarp for executing warp programs.
    Refactor GDALFileFormat to support ReadStrategies.
    Handle Axis Orientation in IndexSystem.
    Add MergeRasters operator.
    Add PathUtils to centralise all the path handling for GDAL and FUSE locations.
    Add RasterChipType.
    Add RasterCleaner for handling deletion and flushing of rasters between stages.
    Add RST_Tessellate expression.
    Add RST_Merge and RST_MergeAgg expressions.
    Add ReTileOnRead read strategy.
    Add ReadAsPath read strategy.
    Add ReadInMemory read strategy.
    milos.colic committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    cfe0a04 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2023

  1. Update the way GDAL is installed.

    Add demo notebooks for prototype.
    milos.colic committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    789b170 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2023

  1. Add new installable and new example notebook to prototypes.

    Fix flushing in operators.
    milos.colic committed Jul 26, 2023
    Configuration menu
    Copy the full SHA
    4ee50f9 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2023

  1. Fix aux.xml issues with rasters in mosaic.

    milos.colic committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    23fb47f View commit details
    Browse the repository at this point in the history
  2. Fix aux.xml issues with rasters in mosaic.

    milos.colic committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    bb85cb3 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2023

  1. Fixing unnecessary IO.

    milos.colic committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    dbaebc2 View commit details
    Browse the repository at this point in the history
  2. Fixing unnecessary IO.

    milos.colic committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    ee80140 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2023

  1. Add examples for stacking bands into a single raster.

    Add functionality for band stacking.
    milos.colic committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    e4588ea View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2023

  1. Improve data prep scripts.

    milos.colic committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    f220d75 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. Fix missing spatial reference when generating index geom.

    Fix SR transformations in MosaicGeometry.
    Add NDVI and Clip expressions.
    Fix projection issues in RST_Tessellate.
    milos.colic committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    cd47656 View commit details
    Browse the repository at this point in the history
  2. fix H3 geometry constructor (#374)

    * fix H3 geometry constructor
    
    * fix function names
    
    * add test for str function
    
    * lint with scalafmt
    
    * add comments
    
    (cherry picked from commit 8a83eed)
    Thomas Maschler authored and milos.colic committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    076bdc2 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2023

  1. Cleanup notebooks.

    Improve data download.
    Add NDVI function.
    Add SAM example.
    milos.colic committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    ed9d80e View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2023

  1. Upload the correct wheel.

    milos.colic committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    c3025bc View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. Fix NDVI and missing pixel issues.

    milos.colic committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    ddab8cc View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2023

  1. Revise all RST_ expressions.

    Clean up raster disposing.
    Clean up duplicated expressions.
    milos.colic committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    43e483f View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. Add the concept of raster-tile to the schema and types.

    Update the logic of expressions and file readers.
    Update tests.
    milos.colic committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    3d6b648 View commit details
    Browse the repository at this point in the history
  2. Remove a comma.

    milos.colic committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    e2455b1 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'main' into feature/grid_tiles

    Milos Colic authored Oct 11, 2023
    Configuration menu
    Copy the full SHA
    8e18317 View commit details
    Browse the repository at this point in the history
  4. Use null for non-specified cell ids.

    milos.colic committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    cfacf5f View commit details
    Browse the repository at this point in the history
  5. Merge remote-tracking branch 'origin/feature/grid_tiles' into feature…

    …/grid_tiles
    milos.colic committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    30f8b31 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2023

  1. Add python signatures.

    Fix index id types.
    milos.colic committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    2a7c34c View commit details
    Browse the repository at this point in the history
  2. applied python formatter

    sllynn committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    35526c0 View commit details
    Browse the repository at this point in the history
  3. fixed python enable_gdal

    sllynn committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    509cf31 View commit details
    Browse the repository at this point in the history
  4. Fix clip.

    milos.colic committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    bef9f04 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    1f886ab View commit details
    Browse the repository at this point in the history
  6. moved MosaicRasterBandGDAL

    sllynn committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    ee1a4ac View commit details
    Browse the repository at this point in the history
  7. restructed core.raster

    sllynn committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    41b9dae View commit details
    Browse the repository at this point in the history
  8. restructed core.raster

    sllynn committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    6d4067c View commit details
    Browse the repository at this point in the history
  9. restructed core.raster

    sllynn committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    9b79122 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2023

  1. Remove redundant UUID definitions.

    Remove Murmur hashing of UUIDs.
    Add a check for tests to warning prints.
    Fix data types in MergeAgg and RasterExpressionSerialization.
    milos.colic committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    7170697 View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'origin/feature/grid_tiles' into feature…

    …/grid_tiles
    milos.colic committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    19773fe View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. Add BNG examples.

    Add back the UUIDs.
    Change GDAL configs.
    Add ST_BufferCapStyle.
    milos.colic committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    27e5870 View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'origin/feature/grid_tiles' into feature…

    …/grid_tiles
    
    # Conflicts:
    #	src/main/scala/com/databricks/labs/mosaic/core/raster/api/RasterAPI.scala
    #	src/main/scala/com/databricks/labs/mosaic/core/raster/operator/gdal/GDALBuildVRT.scala
    milos.colic committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    b9b1856 View commit details
    Browse the repository at this point in the history
  3. Fix merge conflicts.

    milos.colic committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    b7f856d View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2023

  1. Re-enable coverage.

    milos.colic committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    742045a View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2023

  1. Fix PR comments.

    Remove base traits for RasterAPI.
    Centralize around GDAL as the only RasterAPI.
    Propagate class name changes.
    Update the init script for testing to the new tarball scripts.
    Rename RST_Tile to RST_FromFile.
    Rename RST_MergeBands to RST_FromBands.
    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    5e1a47b View commit details
    Browse the repository at this point in the history
  2. Fix github test install of GDAL.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    5cb0a9e View commit details
    Browse the repository at this point in the history
  3. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    9f462eb View commit details
    Browse the repository at this point in the history
  4. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    b47a731 View commit details
    Browse the repository at this point in the history
  5. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    4747771 View commit details
    Browse the repository at this point in the history
  6. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    00063ce View commit details
    Browse the repository at this point in the history
  7. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    624dba6 View commit details
    Browse the repository at this point in the history
  8. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    58e6160 View commit details
    Browse the repository at this point in the history
  9. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    3a66513 View commit details
    Browse the repository at this point in the history
  10. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    7b3c084 View commit details
    Browse the repository at this point in the history
  11. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    1b0cb9d View commit details
    Browse the repository at this point in the history
  12. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    529c95f View commit details
    Browse the repository at this point in the history
  13. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    76b8044 View commit details
    Browse the repository at this point in the history
  14. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    366f430 View commit details
    Browse the repository at this point in the history
  15. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    b3141d5 View commit details
    Browse the repository at this point in the history
  16. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    a50399b View commit details
    Browse the repository at this point in the history
  17. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    093f005 View commit details
    Browse the repository at this point in the history
  18. Add GDAL jni path to LD_LIBRARY_PATH.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    9625eb0 View commit details
    Browse the repository at this point in the history
  19. Remove copying of shared objects.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    8b53c22 View commit details
    Browse the repository at this point in the history
  20. Set python to 3.9 in actions.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    c1a6e6b View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    c235b34 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    b7e0144 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    1c36512 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    9df305c View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    b7eaba5 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    3e7d421 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    e20c0d8 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    2355885 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    9c2d6e5 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    af15929 View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    525afd5 View commit details
    Browse the repository at this point in the history
  32. Remove scala side script installation.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    79213a1 View commit details
    Browse the repository at this point in the history
  33. Add _agg aliases for _aggregate functions.

    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    cdd9b0d View commit details
    Browse the repository at this point in the history
  34. Remove duplicate python bindings.

    Update GDALWarp to use starts with instead of split.
    Update pythong mosaic_context to not depend on raster API.
    Use -wo CUTLINE_ALL_TOUCHED=TRUE for clipping.
    milos.colic committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    ccf39bb View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. R/fix automation (#441)

    - Remove references to RasterAPI in the non-expression-based R methods
    - Fix and simplify the R package build automation, including updating failing tests. The R packages will now appear with our release artefacts 👍
    - Added caching and use of Posit public package manager in Github actions to avoid downloading Spark source every time and building R package dependencies from source
    - Skip coverage checks if being run _outside_ of the main build workflow
    sllynn authored Oct 26, 2023
    Configuration menu
    Copy the full SHA
    b7dedf7 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2023

  1. Add CombineAVG expressions.

    Add GDALCalc support.
    Change NDVI to use GDALCalc.
    Add code docs.
    Add support for Pixel Functions via VRTs.
    Update tests.
    Change all gdal rasters to be passed by name.
    milos.colic committed Oct 29, 2023
    Configuration menu
    Copy the full SHA
    3aabd8f View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'origin/feature/grid_tiles' into feature…

    …/grid_tiles
    milos.colic committed Oct 29, 2023
    Configuration menu
    Copy the full SHA
    74195ce View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2023

  1. Fix temp file data leak.

    milos.colic committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    aa47d9c View commit details
    Browse the repository at this point in the history
  2. Remove the wheel from the examples.

    milos.colic committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    d9c3b5c View commit details
    Browse the repository at this point in the history