Managed checkpoint 0.4.3

.github/actions/scala_build/action.yml

@@ -38,7 +38,30 @@ runs:
         - name: Test and build the scala JAR - skip tests is false
           if: inputs.skip_tests == 'false'
           shell: bash
-          run: sudo mvn -q clean install
+          run: |
+            # [1] mvn tests
+            sudo mvn -q clean install 
+            # [2] pre-clean
+            # - make dirs (just in case)
+            # - get counts
+            sudo mkdir -p /tmp/mosaic_tmp
+            TMP_CNT=$( sudo find /tmp/mosaic_tmp | wc -l )
+            echo "files (pre-clean)? $TMP_CNT in '/tmp/mosaic_tmp'"
+            sudo mkdir -p /dbfs/checkpoint/mosaic_tmp
+            FUSE_CNT=$( sudo find /dbfs/checkpoint/mosaic_tmp | wc -l )
+            echo "files (pre-clean)? $FUSE_CNT in '/dbfs/checkpoint/mosaic_tmp'"
+            # [3] remove local tmp files
+            sudo echo "... cleaning '/tmp/mosaic_tmp'"
+            sudo rm -rf /tmp/mosaic_tmp/*
+            # [4] remove pseudo fuse files
+            sudo echo "... cleaning '/dbfs/checkpoint/mosaic_tmp'"
+            sudo rm -rf /dbfs/checkpoint/mosaic_tmp/*
+            # [5] post-clean
+            # - get counts
+            TMP_CNT=$( sudo find /tmp/mosaic_tmp | wc -l )
+            echo "files (post-clean)? $TMP_CNT in '/tmp/mosaic_tmp'"
+            FUSE_CNT=$( sudo find /dbfs/checkpoint/mosaic_tmp | wc -l )
+            echo "files (post-clean)? $FUSE_CNT in '/dbfs/checkpoint/mosaic_tmp'"
         - name: Build the scala JAR - skip tests is true
           if: inputs.skip_tests == 'true'
           shell: bash

.github/actions/upload_python_artefacts/action.yml

@@ -0,0 +1,10 @@
+name: upload mosaic python artefacts
+description: upload mosaic python artefacts
+runs:
+  using: "composite"
+  steps:
+    - name: upload python artefacts
+      uses: actions/upload-artifact@v3
+      with:
+        name: artefacts
+        path: staging/*.whl

.github/actions/upload_r_artefacts/action.yml

@@ -0,0 +1,10 @@
+name: upload mosaic r artefacts
+description: upload mosaic r artefacts
+runs:
+  using: "composite"
+  steps:
+    - name: upload r artefacts
+      uses: actions/upload-artifact@v3
+      with:
+        name: artefacts
+        path: staging/spark*.tar.gz

.github/actions/upload_scala_artefacts/action.yml

@@ -0,0 +1,10 @@
+name: upload mosaic scala artefacts
+description: upload mosaic scala artefacts
+runs:
+  using: "composite"
+  steps:
+    - name: upload scala artefacts
+      uses: actions/upload-artifact@v3
+      with:
+        name: artefacts
+        path: staging/*.jar

.github/workflows/build_main.yml

@@ -24,11 +24,18 @@ jobs:
     steps:
       - name: checkout code
         uses: actions/checkout@v2
+# ::: SCALA :::
       - name: build scala
         uses: ./.github/actions/scala_build
+      - name: upload scala artefacts
+        uses: ./.github/actions/upload_scala_artefacts
+# ::: PYTHON :::
       - name: build python
         uses: ./.github/actions/python_build
-      - name: build R
+      - name: upload python artefacts
+        uses: ./.github/actions/upload_python_artefacts
+# ::: R :::
+      - name: build r
         uses: ./.github/actions/r_build
-      - name: upload artefacts
-        uses: ./.github/actions/upload_artefacts
+      - name: upload r artefacts
+        uses: ./.github/actions/upload_r_artefacts

.gitignore

@@ -165,3 +165,36 @@ docker/.m2/
 /python/checkpoint/
 /python/checkpoint-new/
 /scripts/docker/docker-build/ubuntu-22-spark-3.4/Dockerfile
+/tmp/
+/scripts/docker/m2/aopalliance/
+/scripts/docker/m2/avalon-framework/
+/scripts/docker/m2/classworlds/
+/scripts/docker/m2/com/
+/scripts/docker/m2/commons-beanutils/
+/scripts/docker/m2/commons-chain/
+/scripts/docker/m2/commons-codec/
+/scripts/docker/m2/commons-collections/
+/scripts/docker/m2/commons-digester/
+/scripts/docker/m2/commons-io/
+/scripts/docker/m2/commons-lang/
+/scripts/docker/m2/commons-logging/
+/scripts/docker/m2/dev/
+/scripts/docker/m2/dom4j/
+/scripts/docker/m2/io/
+/scripts/docker/m2/jakarta/
+/scripts/docker/m2/javax/
+/scripts/docker/m2/jline/
+/scripts/docker/m2/junit/
+/scripts/docker/m2/log4j/
+/scripts/docker/m2/logkit/
+/scripts/docker/m2/net/
+/scripts/docker/m2/org/
+/scripts/docker/m2/oro/
+/scripts/docker/m2/pl/
+/scripts/docker/m2/xml-apis/
+/scripts/docker/m2/xmlpull/
+/checkpoint_table_knn/
+/src/test/resources/binary/zarr-air-unzip/
+/src/test/resources/binary/zarr-example-unzip/
+/src/test/resources/binary/zarr-warp-unzip/
+

CHANGELOG.md

@@ -1,11 +1,48 @@
 ## v0.4.3 [DBR 13.3 LTS]
+- In preparation for upcoming transition to dblabs spatial-utils (follow-on to dblabs mosaic but for DBR 14.3+, adjusted for 
+  various enhancements relating to our productized geospatial APIs):
+  - Significant streamlining of internal GDAL `Dataset` handling to include "hydrating" (loading the object) more lazily
+  - Dropped "Mosaic" from the serialized internal objects: `MosaicRasterTile`, `MosaicRasterGDAL`, and `MosaicRasterBandGDAL`
+  - All newly generated `RasterTile` objects can store the tile payload (`BinaryType` | `StringType` | GDAL `Dataset`) to
+    the configured fuse checkpoint dir (see below); RasterTiles generated in 0.4.1 and 0.4.2 can be loaded as-is 
+    (structure was different prior to 0.4.1)
+- Due to release of numpy 2.0 which has breaking changes with GDAL, numpy now limited to "<2.0,>=1.21.5" to match DBR minimum
 - Pyspark requirement removed from python setup.cfg as it is supplied by DBR
 - Python version limited to "<3.11,>=3.10" for DBR
-- iPython dependency limited to "<8.11,>=7.4.2" for both DBR and keplergl-jupyter 
-- Expanded support for fuse-based checkpointing (persisted raster storage), managed through:
-  - spark config 'spark.databricks.labs.mosaic.raster.use.checkpoint' in addition to 'spark.databricks.labs.mosaic.raster.checkpoint'.
-  - python: `mos.enable_gdal(spark, with_checkpoint_path=path)`.
-  - scala: `MosaicGDAL.enableGDALWithCheckpoint(spark, path)`.
+- iPython dependency limited to "<8.11,>=7.4.2" for both DBR and keplergl-jupyter
+- Expanded support for fuse-based checkpointing (persisted tile storage), managed through:
+  - spark config `spark.databricks.labs.mosaic.raster.checkpoint`
+  - python: `mos.enable_gdal(spark, with_checkpoint_dir=dir)` - additional functions include 
+    `gdal.update_checkpoint_dir`, and `gdal.reset_checkpoint`
+  - scala: `MosaicGDAL.enableGDALWithCheckpoint(spark, dir)` (similar bindings to python as well)
+- Local files generally are no longer eagerly deleted (disposed) but are controlled through 
+  `spark.databricks.labs.mosaic.manual.cleanup.mode`
+  and `spark.databricks.labs.mosaic.cleanup.age.limit.minutes` along with existing ability to specify the session 
+  local storage root dir with `spark.databricks.labs.mosaic.raster.tmp.prefix`
+- `RST_PixelCount` now supports optional 'countNoData' and 'countMask' (defaults are `false`, can now be `true`) to optionally get full 
+   pixel counts where mask is 0.0 and noData is what is configured in the tile
+- Added `RST_Write` to save a generated 'tile' to a specified directory (e.g. fuse) location using its GDAL driver and 
+  tile data / path; useful for formalizing the path when writing a Lakehouse table (allowing removal of interim
+  checkpointed data)
+- Improved `raster_to_grid` reader uses fuse checkpointing for interim steps as well as additional
+  performance improvements; added options to write the various phases to delta lake tables for much better performance
+- Built-in readers now support option "uriDeepCheck" to handle (mostly strip out) file path URI parts beyond "file:", "dbfs:", 
+  and various common GDAL formats, see `FormatLookup` for lists; also new config `spark.databricks.labs.mosaic.uri.deep.check`
+  allows global handling outside of readers, default is `false`.
+- `RST_ReTile`, `RST_ToOverlappingTiles`, `RST_Tessellate`, `RST_SeparateBands` now use checkpoint dir
+- `RST_Clip` GDAL Warp option `CUTLINE_ALL_TOUCHED` configurable (default is `true`, can now be `false`); also, setting 
+  SpatialReferenceSystem in the generated Shapefile Feature Layer (along with the WKB 'geometry' field as before)
+- `RST_MemSize` now returns sum of pixels * datatype bytes as a fallback if size cannot be gotten from a tile file 
+  (e.g. with in-memory only handling), -1 if dataset is null; handling split conditions where size < 1 
+- Python bindings added for `RST_Avg`, `RST_Max`, `RST_Median`, `RST_Min`, and `RST_PixelCount`; also added missing 'driver' 
+  param documented for `RST_FromContent`, missing docs added for `RST_SetSRID`, and standardized `RST_ToOverlappingTiles` 
+  (`RST_To_Overlapping_Tiles` deprecated)
+- `RST_WorldToRasterCoordY` now returns `y` value (was returning `x` value)
+- Doc UDF example added for arbitrary GDAL Warp and Transform ops
+- Quickstart Notebook updated to use MosaicAnalyzer [
+  [Python](https://github.com/databrickslabs/mosaic/blob/main/python/mosaic/models/analyzer/analyzer.py) |
+  [Scala](https://github.com/databrickslabs/mosaic/blob/main/src/main/scala/com/databricks/labs/mosaic/sql/MosaicAnalyzer.scala) ]
+  (was MosaicFrame in 0.3 series)
 
 ## v0.4.2 [DBR 13.3 LTS]
 - Geopandas now fixed to "<0.14.4,>=0.14" due to conflict with minimum numpy version in geopandas 0.14.4.
@@ -20,16 +57,16 @@
 ## v0.4.1 [DBR 13.3 LTS]
 - Fixed python bindings for MosaicAnalyzer functions.
 - Added tiller functions, ST_AsGeoJSONTile and ST_AsMVTTile, for creating GeoJSON and MVT tiles as aggregations of geometries.
-- Added filter and convolve functions for raster data.
+- Added filter and convolve functions for tile data.
 - Raster tile schema changed to be <tile:struct<index_id:bigint, tile:binary, metadata:map<string, string>>.
 - Raster tile metadata will contain driver, parentPath and path.
 - Raster tile metadata will contain warnings and errors in case of failures.
-- All raster functions ensure rasters are TILED and not STRIPED when appropriate.
+- All tile functions ensure rasters are TILED and not STRIPED when appropriate.
 - GDAL cache memory has been decreased to 512MB to reduce memory usage and competition with Spark.
-- Add RST_MakeTiles that allows for different raster creations.
+- Add RST_MakeTiles that allows for different tile creations.
 - Rasters can now be passed as file pointers using checkpoint location.
-- Added logic to handle zarr format for raster data.
-- Added RST_SeparateBands to separate bands from a raster for NetCDF and Zarr formats.
+- Added logic to handle zarr format for tile data.
+- Added RST_SeparateBands to separate bands from a tile for NetCDF and Zarr formats.
 
 ## v0.4.0 [DBR 13.3 LTS]
 - First release for DBR 13.3 LTS which is Ubuntu Jammy and Spark 3.4.1. Not backwards compatible, meaning it will not run on prior DBRs; requires either a Photon DBR or a ML Runtime (__Standard, non-Photon DBR no longer allowed__).
@@ -41,7 +78,7 @@
 
 ## v0.3.14 [DBR < 13]
 - Fixes for Warning and Error messages on mosaic_enable call.
-- Performance improvements for raster functions.
+- Performance improvements for tile functions.
 - Fix support for GDAL configuration via spark config (use 'spark.databricks.labs.mosaic.gdal.' prefix).
 
 ## v0.3.13
@@ -54,8 +91,8 @@
 
 ## v0.3.12
 - Make JTS default Geometry Provider
-- Add raster tile functions.
-- Expand the support for raster manipulation.
+- Add tile tile functions.
+- Expand the support for tile manipulation.
 - Add abstractions for running distributed gdal_translate, gdalwarp, gdalcalc, etc.
 - Add RST_BoundingBox, RST_Clip, RST_CombineAvg, RST_CombineAvgAgg, RST_FromBands, RST_FromFile, RST_GetNoData,
   RST_InitNoData, RST_Merge, RST_MergeAgg, RST_NDVI, RST_ReTile, RST_SetNoData, RST_Subdivide
@@ -87,10 +124,10 @@
 - Fixed automatic SQL registration with GDAL
 
 ## v0.3.9
-- Fixed k-ring interpolation on raster data read
+- Fixed k-ring interpolation on tile data read
 
 ## v0.3.8
-- Added readers for default GDAL raster drivers (https://gdal.org/drivers/raster/index.html)
+- Added readers for default GDAL tile drivers (https://gdal.org/drivers/raster/index.html)
   - TIFF
   - COG
   - NetCDF
@@ -108,7 +145,7 @@
 - Fixed pip release publish script
 
 ## v0.3.6
-- Added GDAL and 32 rst_* raster functions:
+- Added GDAL and 32 rst_* tile functions:
   - RST_BandMetaData
   - RST_GeoReference
   - RST_IsEmpty

CONTRIBUTING.md

@@ -84,7 +84,8 @@ The repository is structured as follows:
 ## Test & build Mosaic
 
 Given that DBR 13.3 is Ubuntu 22.04, we recommend using docker, 
-see [mosaic-docker.sh](https://github.com/databrickslabs/mosaic/blob/main/scripts/docker/mosaic-docker.sh).
+see [mosaic-docker.sh](https://github.com/databrickslabs/mosaic/blob/main/scripts/docker/mosaic-docker.sh) or 
+[mosaic-docker-java-tool-options.sh](https://github.com/databrickslabs/mosaic/blob/main/scripts/docker/mosaic-docker-java-tool-options.sh).
 
 ### Scala JAR
 
@@ -97,33 +98,41 @@ The packaged JAR should be available in `target/`.
 
 ### Python bindings
 
-The python bindings can be tested using [unittest](https://docs.python.org/3/library/unittest.html).
-- Build the scala project and copy to the packaged JAR to the `python/mosaic/lib/` directory.
-- Move to the `python/` directory and install the project and its dependencies:
-    `pip install . && pip install pyspark==<project_spark_version>`
-  (where 'project_spark_version' corresponds to the version of Spark 
-  used for the target Databricks Runtime, e.g. `3.2.1`.
-- Run the tests using `unittest`: `python -m unittest`
-
-The project wheel file can be built with [build](https://pypa-build.readthedocs.io/en/stable/).
-- Install the build requirements: `pip install build wheel`.
-- Build the wheel using `python -m build`.
-- Collect the .whl file from `python/dist/`
+1. Testing - You can run the tests (recommended within docker container) using
+   [unittest](https://docs.python.org/3/library/unittest.html), e.g. `python -m unittest`.
+2. Install the build requirements: `pip install build wheel`.
+3. [Option] Build with Script - If you are within docker container, to build the WHL file you can just run the following
+   from project root (mosaic) dir: `sh scripts/docker/python-local-build.sh` (it will package jar and build WHL).
+4. [Option] If doing build more manually (recommended within docker container using its init scripts):
+   - Build the scala project, e.g. `mvn package -DskipTests=true` if you have already tested successfully; that call. 
+     will copy the packaged JAR to the `python/mosaic/lib/` directory (or you can do so manually) and be sure to verify
+     no older JARs are lingering.
+   - Move to the `python/` directory and install the project and its dependencies:
+       `pip install . && pip install pyspark==<project_spark_version>`
+     (where 'project_spark_version' corresponds to the version of Spark 
+     used for the target Databricks Runtime, e.g. `3.4.1` for DBR 13.3 LTS. 
+   - The project wheel file can be built with [build](https://pypa-build.readthedocs.io/en/stable/), e.g. `python -m build`.
+5. Collect the .whl file from `python/dist/`:
+   - WHL contains the JAR.
+   - It is all that is needed for installing on a cluster (same for the deployed PyPI version).
 
 ### Documentation
 
 The documentation has been produced using [Sphinx](https://www.sphinx-doc.org/en/master/).
 
 To build the docs:
 - Install the pandoc library (follow the instructions for your platform [here](https://pandoc.org/installing.html)).
+- You can run `docs/mosaic-docs.sh` or do the manual steps that follow.
 - Install the python requirements from `docs/docs-requirements.txt`.
 - Build the HTML documentation by running `make html` from `docs/`.
   - For nbconvert you may have to symlink your jupyter share folder, 
     e.g. `sudo ln -s /opt/homebrew/share/jupyter /usr/local/share`. 
-- You can locally host the docs by running the `reload.py` script in the `docs/source/` directory.
+- You can locally host the docs by running the `reload.py` script in the `docs/source/` directory; 
+  recommend running `reload.py` through IntelliJ for auto-refresh features.
 
 ## Style
 
 Tools we use for code formatting and checking:
 - `scalafmt` and `scalastyle` in the main scala project.
-- `black` and `isort` for the python bindings.
+- `black` (python dir), e.g. `black mosaic` and `black test`.  
+- `isort` (python dir), e.g. `isort .` from within "mosaic" and "test" dirs.

R/README.md

@@ -9,7 +9,7 @@ Spark APIs to expose Mosaic's large scale geospatial data processing functions.
 
 ### Requirements
 The only requirement to start using Mosaic is a Databricks cluster running 
-Databricks Runtime 10.0 (or later) with the Mosaic Scala JAR attached and the
+Databricks Runtime (13.3 LTS for 0.4.x series) with the Mosaic Scala JAR attached and the
 relevant R mosaic package installed.
 
 It is recommended to install the Scala JAR suffixed "with-dependencies" for the