Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Managed checkpoint 0.4.3 #568

Open
wants to merge 60 commits into
base: main
Choose a base branch
from
Open

Managed checkpoint 0.4.3 #568

wants to merge 60 commits into from

Commits on May 28, 2024

  1. added python bindings for rst_avg, rst_max, rst_median, rst_min, and … 

    …rst_pixelcount.
    @mjohns-databricks
    mjohns-databricks committed May 28, 2024
    Configuration menu
    Copy the full SHA
    e6cdbbe View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. Move to managed or manual local file cleanup.

    @mjohns-databricks
    mjohns-databricks committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    a03ab78 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2024

  1. CleanUpManager.scala now a long-running thread that manages tmp file … 

    …cleanup based on configured policies, e.g. file age of 30 minutes.
    @mjohns-databricks
    mjohns-databricks committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    7281f6d View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2024

  1. estimated memsize for non-file backed rasters and safer subdivide han… 

    …dling
    @mjohns-databricks
    mjohns-databricks committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    137dda7 View commit details
    Browse the repository at this point in the history

  2. zip install for docker testing, additional file cleanup centralization

    @mjohns-databricks
    mjohns-databricks committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    80a046e View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. PAM file cleanup in tests.

    @mjohns-databricks
    mjohns-databricks committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    09d4240 View commit details
    Browse the repository at this point in the history

  2. fix scalastyle errors

    @mjohns-databricks
    mjohns-databricks committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    7efab04 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. UDF Example for GDAL translate and warp; RST_Write function tests and… 

    … docs.
    @mjohns-databricks
    mjohns-databricks committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    cd939b6 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. raster_to_grid performance, gdal reader, test cleanup.

    @mjohns-databricks
    mjohns-databricks committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    d4a8f96 View commit details
    Browse the repository at this point in the history

  2. for github testing, adjusted to 30 minute cleanup.

    @mjohns-databricks
    mjohns-databricks committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    f9f0ad2 View commit details
    Browse the repository at this point in the history

  3. MosaicRasterGDAL is now able to hydrate dataset.

    @mjohns-databricks
    mjohns-databricks committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    0199047 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. removed tile wrapper functions to raster. improved driver detection.

    @mjohns-databricks
    mjohns-databricks committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    f366d1a View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. compiles only commit (fully expect tests to fail).

    @mjohns-databricks
    mjohns-databricks committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    b548e35 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. local tests passing (rc1)

    @mjohns-databricks
    mjohns-databricks committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    07972ab View commit details
    Browse the repository at this point in the history

  2. removed a local path that was used for debugging.

    @mjohns-databricks
    mjohns-databricks committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    b4a24b6 View commit details
    Browse the repository at this point in the history

  3. separating artefact uploads in github actions, commenting out r build… 

    … (for now).
    @mjohns-databricks
    mjohns-databricks committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    a11746c View commit details
    Browse the repository at this point in the history

  4. fix r tests, reverted accidental find/replace for 'path' to 'rawPath'.

    @mjohns-databricks
    mjohns-databricks committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    356b87e View commit details
    Browse the repository at this point in the history

  5. github build blocks cleanup manager thread, adding action to cleanup … 

    …tmp files.
    @mjohns-databricks
    mjohns-databricks committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    c972224 View commit details
    Browse the repository at this point in the history

  6. actions checkout must be first

    @mjohns-databricks
    mjohns-databricks committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    94897fe View commit details
    Browse the repository at this point in the history

  7. cleanup only after scala

    @mjohns-databricks
    mjohns-databricks committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    71c2597 View commit details
    Browse the repository at this point in the history

  8. add cleanup cmds to 'scala_build' action.

    @mjohns-databricks
    mjohns-databricks committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    fb42c59 View commit details
    Browse the repository at this point in the history

  9. add missing sudo to bash cmds.

    @mjohns-databricks
    mjohns-databricks committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    1c95eff View commit details
    Browse the repository at this point in the history

  10. fix r RST_Clip test (num params changed); also scalastyle cleanup.

    @mjohns-databricks
    mjohns-databricks committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    1ebae94 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. r test use "as_path" instead of "in_memory" read strategy.

    @mjohns-databricks
    mjohns-databricks committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    61328bd View commit details
    Browse the repository at this point in the history

  2. reverted previous adjustments to r tests, needed to handle metadata f… 

    …or "GDAL_MOSAIC_BAND_INDEX".
    @mjohns-databricks
    mjohns-databricks committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    16ee8a8 View commit details
    Browse the repository at this point in the history

  3. reverted wrong driverName for R test; sudo aware delete (for github b… 

    …uilds).
    @mjohns-databricks
    mjohns-databricks committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    a9187a9 View commit details
    Browse the repository at this point in the history

  4. test if user is 'root' to know if 'sudo' required for certain commands.

    @mjohns-databricks
    mjohns-databricks committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    e7b6708 View commit details
    Browse the repository at this point in the history

  5. additional file cleanup for current MosaicContext session tmp dir dur… 

    …ing testing `afterAll`.
    @mjohns-databricks
    mjohns-databricks committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    1882e1b View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. [WIP] post-PR - revert a find/replace issue, 'raster' -> 'tile' in st… 

    …rings and text, may break build.
    @mjohns-databricks
    mjohns-databricks committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    966fba1 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. [WIP] post-PR - additional revert a find/replace issue, 'raster' -> '… 

    …tile' in strings and text.
    @mjohns-databricks
    mjohns-databricks committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    dafae43 View commit details
    Browse the repository at this point in the history

  2. [WIP] post-PR - R bindings revert a find/replace issue, 'raster' -> '… 

    …tile' in strings and text.
    @mjohns-databricks
    mjohns-databricks committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    d56ef84 View commit details
    Browse the repository at this point in the history

  3. [WIP] post-PR - additional R bindings revert a find/replace issue, 'r… 

    …aster' -> 'tile' in strings and text.
    @mjohns-databricks
    mjohns-databricks committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    899191f View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. retile_on_read improvements, doc cleanup, added doc script

    @mjohns-databricks
    mjohns-databricks committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    8422968 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. interim work post PR - for subdatasets raster_to_grid needs more + rs… 

    …t_tessellate
    @mjohns-databricks
    mjohns-databricks committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    e7da172 View commit details
    Browse the repository at this point in the history

  2. cleanup age limit to 5 minutes for testing to overcome "No space left… 

    … on device" on remote build
    @mjohns-databricks
    mjohns-databricks committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    6b7ca39 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. srid defaults to WGS84 again, additional subdataset, interim testing … 

    …output cleanup.
    @mjohns-databricks
    mjohns-databricks committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    9c9f1d1 View commit details
    Browse the repository at this point in the history

  2. streamlined RasterAsGridReaderTest

    @mjohns-databricks
    mjohns-databricks committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    e8d3aea View commit details
    Browse the repository at this point in the history

  3. added "limitTessellate" option for testing in RasterAsGridReader

    @mjohns-databricks
    mjohns-databricks committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    c240d5d View commit details
    Browse the repository at this point in the history

  4. fixed scalastyle error and warnings

    @mjohns-databricks
    mjohns-databricks committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    608f5ef View commit details
    Browse the repository at this point in the history

  5. cleanup python tests

    @mjohns-databricks
    mjohns-databricks committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    f5bcae3 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. SubdivideOnRead setting subdataset prior to splitting

    @mjohns-databricks
    mjohns-databricks committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    782de41 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. table options for raster_to_grid

    @mjohns-databricks
    mjohns-databricks committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    d2dc159 View commit details
    Browse the repository at this point in the history

  2. raster_to_grid option key fix for tables

    @mjohns-databricks
    mjohns-databricks committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    218f0c7 View commit details
    Browse the repository at this point in the history

  3. raster_to_grid adjusted columns and finalize tile to fuse for gdal read.

    @mjohns-databricks
    mjohns-databricks committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    fa4b9cd View commit details
    Browse the repository at this point in the history

  4. raster_to_grid removed trailing "," on select expression (scalastyle … 

    …error)
    @mjohns-databricks
    mjohns-databricks committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    2e53b32 View commit details
    Browse the repository at this point in the history

  5. raster_to_grid added "overwriteSchema" to table write options; modifi… 

    …ed "finalTableFuse" handling.
    @mjohns-databricks
    mjohns-databricks committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    7341b5e View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. fixed checkpointing for "gdal" reader; "raster_to_grid" added "deltaF… 

    …ileRecords" and "stepTessellate" options.
    @mjohns-databricks
    mjohns-databricks committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    1774a09 View commit details
    Browse the repository at this point in the history

  2. ran black and isort on python.

    @mjohns-databricks
    mjohns-databricks committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    044e1b7 View commit details
    Browse the repository at this point in the history

  3. "raster_to_grid" geo-scientific handling adjustments.

    @mjohns-databricks
    mjohns-databricks committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    c5bac6d View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. "raster_to_grid" [iterim] "toTif" option.

    @mjohns-databricks
    mjohns-databricks committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    83268eb View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. "raster_to_grid" cleanup.

    @mjohns-databricks
    mjohns-databricks committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    de3d9e9 View commit details
    Browse the repository at this point in the history

  2. python tests adjusted to checkpoint = true as default.

    @mjohns-databricks
    mjohns-databricks committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    c4bf4d1 View commit details
    Browse the repository at this point in the history

  3. r tests adjusted to "as_path" strategy to match checkpoint = true as … 

    …default.
    @mjohns-databricks
    mjohns-databricks committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    f987f70 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Additional ExprConfig handling. Specified "in_memory" for R raster te… 

    …sting.
    @mjohns-databricks
    mjohns-databricks committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    da2a777 View commit details
    Browse the repository at this point in the history

  2. clean-up local tmp files immediately on finalize to fuse for StringTy… 

    …pe or after writing to BinaryType.
    @mjohns-databricks
    mjohns-databricks committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    9edbf11 View commit details
    Browse the repository at this point in the history

  3. added starts with "/tmp" check on local file delete.

    @mjohns-databricks
    mjohns-databricks committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    f4a5ffc View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2024

  1. added more checks prior to local file delete.

    @mjohns-databricks
    mjohns-databricks committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    fb0a557 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Reduced size of raster_to_grid tests.

    @mjohns-databricks
    mjohns-databricks committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    90968e0 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. tile finalizes to fuse; 'raster' field set only for BinaryType.

    @mjohns-databricks
    mjohns-databricks committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    8d9830c View commit details
    Browse the repository at this point in the history

  2. accidental keystroke in sparklyr test.

    @mjohns-databricks
    mjohns-databricks committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    ca96125 View commit details
    Browse the repository at this point in the history