Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for continuously starting load jobs as slots free up in the loader #1494

Merged
merged 104 commits into from
Aug 4, 2024
Merged

Add support for continuously starting load jobs as slots free up in the loader #1494

merged 104 commits into from
Aug 4, 2024

Commits on Jun 19, 2024

  1. add support for starting load jobs as slots free up

    @sh-rp
    sh-rp committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    78a5989 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Merge branch 'devel' into feat/continuous-load-jobs 

    # Conflicts:
#	dlt/load/load.py
#	tests/load/test_dummy_client.py
    @sh-rp
    sh-rp committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    b4d05c8 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. Merge branch 'devel' into feat/continuous-load-jobs

    @sh-rp
    sh-rp committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    8f1c9bc View commit details
    Browse the repository at this point in the history

  2. update loader class to devel changes

    @sh-rp
    sh-rp committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    c516fbc View commit details
    Browse the repository at this point in the history

  3. update failed w_d test

    @sh-rp
    sh-rp committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    da8c9e6 View commit details
    Browse the repository at this point in the history

  4. reduce sleep time for now

    @sh-rp
    sh-rp committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    b8ff71d View commit details
    Browse the repository at this point in the history

  5. add first implementation of futures on custom destination

    @sh-rp
    sh-rp committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    fa66386 View commit details
    Browse the repository at this point in the history

  6. rename start_file_load to get_load_job

    @sh-rp
    sh-rp committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    d59e4eb View commit details
    Browse the repository at this point in the history

  7. add first version of working follow up jobs for new loader setup

    @sh-rp
    sh-rp committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    3a8ec86 View commit details
    Browse the repository at this point in the history

  8. require jobclient in constructor for duckdb

    @sh-rp
    sh-rp committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    1768e17 View commit details
    Browse the repository at this point in the history

  9. fixes some dummy tests

    @sh-rp
    sh-rp committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    1707413 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. update all jobs to have the new run method

    @sh-rp
    sh-rp committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    189988c View commit details
    Browse the repository at this point in the history

  2. unify file_path argument in loadjobs

    @sh-rp
    sh-rp committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    a53a9b7 View commit details
    Browse the repository at this point in the history

  3. fixes some filepath related tests

    @sh-rp
    sh-rp committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    37108a6 View commit details
    Browse the repository at this point in the history

  4. renames job classes for more clarity and small updates

    @sh-rp
    sh-rp committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    aaa14fe View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. re-organize jobs a bit more 

    fix some tests
    @sh-rp
    sh-rp committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    78f5dbc View commit details
    Browse the repository at this point in the history

  2. fix destination parallelism

    @sh-rp
    sh-rp committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    a8d4a7a View commit details
    Browse the repository at this point in the history

  3. remove changed in config.toml

    @sh-rp
    sh-rp committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    2d1c3b0 View commit details
    Browse the repository at this point in the history

  4. replace emptyloadjob with finalized load job

    @sh-rp
    sh-rp committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    c93fea8 View commit details
    Browse the repository at this point in the history

  5. make sure files are only moved on main thread

    @sh-rp
    sh-rp committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    9c4ee47 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'devel' into feat/continuous-load-jobs

    @sh-rp
    sh-rp committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    331d74a View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. tmp

    @sh-rp
    sh-rp committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    2f6d3db View commit details
    Browse the repository at this point in the history

  2. wrap job instantiation in try catch block (still needs improvement)

    @sh-rp
    sh-rp committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    f61151a View commit details
    Browse the repository at this point in the history

  3. Merge branch 'devel' into feat/continuous-load-jobs 

    # Conflicts:
#	dlt/destinations/impl/lancedb/lancedb_client.py
#	dlt/destinations/impl/qdrant/qdrant_client.py
#	dlt/load/load.py
    @sh-rp
    sh-rp committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    145dbfb View commit details
    Browse the repository at this point in the history

  4. post devel merge fix

    @sh-rp
    sh-rp committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    3765b01 View commit details
    Browse the repository at this point in the history

  5. simplify followupjob creation 

    assumes followup jobs can always be created without error
    @sh-rp
    sh-rp committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    4d05dd5 View commit details
    Browse the repository at this point in the history

  6. refactor job restoring

    @sh-rp
    sh-rp committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    5ddb8ed View commit details
    Browse the repository at this point in the history

  7. Merge branch 'devel' into feat/continuous-load-jobs 

    # Conflicts:
#	dlt/destinations/impl/snowflake/snowflake.py
    @sh-rp
    sh-rp committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    14794e7 View commit details
    Browse the repository at this point in the history

  8. simplify common fields on loadjobs 

    mark load job vars private
    @sh-rp
    sh-rp committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    efb21b1 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. Merge branch 'devel' into feat/continuous-load-jobs

    @sh-rp
    sh-rp committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    75bbb59 View commit details
    Browse the repository at this point in the history

  2. completely separate followupjobs from regular loadjobs

    @sh-rp
    sh-rp committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    1f857a0 View commit details
    Browse the repository at this point in the history

  3. unify some more loadjob vars

    @sh-rp
    sh-rp committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    d6ad935 View commit details
    Browse the repository at this point in the history

  4. fix job client tests

    @sh-rp
    sh-rp committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    d6d2dc7 View commit details
    Browse the repository at this point in the history

  5. amend last commit

    @sh-rp
    sh-rp committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    1a5d2de View commit details
    Browse the repository at this point in the history

  6. fix handling of jobs in loader

    @sh-rp
    sh-rp committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    58ae445 View commit details
    Browse the repository at this point in the history

  7. fix a couple more tests

    @sh-rp
    sh-rp committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    802b168 View commit details
    Browse the repository at this point in the history

  8. fix deltalake load jobs

    @sh-rp
    sh-rp committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    18fbca2 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. fix pending exceptions code

    @sh-rp
    sh-rp committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    26d3ca1 View commit details
    Browse the repository at this point in the history

  2. fix partial load tests

    @sh-rp
    sh-rp committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    47f5298 View commit details
    Browse the repository at this point in the history

  3. fix custom destination and delta table tests

    @sh-rp
    sh-rp committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    0d97352 View commit details
    Browse the repository at this point in the history

  4. remove one unclear assertion for now

    @sh-rp
    sh-rp committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    6f7c940 View commit details
    Browse the repository at this point in the history

  5. fix clickhouse loadjob

    @sh-rp
    sh-rp committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    7980cd1 View commit details
    Browse the repository at this point in the history

  6. fix databricks loadjob

    @sh-rp
    sh-rp committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    20ad945 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. fix one weaviate and the qdrant local tests (hopefully :)

    @sh-rp
    sh-rp committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    dafd93c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'devel' into feat/continuous-load-jobs

    @sh-rp
    sh-rp committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    6cb31b5 View commit details
    Browse the repository at this point in the history

  3. fix one pipeline test

    @sh-rp
    sh-rp committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    acdac15 View commit details
    Browse the repository at this point in the history

  4. add a couple of loader test stubs

    @sh-rp
    sh-rp committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    b2f1ad6 View commit details
    Browse the repository at this point in the history

  5. update bigquery load jobs to new format

    @sh-rp
    sh-rp committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    9903b18 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'devel' into feat/continuous-load-jobs 

    # Conflicts:
#	dlt/destinations/impl/destination/destination.py
#	dlt/destinations/job_impl.py
    @sh-rp
    sh-rp committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    933e962 View commit details
    Browse the repository at this point in the history

  7. fix bigquery resume test

    @sh-rp
    sh-rp committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    12adb5c View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. add additional check to bigquery job resume test

    @sh-rp
    sh-rp committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    695c209 View commit details
    Browse the repository at this point in the history

  2. write to delta tables in single commit 

    revert all delta table tests to original
ensure delta tables are still executed on a thread
    @sh-rp
    sh-rp committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    59c09cc View commit details
    Browse the repository at this point in the history

  3. fix broken filesystem loading

    @sh-rp
    sh-rp committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    7feafab View commit details
    Browse the repository at this point in the history

  4. add some simple jobs tests

    @sh-rp
    sh-rp committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    4275308 View commit details
    Browse the repository at this point in the history

  5. fix recursion problem

    @sh-rp
    sh-rp committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    79a610a View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. remove a bit of unneded code

    @sh-rp
    sh-rp committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8dcac5b View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/feat/continuous-load-jobs' into … 

    …feat/continuous-load-jobs
    @sh-rp
    sh-rp committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    ffdb01b View commit details
    Browse the repository at this point in the history

  3. do not open remote connection when creating a load job

    @sh-rp
    sh-rp committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    5ba9124 View commit details
    Browse the repository at this point in the history

  4. fix weaviate

    @sh-rp
    sh-rp committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    b25b857 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'devel' into feat/continuous-load-jobs

    @sh-rp
    sh-rp committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    9179c99 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'devel' into feat/continuous-load-jobs 

    # Conflicts:
#	dlt/destinations/impl/snowflake/snowflake.py
    @sh-rp
    sh-rp committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    124316e View commit details
    Browse the repository at this point in the history

  7. post devel merge fixes

    @sh-rp
    sh-rp committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    3f79ddc View commit details
    Browse the repository at this point in the history

  8. only update load package info if jobs where finalized

    @sh-rp
    sh-rp committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    7109d33 View commit details
    Browse the repository at this point in the history

  9. fix two obviously wrong tests...

    @sh-rp
    sh-rp committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    3d43ddb View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. create client on thread for jobs

    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    06015e0 View commit details
    Browse the repository at this point in the history

  2. fix sql_client / job_client vars 

    improve performance on starting new jobs (tests pending)
    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    00eda96 View commit details
    Browse the repository at this point in the history

  3. add tests for available slots and update tests for getting filtering … 

    …new jobs
    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    928e070 View commit details
    Browse the repository at this point in the history

  4. clean up complete package condition

    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    1785641 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'devel' into feat/continuous-load-jobs 

    # Conflicts:
#	dlt/destinations/impl/clickhouse/clickhouse.py
#	tests/load/bigquery/test_bigquery_client.py
    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    dcb683f View commit details
    Browse the repository at this point in the history

  6. Merge branch 'devel' into feat/continuous-load-jobs 

    # Conflicts:
#	dlt/destinations/impl/clickhouse/clickhouse.py
#	tests/load/bigquery/test_bigquery_client.py
    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    187a5eb View commit details
    Browse the repository at this point in the history

  7. improve table-sequential job filtering

    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    258f5d4 View commit details
    Browse the repository at this point in the history

  8. fix resume job test

    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    607990c View commit details
    Browse the repository at this point in the history

  9. fix load job init exceptions tests

    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    ea25801 View commit details
    Browse the repository at this point in the history

  10. remove test stubs for tests that already exist

    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    828bf4c View commit details
    Browse the repository at this point in the history

  11. add some benchmark code to loader tests (in progress)

    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    f3ca312 View commit details
    Browse the repository at this point in the history

  12. amend loader benchmark test

    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    0e87a69 View commit details
    Browse the repository at this point in the history

  13. remove job_client from RunnableLoadJob initializer params

    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    9fb8c5c View commit details
    Browse the repository at this point in the history

  14. fix bg streaming insert

    @sh-rp
    sh-rp committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    b8f7420 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. fix bigquery streaming insert

    @sh-rp
    sh-rp committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    eb882d0 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. small renaming and logging changes

    @sh-rp
    sh-rp committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    f3161af View commit details
    Browse the repository at this point in the history

  2. remove delta job type in favor of using the reference jobs

    @sh-rp
    sh-rp committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    07c279a View commit details
    Browse the repository at this point in the history

  3. nicer logging when jobs pool is being drained

    @sh-rp
    sh-rp committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    26fef1b View commit details
    Browse the repository at this point in the history

  4. small comment change

    @sh-rp
    sh-rp committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    36b1997 View commit details
    Browse the repository at this point in the history

  5. test exception in followup job creation

    @sh-rp
    sh-rp committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    999ab9d View commit details
    Browse the repository at this point in the history

  6. add tests for followup jobs

    @sh-rp
    sh-rp committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    2d4c7d4 View commit details
    Browse the repository at this point in the history

  7. improve dummy tests for better followup job testing

    @sh-rp
    sh-rp committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    19a90ac View commit details
    Browse the repository at this point in the history

  8. Merge branch 'devel' into feat/continuous-load-jobs 

    # Conflicts:
#	dlt/destinations/impl/filesystem/filesystem.py
    @sh-rp
    sh-rp committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    b6e4fca View commit details
    Browse the repository at this point in the history

  9. fix linter

    @sh-rp
    sh-rp committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    1c73de1 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. Merge branch 'devel' into feat/continuous-load-jobs

    @sh-rp
    sh-rp committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    d35842c View commit details
    Browse the repository at this point in the history

  2. put sleep amount back to 1.0 while checking for completed load jobs

    @sh-rp
    sh-rp committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    90f820c View commit details
    Browse the repository at this point in the history

  3. create explicit exceptions for failed table chain jobs

    @sh-rp
    sh-rp committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    6ba32f8 View commit details
    Browse the repository at this point in the history

  4. make the large load package test faster

    @sh-rp
    sh-rp committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    9142a1b View commit details
    Browse the repository at this point in the history

  5. fix trace test

    @sh-rp
    sh-rp committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    9fc995e View commit details
    Browse the repository at this point in the history

  6. allow clients to prepare for job execution on thread and move query t… 

    …ag execution there.
    @sh-rp
    sh-rp committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    bf9f912 View commit details
    Browse the repository at this point in the history

  7. fix runnable job tests and linter

    @sh-rp
    sh-rp committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    5c07c07 View commit details
    Browse the repository at this point in the history

  8. fix linter again and remove wrong value from tests

    @sh-rp
    sh-rp committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    ce3e1c9 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. test

    @sh-rp
    sh-rp committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    7fe2f46 View commit details
    Browse the repository at this point in the history

  2. update detection of pending jobs, will probably break some tests

    @sh-rp
    sh-rp committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    7e569af View commit details
    Browse the repository at this point in the history

  3. fix two tests of pending packages

    @sh-rp
    sh-rp committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    960f309 View commit details
    Browse the repository at this point in the history

  4. fix test_remove_pending_packages test

    @sh-rp
    sh-rp committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    1cf2207 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Merge branch 'devel' into feat/continuous-load-jobs 

    # Conflicts:
#	dlt/destinations/impl/filesystem/filesystem.py
    @sh-rp
    sh-rp committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    5b6717c View commit details
    Browse the repository at this point in the history

  2. switch to docker compose subcommand

    @sh-rp
    sh-rp committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    3423ca7 View commit details
    Browse the repository at this point in the history

  3. fix compose deployments

    @sh-rp
    sh-rp committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    4b21365 View commit details
    Browse the repository at this point in the history

  4. fix test for arrow version in delta tables

    @sh-rp
    sh-rp committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    2c38f13 View commit details
    Browse the repository at this point in the history