Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: multithreaded bulk import #136

Merged
merged 3 commits into from
Dec 19, 2024
Merged

feat: multithreaded bulk import #136

merged 3 commits into from
Dec 19, 2024

Conversation

sattvikc
Copy link
Collaborator

Summary of change

(A few sentences about this PR)

Related issues

  • Link to issue1 here
  • Link to issue1 here

Test Plan

(Write your test plan here. If you changed any code, please provide us with clear instructions on how you verified your
changes work. Bonus points for screenshots and videos!)

Documentation changes

(If relevant, please create a PR in our docs repo, or create a checklist here
highlighting the necessary changes)

Checklist for important updates

  • Changelog has been updated
  • pluginInterfaceSupported.json file has been updated (if needed)
  • Changes to the version if needed
    • In build.gradle
  • Had installed and ran the pre-commit hook
  • If there are new dependencies that have been added in build.gradle, please make sure to add them
    in implementationDependencies.json.
  • Issue this PR against the latest non released version branch.
    • To know which one it is, run find the latest released tag (git tag) in the format vX.Y.Z, and then find the
      latest branch (git branch --all) whose X.Y is greater than the latest released tag.
    • If no such branch exists, then create one from the latest released branch.
  • When adding new recipes, ensure that its performance is being measured in the OneMillionUsersTest

Remaining TODOs for this PR

  • Item1
  • Item2

tamassoltesz and others added 3 commits December 19, 2024 14:14
* feat: Add bulk import queries

* fix: PR changes

* fix: PR changes

* fix: removing restriction of connection pool size for bulk import

* feat: enable test run trigger from ci by hand

* fix: enable test run trigger from ci by hand

* Updated config.yml

* fix: reverting CI changes

* "fix: reverting CI config changes"

This reverts commit 4586b90.

* fix: fixing transaction rolled back issues with multithreaded bulk import

* fix: changelog

* fix: put back accidentally deleted comment

* feat: mysql implementation for bulk import

* fix: using right transaction isolation level for mysql in bulkimport

* fix: review fix, onemillion user test move

* fix: fixing imports

---------

Co-authored-by: Ankit Tiwari <[email protected]>
@sattvikc sattvikc self-assigned this Dec 19, 2024
@sattvikc sattvikc merged commit 9fa239d into 7.3 Dec 19, 2024
2 of 3 checks passed
@sattvikc sattvikc deleted the feat/bulk-import-base branch December 19, 2024 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants