Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IO: Improve BulkProcessor when running per-record operations #286

Merged
merged 2 commits into from
Oct 9, 2024

Conversation

amotl
Copy link
Member

@amotl amotl commented Oct 9, 2024

About

When batches are of size one, SQLAlchemy relays a single insert operation to CrateDB, not a bulk one. This patch handles that situation better, effectively checking rowcount for handling INSERT OK, 0 rows responses.

Background

products.json#L11 demonstrates a record that can't be inserted into CrateDB. The behavior is that CrateDB doesn't fail on it, but responds with rowcount: 0.

crash -c "drop table if exists t01;"
crash -c "create table t01 (data OBJECT(DYNAMIC));"

http "127.0.0.1:4200/_sql?error_trace=false" \
  stmt="INSERT into t01 (data) values (?)" \
  args:='[{"_id": "507d95d5719dbef170f15c22", "item": [{"amount": { "percent_of_service" : 0.06 }}, { "amount" : 2.25 }]}]' --print bB

crash -c "refresh table t01;"
crash -c "select * from t01;"

... by also checking `rowcount` for handling `INSERT OK, 0 rows`
responses.
@cla-bot cla-bot bot added the cla-signed label Oct 9, 2024
@amotl amotl marked this pull request as ready for review October 9, 2024 08:40
@amotl amotl merged commit c6e42b9 into main Oct 9, 2024
32 checks passed
@amotl amotl deleted the improve-bulk-processor branch October 9, 2024 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant