-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A PR for reactive streams support #151
Commits on May 12, 2024
-
Add a proof-of-concept for "Observer-like" batch loading
**Note**: This commit, as-is, is not (yet) intended for merge. It is created to provide a proof-of-concept and gauge interest as polishing/testing this requires a non-trivial amount of effort. Motivation ========== The current DataLoader mechanism completes the corresponding `CompletableFuture` for a given key when the corresponding value is returned. However, DataLoader's `BatchLoader` assumes that the underlying batch function can only return all of its requested items at once (as an example, a SQL database query). However, the batch function may be a service that can return items progressively using a subscription-like architecture. Some examples include: - Project Reactor's [Subscriber](https://www.reactive-streams.org/reactive-streams-1.0.4-javadoc/org/reactivestreams/Subscriber.html). - gRPC's [StreamObserver](https://grpc.github.io/grpc-java/javadoc/io/grpc/stub/StreamObserver.html). - RX Java's [Flowable](https://reactivex.io/RxJava/3.x/javadoc/io/reactivex/rxjava3/core/Flowable.html). Streaming results in this fashion offers several advantages: - Certain values may be returned earlier than others (for example, the batch function may have cached values it can return early). - Memory load is lessened on the batch function (which may be an external service), as it does not need to keep hold of the retrieved values before it can send them out at once. - We are able to save the need to stream individual error values by providing an `onError` function to terminate the stream early. Proposal ======== We provide two new `BatchLoader`s and support for them in `java-dataloader`: - `ObserverBatchLoader`, with a load function that accepts: - a list of keys. - a `BatchObserver` intended as a delegate for publisher-like structures found in Project Reactor and Rx Java. This obviates the need to depend on external libraries. - `MappedObserverBatchLoader`, similar to `ObserverBatchLoader` but with an `onNext` that accepts a key _and_ value (to allow for early termination of streams without needing to process `null`s). - `*WithContext` variants for the above. The key value-add is that the implementation of `BatchObserver` (provided to the load functions) will immediately complete the queued future for a given key when `onNext` is called with a value. This means that if we have a batch function that can deliver values progressively, we can continue evaluating the query as the values arrive. As an arbitrary example, let's have a batch function that serves both the reporter and project fields on a Jira issue: ```graphql query { issue { project { issueTypes { ... } } reporter { ... } } } ``` If the batch function can return a `project` immediately but is delayed in when it can `reporter`, then our batch loader can return `project` and start evaluating the `issueTypes` immediately while we load the `reporter` in parallel. This would provide a more performant query evaluation. As mentioned above, this is not in a state to be merged - this is intended to gauge whether this is something the maintainers would be interested in owning. Should this be the case, the author is willing to test/polish this pull request so that it may be merged.
Configuration menu - View commit details
-
Copy full SHA for 6e30220 - Browse repository at this point
Copy the full SHA 6e30220View commit details
Commits on May 17, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 95540ff - Browse repository at this point
Copy the full SHA 95540ffView commit details -
Merge remote-tracking branch 'origin/master' into reactive-streams-br…
…anch # Conflicts: # build.gradle
Configuration menu - View commit details
-
Copy full SHA for 2cdba8a - Browse repository at this point
Copy the full SHA 2cdba8aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1d78255 - Browse repository at this point
Copy the full SHA 1d78255View commit details
Commits on May 18, 2024
-
Merge remote-tracking branch 'origin/master' into observer-batch-load…
…er-proof-of-concept * origin/master: Bump to Java 11
Configuration menu - View commit details
-
Copy full SHA for 2032e33 - Browse repository at this point
Copy the full SHA 2032e33View commit details -
Eliminate *BatchObserver in favour of Publisher
`reactive-streams` has become the de-facto standard for reactive frameworks; we thus use this as a base to allow seamless interop (rather than prompt an extra adapter layer).
Configuration menu - View commit details
-
Copy full SHA for 6b5a732 - Browse repository at this point
Copy the full SHA 6b5a732View commit details -
Use internal Assertions over Java's raw assert
This gives us more workable exceptions.
Configuration menu - View commit details
-
Copy full SHA for 68d7f54 - Browse repository at this point
Copy the full SHA 68d7f54View commit details -
Remove handling of Throwable passed into onNext
Passing an exception into `onNext` is not typically done in reactive-land - we would instead call `onError(Throwable)`. We can thus avoid handling this case.
Configuration menu - View commit details
-
Copy full SHA for a3132b7 - Browse repository at this point
Copy the full SHA a3132b7View commit details -
Expose
new*DataLoader
methods for *PublisherBatchLoaderThis is keeping in line with the other methods found in `DataLoaderFactory`.
Configuration menu - View commit details
-
Copy full SHA for fbeffae - Browse repository at this point
Copy the full SHA fbeffaeView commit details -
Copy/tweak original/ DataLoader tests for publisher equivalents
Given the large number of existing tests, we copy across this existing set for our publisher tests. What this really indicates is that we should invest in parameterised testing, but this is a bit painful in JUnit 4 - so we'll bump to JUnit 5 independently and parameterise when we have this available. This is important because re-using the existing test suite reveals a failure that we'll need to address.
Configuration menu - View commit details
-
Copy full SHA for b2a662d - Browse repository at this point
Copy the full SHA b2a662dView commit details -
Rename '*PublisherBatchLoader' to 'BatchPublisher'
This keeps in line with the original suggestion (because yours truly couldn't read, apparently). We also purge any remaining mention of 'observer', which was the first swing at this code.
Configuration menu - View commit details
-
Copy full SHA for 0d0b2f8 - Browse repository at this point
Copy the full SHA 0d0b2f8View commit details -
Ensure DataLoaderSubscriber is only called by one thread
Multiple threads may call `onNext` - we thus (lazily) chuck a `synchronized` to ensure correctness at the cost of speed. In future, we should examine how we should manage this concurrency better.
Configuration menu - View commit details
-
Copy full SHA for 14002f6 - Browse repository at this point
Copy the full SHA 14002f6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f303a8 - Browse repository at this point
Copy the full SHA 0f303a8View commit details
Commits on May 19, 2024
-
Configuration menu - View commit details
-
Copy full SHA for ce115fd - Browse repository at this point
Copy the full SHA ce115fdView commit details -
Merge pull request #148 from AlexandreCarlton/observer-batch-loader-p…
…roof-of-concept Add a proof-of-concept for "Observer-like" batch loading
Configuration menu - View commit details
-
Copy full SHA for 288be41 - Browse repository at this point
Copy the full SHA 288be41View commit details
Commits on May 20, 2024
-
Merge remote-tracking branch 'origin/master' into reactive-streams-br…
…anch # Conflicts: # build.gradle
Configuration menu - View commit details
-
Copy full SHA for e16fa65 - Browse repository at this point
Copy the full SHA e16fa65View commit details -
Configuration menu - View commit details
-
Copy full SHA for a93112a - Browse repository at this point
Copy the full SHA a93112aView commit details
Commits on May 21, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 74567fe - Browse repository at this point
Copy the full SHA 74567feView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4396624 - Browse repository at this point
Copy the full SHA 4396624View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8a64483 - Browse repository at this point
Copy the full SHA 8a64483View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3e8ac9c - Browse repository at this point
Copy the full SHA 3e8ac9cView commit details -
Inline BatchPublisher tests into DataLoaderTest
We now have the same coverage but with less code. Note that: - this is currently failing on 'duplicate keys when caching disabled'. - we still need to add tests that only make sense for the Publisher variants (e.g. half-completed keys).
Configuration menu - View commit details
-
Copy full SHA for eb2b40c - Browse repository at this point
Copy the full SHA eb2b40cView commit details -
Fix MappedBatchPublisher loaders to work without cache
If we did not cache the futures, then the MappedBatchPublisher DataLoader would not work as we were only completing the last future for a given key.
Configuration menu - View commit details
-
Copy full SHA for 651e561 - Browse repository at this point
Copy the full SHA 651e561View commit details
Commits on May 22, 2024
-
Merge pull request #155 from AlexandreCarlton/migrate-publisher-tests
Migrate publisher tests
Configuration menu - View commit details
-
Copy full SHA for 8295396 - Browse repository at this point
Copy the full SHA 8295396View commit details -
Merge remote-tracking branch 'origin/reactive-streams-branch' into re…
…active-streams-common-publisher-impl # Conflicts: # src/main/java/org/dataloader/DataLoaderHelper.java # src/test/java/org/dataloader/DataLoaderBatchPublisherTest.java # src/test/java/org/dataloader/DataLoaderMappedBatchPublisherTest.java
Configuration menu - View commit details
-
Copy full SHA for 86ec5c8 - Browse repository at this point
Copy the full SHA 86ec5c8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6d3c4eb - Browse repository at this point
Copy the full SHA 6d3c4ebView commit details -
Merge pull request #154 from graphql-java/reactive-streams-common-pub…
…lisher-impl Making the Subscribers use a common base class
Configuration menu - View commit details
-
Copy full SHA for 3fddb8b - Browse repository at this point
Copy the full SHA 3fddb8bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 034c68f - Browse repository at this point
Copy the full SHA 034c68fView commit details
Commits on May 23, 2024
-
Merge remote-tracking branch 'origin/master' into reactive-streams-br…
…anch-extra-tests-for-reactive
Configuration menu - View commit details
-
Copy full SHA for b09ac60 - Browse repository at this point
Copy the full SHA b09ac60View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5d826b8 - Browse repository at this point
Copy the full SHA 5d826b8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8b344db - Browse repository at this point
Copy the full SHA 8b344dbView commit details -
Merge pull request #158 from graphql-java/reactive-streams-branch-ext…
…ra-tests-for-reactive More tests for Publishers on reactive branch
Configuration menu - View commit details
-
Copy full SHA for e9bfc2b - Browse repository at this point
Copy the full SHA e9bfc2bView commit details
Commits on May 24, 2024
-
This moves the reactive code pout into its own package because DataLo…
…aderHelper is way too big
Configuration menu - View commit details
-
Copy full SHA for 91d3036 - Browse repository at this point
Copy the full SHA 91d3036View commit details -
Configuration menu - View commit details
-
Copy full SHA for e98621b - Browse repository at this point
Copy the full SHA e98621bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6523015 - Browse repository at this point
Copy the full SHA 6523015View commit details -
Configuration menu - View commit details
-
Copy full SHA for 170ccf8 - Browse repository at this point
Copy the full SHA 170ccf8View commit details -
Merge pull request #159 from graphql-java/reactive-streams-branch-mov…
…e-reactive-classes-out-of-dataloader-helper Reactive streams branch move reactive classes out of dataloader helper
Configuration menu - View commit details
-
Copy full SHA for 77fd0dd - Browse repository at this point
Copy the full SHA 77fd0ddView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4b9356e - Browse repository at this point
Copy the full SHA 4b9356eView commit details
Commits on May 26, 2024
-
Have MappedBatchPublisher take in a Set<K> keys
This is more symmetric with `MappedbatchLoader` and preserves efficiency; we do not need to emit a `Map.Entry` for duplicate keys (given the strong intention that this will be used to create a `Map`).
Configuration menu - View commit details
-
Copy full SHA for 3c3cc99 - Browse repository at this point
Copy the full SHA 3c3cc99View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2e82858 - Browse repository at this point
Copy the full SHA 2e82858View commit details -
Merge pull request #160 from AlexandreCarlton/add-documentation-for-p…
…ublishers Have MappedBatchPublisher take in a Set<K> keys (and add README sections)
Configuration menu - View commit details
-
Copy full SHA for c3e6ee5 - Browse repository at this point
Copy the full SHA c3e6ee5View commit details