Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue 191: Add at-least-once stream processing example #192

Open
wants to merge 82 commits into
base: dev
Choose a base branch
from
Open

Issue 191: Add at-least-once stream processing example #192

wants to merge 82 commits into from

Commits on Aug 21, 2020

  1. Issue 191: Add non-Flink stream processing example. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    85ea475 View commit details
    Browse the repository at this point in the history

  2. Issue 191: Moved inner classes to top level. Removed obsolete classes. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    666c86e View commit details
    Browse the repository at this point in the history

  3. Issue 191: Various updates. This PR is still incomplete. 

    - Removed deleteStream calls.
- No longer waiting for transactions to go into COMMIT state after commit() call.
- Now using Callable instead of Runnable.
- Improved shutdown cleanup.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    eded361 View commit details
    Browse the repository at this point in the history

  4. Issue 191: Processor is now stateless to avoid complications with man… 

    …aging state.

- Added additional startup scripts.
- Updated default parameters.
- Updated documentation.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    13c3931 View commit details
    Browse the repository at this point in the history

  5. Issue 191: Cleanup 

    - Moved SimpleDataFormat creation outside of loop.
- Added missing license headers.
- Updated various comments.
- Removed unneeded variable.
- Updated routing key generation.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    178957e View commit details
    Browse the repository at this point in the history

  6. Issue 191: Allow variable number of workers; simplify recovery 

    - Now allows the number of workers to be different from the number of workers in previous executions.
- Simplified the recovery process by terminating the entire process upon any exceptions during checkpointing.
- Changed formatting of routing key.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    90af788 View commit details
    Browse the repository at this point in the history

  7. Update to use EventStreamClientFactory. Removed ExactlyOnceMultithrea… 

    …dedProcessor.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    c2c5fe6 View commit details
    Browse the repository at this point in the history

  8. Add test structure 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    666ea50 View commit details
    Browse the repository at this point in the history

  9. Add AtLeastOnceProcessor 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    c76d165 View commit details
    Browse the repository at this point in the history

  10. Add https://github.com/pravega/pravega/blob/c7ac009970787df2633163644… 

    …691aa3d72fc1a4e/client/src/test/java/io/pravega/client/state/examples/MembershipSynchronizer.java

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    18b8fb4 View commit details
    Browse the repository at this point in the history

  11. Add ReaderGroupPruner (incomplete) 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    6905085 View commit details
    Browse the repository at this point in the history

  12. Add ReaderGroupPruner (incomplete) 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    adb61b8 View commit details
    Browse the repository at this point in the history

  13. Bug fixes in MembershipSynchronizer 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    6bb05b1 View commit details
    Browse the repository at this point in the history

  14. Document new classes. Add various parameters. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    9584ce0 View commit details
    Browse the repository at this point in the history

  15. Use new AppConfiguration class. Update license. Mark obsolete section… 

    …s in README.md.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    4f11177 View commit details
    Browse the repository at this point in the history

  16. Add writer to AtLeastOnceApp 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    7eac911 View commit details
    Browse the repository at this point in the history

  17. Add graceful shutdown 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    9daa883 View commit details
    Browse the repository at this point in the history

  18. Add logging 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    f17131b View commit details
    Browse the repository at this point in the history

  19. Fix typo in comment 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    d2f3b26 View commit details
    Browse the repository at this point in the history

  20. Create runAtLeastOnceApp.sh 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    39a9f92 View commit details
    Browse the repository at this point in the history

  21. Now using SampleEvent class with JSON serializer. InstanceId can now … 

    …be set as an application parameter.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    2af2fb7 View commit details
    Browse the repository at this point in the history

  22. Update README (incomplete) 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    c18bfe9 View commit details
    Browse the repository at this point in the history

  23. Update README (incomplete) 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    353b346 View commit details
    Browse the repository at this point in the history

  24. Remove unused test code. Revert changes to logback.xml. Add new scrip… 

    …ts to build.gradle.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    f31a483 View commit details
    Browse the repository at this point in the history

  25. Remove unused test class. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    753c7f6 View commit details
    Browse the repository at this point in the history

  26. EventGenerator logs when events are acked. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    faddc14 View commit details
    Browse the repository at this point in the history

  27. Minor logging and formatting changes. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    0d6ebd1 View commit details
    Browse the repository at this point in the history

  28. Move runAtLeastOnceApp.sh 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    a1f3a5f View commit details
    Browse the repository at this point in the history

  29. Documentation, logging, and formatting updates 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    82a074c View commit details
    Browse the repository at this point in the history

  30. Fix typo. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    13076f2 View commit details
    Browse the repository at this point in the history

  31. Fix /README.md. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    bf5a9b0 View commit details
    Browse the repository at this point in the history

  32. Add integration test (incomplete) 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    7836d27 View commit details
    Browse the repository at this point in the history

  33. Add integration test (incomplete) 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    5629ffc View commit details
    Browse the repository at this point in the history

  34. Add integration test (incomplete) 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    ca59a61 View commit details
    Browse the repository at this point in the history

  35. Basic graceful shutdown test working 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    36b18cd View commit details
    Browse the repository at this point in the history

  36. AtLeastOnceProcessor now uses suppliers. Added basic structure for fa… 

    …ult injection.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    b47ed44 View commit details
    Browse the repository at this point in the history

  37. Add pause and unpause to ReaderGroupPruner 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    ad26df4 View commit details
    Browse the repository at this point in the history

  38. Basic pause of worker working 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    f43a2cd View commit details
    Browse the repository at this point in the history

  39. Use TestContext 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    5f9aeb0 View commit details
    Browse the repository at this point in the history

  40. Now validating that stopped instances do not process events 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    f2bdf1a View commit details
    Browse the repository at this point in the history

  41. Add license headers 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    4601f08 View commit details
    Browse the repository at this point in the history

  42. Add unit tests for TestEventValidator 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    5f46551 View commit details
    Browse the repository at this point in the history

  43. Add testing documentation 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    da84a01 View commit details
    Browse the repository at this point in the history

  44. Add test forcing duplicates. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    d409f44 View commit details
    Browse the repository at this point in the history

  45. Add pause/unpause to AtLeastOnceProcessorInstrumented. Add failing te… 

    …sts handleExceptionDuringFlushTest and killAndRestart1of1ForcingDuplicatesTest.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    bb00ed4 View commit details
    Browse the repository at this point in the history

  46. Add different write modes. Split writeEventsAndValidate. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    52e5caa View commit details
    Browse the repository at this point in the history

  47. Now using EndToEndTestConfig with builder 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    b2b2873 View commit details
    Browse the repository at this point in the history

  48. Add timeout to failing tests 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    0fc8783 View commit details
    Browse the repository at this point in the history

  49. Created failing test handleExceptionDuringProcessTest 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    a429894 View commit details
    Browse the repository at this point in the history

  50. Bug fix: Fix handling of position when an exception occurs during pro… 

    …cessing or flushing

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    e34bb90 View commit details
    Browse the repository at this point in the history

  51. Fix killAndRestart1of1ForcingDuplicatesTest 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    265fd13 View commit details
    Browse the repository at this point in the history

  52. Updated documentation. Now using in-process Pravega cluster. 

    Removed unnecessary sleep from killAndRestart1of1ForcingDuplicatesTest.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    6aab6bb View commit details
    Browse the repository at this point in the history

  53. Update README. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    0439f09 View commit details
    Browse the repository at this point in the history

  54. Remove unneeded dependency JUnitParams. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    1b5d275 View commit details
    Browse the repository at this point in the history

  55. Add description to getStream1Name and getStream2Name 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    f4a0be8 View commit details
    Browse the repository at this point in the history

  56. Randomize execution period of ReaderGroupPruner 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    dc9dfcf View commit details
    Browse the repository at this point in the history

  57. Revert log level to ERROR 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    f329c6b View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2020

  1. Move inline class definition to top-level class SampleEventProcessor 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    c3eae91 View commit details
    Browse the repository at this point in the history

  2. Clean up EventDebugSink: Make methods private. Add final. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    c80ef0d View commit details
    Browse the repository at this point in the history

  3. Remove commented code 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    ca1d896 View commit details
    Browse the repository at this point in the history

  4. Replace some try-with-resource blocks with Lombok @cleanup 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    3a9978a View commit details
    Browse the repository at this point in the history

  5. Add final 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    5c56356 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2021

  1. Merge branch 'dev' into issue-191-streamprocessing

    Claudio Fahey authored Mar 8, 2021
    Configuration menu
    Copy the full SHA
    6b4f1bb View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2021

  1. Merge branch 'dev' into issue-191-streamprocessing

    @RaulGracia
    RaulGracia authored Mar 18, 2021
    Configuration menu
    Copy the full SHA
    6aebc88 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2021

  1. Move annotations to separate lines 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 19, 2021
    Configuration menu
    Copy the full SHA
    f8155cf View commit details
    Browse the repository at this point in the history

  2. Fix samples version number and improve feedback in runAtLastOnceApp.sh 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 19, 2021
    Configuration menu
    Copy the full SHA
    2ed765d View commit details
    Browse the repository at this point in the history

  3. Fix typo in streamprocessing/README 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 19, 2021
    Configuration menu
    Copy the full SHA
    913ef6e View commit details
    Browse the repository at this point in the history

  4. Fix formatting 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 19, 2021
    Configuration menu
    Copy the full SHA
    ec39792 View commit details
    Browse the repository at this point in the history

  5. Removed all commented and unused code from SetupUtils.java 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 19, 2021
    Configuration menu
    Copy the full SHA
    097fb8b View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2021

  1. Decrease heartbeat interval for kill5of6Test to speed up dead reader … 

    …detection.

Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    a06dbc6 View commit details
    Browse the repository at this point in the history

  2. Adding logging 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    5c09d55 View commit details
    Browse the repository at this point in the history

  3. Add retry to killAndRestart1of1ForcingDuplicatesTest 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    bd6029c View commit details
    Browse the repository at this point in the history

  4. Fix retry in killAndRestart1of1ForcingDuplicatesTest 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    9215290 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2021

  1. Merge branch 'dev' into issue-191-streamprocessing

    Claudio Fahey authored Mar 26, 2021
    Configuration menu
    Copy the full SHA
    33902d8 View commit details
    Browse the repository at this point in the history

  2. Fixed Scala version of Spark dependency in Hadoop samples. 

    Signed-off-by: Raúl Gracia <[email protected]>
    @RaulGracia
    RaulGracia committed Mar 26, 2021
    Configuration menu
    Copy the full SHA
    843d4c2 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2021

  1. Make methods private 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 27, 2021
    Configuration menu
    Copy the full SHA
    8c880cd View commit details
    Browse the repository at this point in the history

  2. Create private method createStream 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 27, 2021
    Configuration menu
    Copy the full SHA
    4dc1aae View commit details
    Browse the repository at this point in the history

  3. Make SampleEvent immutable. Replace @cleanup with try-with-resources. 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 27, 2021
    Configuration menu
    Copy the full SHA
    1ad8163 View commit details
    Browse the repository at this point in the history

  4. Update README 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 27, 2021
    Configuration menu
    Copy the full SHA
    afa5139 View commit details
    Browse the repository at this point in the history

  5. Add createStreams method to AtLeastOnceApp 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 27, 2021
    Configuration menu
    Copy the full SHA
    40675a4 View commit details
    Browse the repository at this point in the history

  6. Add createStreams private method to EventGenerator 

    Signed-off-by: Claudio Fahey <[email protected]>
    Claudio Fahey committed Mar 27, 2021
    Configuration menu
    Copy the full SHA
    f67ea43 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2021

  1. Merge branch 'dev' into issue-191-streamprocessing

    @RaulGracia
    RaulGracia authored Jul 27, 2021
    Configuration menu
    Copy the full SHA
    55a8607 View commit details
    Browse the repository at this point in the history