Feature: purely in-memory (non-database) DataManager implementations for various entity types #3531
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dbmalkovsky
reviewed
Nov 23, 2022
...-inmemory-data/src/main/java/org/flowable/engine/data/inmemory/AbstractEntityComparator.java
Outdated
Show resolved
Hide resolved
ikaakkola
force-pushed
the
feature/memory-data-managers
branch
from
d5a32d6
to
93d96fa
Compare
dbmalkovsky
reviewed
Nov 27, 2022
...rc/test/java/org/flowable/engine/data/inmemory/execution/MemoryExecutionDataManagerTest.java
Outdated
Show resolved
Hide resolved
dbmalkovsky
reviewed
Nov 27, 2022
.../src/test/java/org/flowable/engine/data/inmemory/activity/MemoryActivityDataManagerTest.java
Outdated
Show resolved
Hide resolved
dbmalkovsky
reviewed
Nov 27, 2022
...s/flowable-inmemory-data/src/main/java/org/flowable/engine/data/inmemory/util/QueryUtil.java
Outdated
Show resolved
Hide resolved
ikaakkola
force-pushed
the
feature/memory-data-managers
branch
2 times, most recently
from
e04bdb6
to
f5e84c0
Compare
ikaakkola
changed the title
ikaakkola
force-pushed
the
feature/memory-data-managers
branch
from
42f9a8d
to
f5e84c0
Compare
|
I will rebase this to |
ikaakkola
force-pushed
the
feature/memory-data-managers
branch
3 times, most recently
from
202f96d
to
49c54c0
Compare
ikaakkola
changed the title
|
@tijsrademakers @dbmalkovsky any comments on if you are interested in this feature? Would it make more sense for you, if only the changes towards other parts of Flowable (required for this feature to function) would be included in a PR and I would maintain this module independently ? |
|
Hi @ikaakkola, thanks for your question. We will have a discussion about it in the coming days, and then update this PR with feedback on the feature. |
ikaakkola
force-pushed
the
feature/memory-data-managers
branch
from
fef4b99
to
14317cc
Compare
A new module providing high performance non-sql in memory data managers for various entity types. The implementation is based on Concurrent maps serving as the storage layer of the entities. This removes the chance of locking exceptions entirely and increases the performance of process execution significantly (2-5x faster compared to H2 or hsqldb even for simple processes), but the managers do not support (full) transactions. They implement a simple rollback / commit strategy where things like deadletter jobs work, and items are removed from memory on rollback, but many parts of true transactions, like rolling back updated objects, are not yet implemented. Any data manager that does not have a no-sql implementation will keep using the existing database based datamanagers. When running the non-sql in-memory datamanagers, it makes most sense to also run an in-memory database (eg. H2 or hsqldb) instead of a real database. The database is used for things like process deployments and definitions, byte arrays and other entities that either have limited performance improvements from memory data managers, or are complicated to implement due to references to table data on the database level. The in-memory data managers are for runtime processes only. It is intended that an executor using them has history set excplicitly to NONE - if history is enabled the benefits of the in-memory data managers vanish because each process still needs to do database operations to store history.
ikaakkola
force-pushed
the
feature/memory-data-managers
branch
from
14317cc
to
01cee69
Compare
fix: add findJobsByWorkerId impl
Please let me know your opinion on this PR. |
added 3 commits
December 20, 2023 11:07
Implement missing DataManager query options and implement new Flowable 7.1.0-SNAPSHOT specific features
ikaakkola
force-pushed
the
feature/memory-data-managers
branch
from
eef4a64
to
f71653a
Compare
|
I will opt to release and maintain this feature as a separate module. To help achieve that, I will create a few PRs that make it easier to override the datamanagers without resorting to Reflection. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(Fixes #3532)
This PR adds a new module ('flowable-inmemory-data') that provides in-memory, non-database DataManager implementations for activities, executions, variables, jobs, event subscriptions and identity links.
The in-memory DataManagers are based on concurrent maps and are fast compared to their Mybatis counterparts (even for simple processes 4-5x faster than H2 or HSQLDB in-memory database). They do not suffer from optimistic locking failures but provide no true rollbacks (see Limitations below)
The intended use of the high performance In-memory DataManagers is "lambda" style execution of BPMN processes where the individual processes are very short lived, usually synchronous and with little to no external integrations, but the amount of processes executed is high (we are running this implementation with 100+ process instances/second).
Limitations
The DataManager implementations are for entities of active processes only - there is no history implementations of in-memory DataManagers and as such, any BPMN engine running these datamanagers should always set historyLevel to NONE.
The DataManagers do not support Native SQL queries at all. Any Native SQL query against any of the implementations will simply throw an exception.
The implementation has partial support for transactions, where inserts and deletes will be rolled back, but updates are left as-is.
Implementation details
The implementation is done as a separate module because some of the current modules "cheat" and access data from other modules directly on the database level. This means that for example the Timer JobDataManager cannot be implemented as part of flowable-job-service , as it needs access to process definition classes and repository service, which the 'flowable-job-service' module does not have. Also having this as a separate module makes it clear that it is (somewhat) a custom feature that standard BPMN execution environments are probably not expected to ever use.
The entity types that this implementation supports are the ones that our testing has shown to get the most database operations (ByteArrayDataManager would be the one that is still left out of this feature for now) and as such provide the most benefit for performance. There is no technical reason not to support all the DataManagers, but implementing them is a rather slow process and things like deployments and definitions do not see that much database operations.
Check List: