Add log processor for global db-cache invalidation

[porunov]

TODO:

• Fix buildCacheInvalidationChangeProcessor to be able to invalidate updated vertices which are already removed by the time processor is started
• Add parameter to JanusGraph which will allow automatically starting this log processor
• Cover added logic with tests
• Document global db-cache invalidation logic (like: lag time between transaction is committed and cache entry is evicted, log configuration for db-cache invalidation log processor, transactions usage for it to work)

Fixes #3155

Signed-off-by: Oleksandr Porunov [email protected]

---

Thank you for contributing to JanusGraph!

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

• Is there an issue associated with this PR? Is it referenced in the commit message?
• Does your PR body contain #xyz where xyz is the issue number you are trying to resolve?
• Has your PR been rebased against the latest commit within the target branch (typically master)?
• Is your initial contribution a single, squashed commit?

For code changes:

• Have you written and/or updated unit tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE.txt file, including the main LICENSE.txt file in the root of this repository?
• If applicable, have you updated the NOTICE.txt file, including the main NOTICE.txt file found in the root of this repository?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered?

[jekramer]

Hi @porunov

I've been referencing this and have some more questions. For cache invalidation purposes, should it matter whether an IndexUpdate is new or removed? I've noticed that getIndexUpdatesNoConstraints for the properties pulled from changeState.getRelations throws an AssertionError, because their ElementLifeCycle is "Loaded" instead of New or Removed.

I can avoid this by creating a new StandardVertexProperty with one of the accepted statuses, like you do with reverseProperty in this PR. From what I can tell, we only need the key from IndexUpdate for invalidation, and that is the same no matter what lifecycle is. Is there any reason I shouldn't do that? I wanted to check because you add both the property and reverseProperty to the internalRelations list here, and I didn't understand why.

[porunov]

Hi @porunov

I've been referencing this and have some more questions. For cache invalidation purposes, should it matter whether an IndexUpdate is new or removed? I've noticed that getIndexUpdatesNoConstraints for the properties pulled from changeState.getRelations throws an AssertionError, because their ElementLifeCycle is "Loaded" instead of New or Removed.

I can avoid this by creating a new StandardVertexProperty with one of the accepted statuses, like you do with reverseProperty in this PR. From what I can tell, we only need the key from IndexUpdate for invalidation, and that is the same no matter what lifecycle is. Is there any reason I shouldn't do that? I wanted to check because you add both the property and reverseProperty to the internalRelations list here, and I didn't understand why.

I actually didn't check that myself and added that code quickly. Initially I thought the lifecycle may matter, but now I think it doesn't actually matter for cache invalidation purposes. Thus, that code which I shared should probably be changed without reverseProperty, but I didn't investigate that much yet.

[jekramer]

Initially I thought the lifecycle may matter, but now I think it doesn't actually matter for cache invalidation purposes. Thus, that code which I shared should probably be changed without reverseProperty, but I didn't investigate that much yet.

ah okay, that addresses my concerns then, thank you! Actually reverseProperty is the one that works, it's the initial property that throws an error. So I'm glad you included it!