DRILL-8504: Add Schema Caching to Splunk Plugin #2929
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cgivre
changed the title
cgivre
added
enhancement
jnturton
reviewed
Aug 3, 2024
| <dependency> | ||
| <groupId>com.github.ben-manes.caffeine</groupId> | ||
| <artifactId>caffeine</artifactId> | ||
| <version>2.9.3</version> |
There was a problem hiding this comment.
Can we achieve the same thing using Guava's caching? The reason I ask is that we already have this insanely big dependency tree and Guava is already in it...
There was a problem hiding this comment.
But so is caffeine now that I look! So I guess we can ignore this suggestion.
There was a problem hiding this comment.
@jnturton Is that a +1? I somehow broke the versioning when I rebased on the current master, but I'll fix before merging.
There was a problem hiding this comment.
Sorry, I got pulled away before I could continue but will complete the review today.
cgivre
force-pushed
the
splunk_schema_cache
branch
from
af3d11b
to
f372ad6
Compare
cgivre
force-pushed
the
splunk_schema_cache
branch
from
74f6b90
to
0d8364b
Compare
|
@jnturton It looks like the GitHub CI is failing on the Hadoop 2 tests with Hive. |
cgivre
force-pushed
the
splunk_schema_cache
branch
from
2ff2983
to
fd2549a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DRILL-8504: Add Schema Caching to Splunk Plugin
Description
Whenever Drill executes a Splunk query, it must retrieve a list of indexes from Splunk. This step can add a considerable amount of time to the planning phase. This PR introduces a simple in-memory cache for the Splunk plugin which caches the list of indexes to avoid having to query Splunk repeatedly to obtain this information.
This PR also makes a few unrelated minor improvements:
Documentation
(Added to README)
For every query that you send to Splunk from Drill, Drill will have to pull schema information from Splunk. If you have a lot of indexes, this process can cause slow planning time. To improve planning time, you can configure Drill to cache the index names so that it does not need to make additional calls to Splunk.
There are two configuration parameters for the schema caching:
maxCacheSizeandcacheExpiration. The maxCacheSize defaults to 10k bytes and thecacheExpirationdefaults to 1024 minutes. To disable schema caching simply set thecacheExpirationparameter to a value less than zero.Testing
Ran all unit tests and tested manually.