Improve cache for MetricLookup.linkable_elements_for_measure()
#1443
Conversation
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
| lambda: f"Getting valid linkable elements for measure '{measure_reference.element_name}' took: {time.time() - start_time:.2f}s" | ||
| ) | ||
| ) | ||
| # Don't cache the result when group-by-metrics are selected as there can be many of them and may significantly |
There was a problem hiding this comment.
If we remove the cache for group by metrics, doesn't that mean this will be super slow when we try to fetch group by metric options?
There was a problem hiding this comment.
I had some concerns about memory usage, so caching of group-by-metrics is handled later through a separate cache that's LRU and avoids GC issues.
| measure_reference=measure_reference, | ||
| element_filter=element_filter, | ||
| ) | ||
| logger.debug( |
There was a problem hiding this comment.
Curious why you removed this log? Seems helpful to see if there was improvement on performance for this method. Also seems like it would be a good one to make INFO instead of DEBUG so that we can see how it performs for customers with larger manifests.
There was a problem hiding this comment.
This method can potentially be called a lot during query parsing, so it's easier to assess the runtime through DD tooling. e.g. a query with ~100 metrics, ~100 group-by items (including ones in filters) leads to a lot of log lines. Could leave a debug line in there though.
There was a problem hiding this comment.
Added that debug line.
plypaul
force-pushed
the
p--short-term-perf--18
branch
from
7469801
to
9cc2d57
Compare
…linkable_elements_for_measure()`.
plypaul
force-pushed
the
p--short-term-perf--18
branch
from
9cc2d57
to
38e78d7
Compare
This PR changes the cache for `MetricLookup.linkable_elements_for_measure()` to allow caching all measures. Memory use remained reasonable and because it's bound to `MetricLookup`, it will be GCed when a given manifest is no longer used.
This PR changes the cache for
MetricLookup.linkable_elements_for_measure()to allow caching all measures. Memory use remained reasonable and because it's bound toMetricLookup, it will be GCed when a given manifest is no longer used.