Use Intersect to Narrow Iterate Range and Reduce Memory Allocation

[gooohgb]

Description

In our index data, some hot keys are associated with a large number of UIDs, but the set filtered by our function is relatively small. During performance testing on this dataset, I noticed that pl.Uids consumes a significant amount of CPU time for slice memory allocation. Therefore, I propose an optimization in the Uids function to leverage the range provided by Intersect to reduce the scope of temporary result sets, thereby minimizing memory allocation.

[harshil-goel]

Thanks a lot for the input. We have recently upgraded how the list works, and now we have an even better function that you could use instead. Right now there would be an issue if your uid range is too high, but still the numbers are too low. Now you can basically just call list.FindPosting(uid) to see if uid is present or not. But we shouldn't use it if uid intersection range is bigger than uids actually present. So we could even add a check using ApproxLen(). Let me know if you are willing to make the change, or should we?

[gooohgb]

Thanks for your suggestion! I'm glad to contribute to this project. I’d like to make this change myself.

[gooohgb]

Thanks a lot for the input. We have recently upgraded how the list works, and now we have an even better function that you could use instead. Right now there would be an issue if your uid range is too high, but still the numbers are too low. Now you can basically just call list.FindPosting(uid) to see if uid is present or not. But we shouldn't use it if uid intersection range is bigger than uids actually present. So we could even add a check using ApproxLen(). Let me know if you are willing to make the change, or should we?

By the way, I’d like to ask if this PR (#9218) is likely to be merged. The optimization of the eq execution plan seems to significantly improve performance.

[harshil-goel]

Yeah that PR is scheduled to be merged. We are still evaluating and reviewing that PR.