Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

Unexpected heap memory and off-heap memory allocation #1208

Open
ziyangRen opened this issue Jan 12, 2023 · 0 comments
Open

Unexpected heap memory and off-heap memory allocation #1208

ziyangRen opened this issue Jan 12, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@ziyangRen
Copy link

Describe the bug
Running Spark tasks using gazelle should use more off-heap memory. However, when we run 2TB tpc-ds, we find that for most applications, the onheap memory usage is much greater than that in the off-heap memory, and some SQL statements do not even use off-heap memory, which troubles our parameter configuration.
notes:Traverse all SparkListenerTaskEnd events in the eventlog, obtain the maximum values of JVMHeapMemory and OffHeapExecutionMemory in all Task Executor Metrics.Used as the basis for determining the memory usage onheap and offheap.The test results are as follows(The memory unit is MB.):
image
image

@ziyangRen ziyangRen added the bug Something isn't working label Jan 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant