Discoverability of perf event types

[adamtassier]

Is your feature request related to a problem? Please describe.
Hotspot was my first introduction to using perf. It took a me an (embarrassingly) long time to figure out there's other events than just 'cycles' and how good this is all integrated in hotspot.

Neither does the README contain any examples of using other events nor screenshots of the UI with multiple events enabled.

Describe the solution you'd like
An addition to the README's 'Using' section that demonstrates using multiple perf events, a list of some useful ones and a link to more information about perf events.

Maybe it could be integrated in the 'record' UI as well? I was thinking about adding some checkboxes next to the 'Event Type(s)' field containing the pre-defined events from perf list?

Describe alternatives you've considered
Reading the perf manual sooner :)

Additional context
UI mock:

[milianw]

Thanks for raising this topic, you are not wrong, but I do think this topic is way more complicated than you might assume ;-) I.e. we definitely don't want to do the naive sampling on multiple PMUs as that can lead to too much noise. Instead, leader sampling should be used, but that is somewhat flaky too in my testing (it sometimes breaks in kernel updates). Still, that's what should be done by default I think.

https://www.man7.org/linux/man-pages/man1/perf-list.1.html#LEADER_SAMPLING

Also hard coding an event in the way you suggest in your mockup will lead to failures in contexts where no PMU is available but the timer based fallback would still work. This is why not doing anything by default and letting the user write the more complex requests in the "event type(s)" edit is better imo. Maybe we should include some common examples as predefined values in the drop down instead?

anyhow, patches welcome!

[adamtassier]

Thanks for raising this topic, you are not wrong, but I do think this topic is way more complicated than you might assume ;-)

I feared as much :)

I.e. we definitely don't want to do the naive sampling on multiple PMUs as that can lead to too much noise.

Sorry I don't know if I'm understanding you 100% correctly, are you saying that:
a) Different events map to different PMU's and we shouldn't mix them?
b) Different events map to the same PMU but they can interfere?
c) Theres often too few PMU's to capture multiple events?
d) something else completely?

I was under the impression that at least a few hardware performance counters were available. Seems I was wrong as checking on my home PC cpuid tells me I have none... (ryzen 3600).

Also hard coding an event in the way you suggest in your mockup will lead to failures in contexts where no PMU is available...
Maybe we should include some common examples as predefined values in the drop down instead?

Certainly didn't want to suggest just hardcoding them. I was thinking about adding them to the dropdown as well. It's just that speaking from experience, I only discovered today that it was in fact a dropdown (after 6months of intensive usage of hotspot)

anyhow, patches welcome!

I'll try to create a PR with a small section added to the readme clarifying event usage and maybe add some defaults to the dropdown (and maybe hack a little on the UI in the meantime) .

[milianw]

Sorry I don't know if I'm understanding you 100% correctly, are you saying that:
a) Different events map to different PMU's and we shouldn't mix them?
b) Different events map to the same PMU but they can interfere?
c) Theres often too few PMU's to capture multiple events?
d) something else completely?

It's a combination of all this:

The PMU has a limited slot of programmable PMUs (I think often just four?), and if you measure too many events then perf will start muxing which obviously deteriorates the accuracy.

What's imo worse though is that a naive perf record -e cycles,instructions,branch-misses,cache-misses is going to configure the PMU to sample each of these four metrics separately, at a default frequency of roughly 1000Hz. The problem is when you later then want to correlate the metrics, the time at which the samples are taken will be very different. Furthermore, branche/cache misses are going to be encountered much less often compared to a quasi-steady metric like cycles and instructions (let's ignore frequency scaling etc), meaning you get much less samples for that! Finally, the data file will baloon in size, esp. when you use --call-graph dwarf, since every sample for each of the four metrics would get its own perf event with the stack copy for backtraces etc. pp.

If you instead use leader sampling via perf record -e "{cycles,instructions,branch-misses,cache-misses}:S" , then only cycles will drive the sampling, but everytime a sample is taken thereof, the other metrics are counted too and added to the single perf event - which is much more efficient and better allows inter-metric correlation.

I was under the impression that at least a few hardware performance counters were available.
Seems I was wrong as checking on my home PC cpuid tells me I have none... (ryzen 3600).

That sounds wrong to me, try something like this (I have a Ryzen 3900x here):

$ sudo dmesg | grep -i pmu
[    0.134280] Performance Events: Fam17h+ core perfctr, AMD PMU driver.
[    0.138248] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[   21.940428] RAPL PMU: API unit is 2^-32 Joules, 1 fixed counters, 163840 ms ovfl timer
[   21.940431] RAPL PMU: hw unit of domain package 2^-16 Joules

Intel CPUs give an exact number of available PMU counters, sadly not the case for AMD apparently. Also note how the NMI watchdog takes away one of them, so I probably only have 3 slots before muxing kicks in.