|
| 1 | +--- |
| 2 | +title: "AI-Powered Flamegraph Interpreter in Grafana Pyroscope" |
| 3 | +description: "Explore how our AI-powered tool is revolutionizing flamegraph interpretation" |
| 4 | +sidebar_label: "AI-Powered Flamegraph Interpreter" |
| 5 | +image: "https://github.com/grafana/pyroscope/assets/23323466/441f89c8-8757-4c3d-b1b9-d8a1960c4c5d" |
| 6 | +slug: /ai-powered-flamegraph-interpreter |
| 7 | +date: "2023-12-04" |
| 8 | + |
| 9 | +authors: |
| 10 | + - name: Ryan Perry |
| 11 | + url: https://github.com/Rperry2174 |
| 12 | + image_url: https://avatars.githubusercontent.com/u/23323466?v=4 |
| 13 | + - name: ChatGPT |
| 14 | + title: Editor |
| 15 | + url: https://chat.openai.com/ |
| 16 | + image_url: https://chat.openai.com/favicon-32x32.png |
| 17 | +--- |
| 18 | + |
| 19 | +import {OpenAI} from '../src/components/Blog/OpenAI'; |
| 20 | +import {DistributorCpu} from '../src/components/Blog/DistributorCpu'; |
| 21 | +import {Flamegraph} from '../src/components/Blog/Flamegraph'; |
| 22 | + |
| 23 | +import {Response1, BaseResponse, BeginnerResponse, PoemResponse, CustomerSuccessResponse, SpanishResponse, ITDepartmentHumorResponse, HackerNewsThreadResponse, MichaelScottExplainsResponse} from '../src/components/Blog/OpenAI'; |
| 24 | + |
| 25 | + |
| 26 | + |
| 27 | +If you're of the opinion that AI isn't going to solve all the world's problems, you're probably right. And if you think it could, well, you might be onto something too. |
| 28 | + |
| 29 | +In the world of software engineering, particularly observability, where complexity often outpaces comprehension, **AI isn't just a fancy tool; it's becoming an inevitable necessity**. |
| 30 | + |
| 31 | +During the Grafana Hackathon, the Pyroscope team embraced this reality. We tackled a genuine challenge head-on with AI: making flamegraph analysis intuitive, even for those new to profiling. |
| 32 | + |
| 33 | +We've seen that AI excels at tasks like language interpretation. So, why not leverage it to demystify flamegraphs? This led us to our Hackathon mission: |
| 34 | + |
| 35 | +**Demonstrate how AI can transform the user experience in analyzing and interpreting flamegraphs.** |
| 36 | + |
| 37 | +We did this by giving humans and AI the same flamegraph and asking them to interpret it as shown below. We then compared the results and analyzed the differences. |
| 38 | + |
| 39 | +<video id="vid" style={{width: "100%"}} autoPlay loop muted playsinline src="https://s3.amazonaws.com/pyroscope.io/videos/ai5.mp4"/> |
| 40 | + |
| 41 | +## A Direct Challenge: How Well Can You Interpret This Flamegraph? |
| 42 | + |
| 43 | +Before diving into AI’s capabilities, let's set a baseline. Here’s a flamegraph for you to analyze. See if you can answer these key profiling questions: |
| 44 | + |
| 45 | +1. **Performance Bottleneck**: What's causing the slowdown? |
| 46 | +2. **Root Cause**: Why is it happening? |
| 47 | +3. **Recommended Fix**: How would you resolve it? |
| 48 | + |
| 49 | +<Flamegraph profileData={DistributorCpu}></Flamegraph> |
| 50 | + |
| 51 | +This is where many face challenges, particularly beginners. But understanding these flamegraphs is key to simplifying the code that powers them. |
| 52 | + |
| 53 | +## AI’s Flamegraph Interpretation: An Eye-Opening Comparison |
| 54 | + |
| 55 | +Now for the fun part: let's see how our AI interprets this same flamegraph. We use a prompt that is roughly equivalent to the questions above. |
| 56 | + |
| 57 | +### Click the button to see the AI's interpretation of the flamegraph below |
| 58 | + |
| 59 | +<OpenAI response={BaseResponse} pulsating /> |
| 60 | +<Flamegraph profileData={DistributorCpu}></Flamegraph> |
| 61 | + |
| 62 | +It was given a similar prompt to the questions above: |
| 63 | +```txt |
| 64 | +interpret this flamegraph for me and answer the following three questions: |
| 65 | +- **Performance Bottleneck**: What's slowing things down? |
| 66 | +- **Root Cause**: Why is this happening? |
| 67 | +- **Recommended Fix**: How can we resolve it? |
| 68 | +
|
| 69 | +[ ... specially compressed flamegraph data ] |
| 70 | +``` |
| 71 | + |
| 72 | +How does its analysis stack up against yours? Statistically it probably did either better or worse than you (obviously)... |
| 73 | + |
| 74 | +## Bots vs. Brains: Who's better at Flamegraph Interpretation? |
| 75 | + |
| 76 | +We didn't stop at theoreticals. We put AI to a real-world test, sending the same flamegraph to a diverse group of individuals, categorizing them by their expertise in flamegraph analysis, and comparing their responses to AI's. |
| 77 | + |
| 78 | +### Distribution of participants by skill level: |
| 79 | + |
| 80 | + |
| 81 | + |
| 82 | +### The Results Are In: AI is better than (most) humans at interpreting flamegraphs |
| 83 | + |
| 84 | + |
| 85 | + |
| 86 | +- Flamegraph Experts: **Score: 83% passed**. They demonstrated high accuracy and detailed understanding, quickly pinpointing issues and interpreting them correctly |
| 87 | + |
| 88 | +- Flamegraph Advanced: **Score: 70% passed**. Their responses varied, some were spot on while others often didn't dig far enough into the flamegraph to identify the root cause |
| 89 | + |
| 90 | +- Non-Technical Professionals: **Score: 23% passed**. This group most frequently took thie `idk` option on especially the root cause and recommended fix question -- some very entertaining guesses though! |
| 91 | + |
| 92 | +- **AI Interpreter**: Pass Score: 100% (10 iterations with same prompt). The AI consistently outperformed `beginners` and `advanced` users, providing accurate, albeit less detailed/nuanced, interpretations than the `experts`. |
| 93 | + |
| 94 | +**These initial results at least point towards a great opportunity in adding value to _most_ users by incorporating AI.** |
| 95 | + |
| 96 | +We will definitely be exploring this further via more formal testing and would love to hear your thoughts on this as well -- where do you see it fitting in best to your workflow? |
| 97 | + |
| 98 | +## AI in Pyroscope: A clear win for UX |
| 99 | + |
| 100 | +While we still have the rest of the week to tweak our project for the Hackathon, what we've learned so far is that **AI's strength lies not just in analysis but more importantly in "filling the gaps" by augmenting and enhancing user experience** no matter what level you're at. |
| 101 | + |
| 102 | +It only takes using a flamegraph successfully once to really move from the `beginner` to `advanced` category. The thing is, from a product standpoint our biggest challenge has been building a user-experience that can span from `beginner` to `expert` and still be useful for both. |
| 103 | + |
| 104 | +However, with just a little bit of prompt engineering we can use AI to bridge this gap and provide and **endless array of tailored experiences for each user**. |
| 105 | + |
| 106 | +## AI Tailored Responses for Diverse User Needs |
| 107 | + |
| 108 | +### Explain the flamegraph to a beginner with no flamegraph knowledge... |
| 109 | +<OpenAI response={BeginnerResponse}/> |
| 110 | + |
| 111 | +### Explain the flamegraph in the form of a HN thread... |
| 112 | +<OpenAI response={HackerNewsThreadResponse}/> |
| 113 | + |
| 114 | +### Explain the flamegraph in the form of IT department humor... |
| 115 | +<OpenAI response={ITDepartmentHumorResponse}/> |
| 116 | + |
| 117 | +### Explain the flamegraph in the form of a poem... |
| 118 | +<OpenAI response={PoemResponse}/> |
| 119 | + |
| 120 | +### Explain the flamegraph in the form of a Michael Scott quote... |
| 121 | +<OpenAI response={MichaelScottExplainsResponse}/> |
| 122 | + |
| 123 | +Whether you're a beginner, an expert, or just someone bored at work trying to find an entertaining way to do an otherwise boring task -- all it takes is a little prompt engineering to tune the experience to your liking. |
| 124 | + |
| 125 | +With just a few lines of code we can adjust these prompts (or allow users to adjust them) to guide everyone from `beginner` to `expert`. |
| 126 | + |
| 127 | +## Your Turn: Test AIs analysis of _your_ code! |
| 128 | + |
| 129 | +It's worth noting that, the flamegraph used for this post is a current representation of our distributor. As you can see, our distributors have bottlnecks in two notorious tasks that most companies are likely not doing perfectly either: [regular expressions](https://github.com/grafana/pyroscope/blob/0cf0bcaf4febe9c611ef3bf7782492735b1ae9ed/pkg/pprof/pprof.go#L1122-L1132) which is technically the bottlneck, but [compression/parsing](https://github.com/grafana/pyroscope/blob/731f434bfa33b626437195ee564f0f87b19e1e84/pkg/distributor/distributor.go#L245-L347) is also an acceptable answer for bottlneck (fixes comoing soon!). |
| 130 | + |
| 131 | +However, while AI was successful in analyzing _this particular_ flamegraph, there are probably 51,331,542 more cases where AI fails (or succeeds) spectacularly and **we'd love for you to find and share real examples with us!** |
| 132 | + |
| 133 | +### How to use our AI-powered flamegraph interpreter on your own flamegraphs |
| 134 | +#### Option 1: Upload pprof to Flamegraph.com |
| 135 | +Upload `pprof` file containing _one_ flamegraph to [flamegraph.com](http://flamegraph.com) and click the "ai explainer" button. |
| 136 | +You can get a pprof file from most easily from Go runtime, but Pyroscope exports pprof from _all_ languages via the export button. |
| 137 | + |
| 138 | +**Note: You do have to export `pprof` format specifically and upload to [flamegraph.com](http://flamegraph.com) separately; exporting directly to flamegraph.com via the `flamegraph.com` button will not (yet) work** |
| 139 | + |
| 140 | + |
| 141 | + |
| 142 | +#### Option 2: View flamegraph in Grafana Cloud: |
| 143 | +Send profiling data to [Grafana Cloud Profiles](https://grafana.com/auth/sign-up/create-user?pg=pyroscope-ai-blog) and look at the Cloud profiles app (you can sign up for a free account) and click the "ai explainer" button |
| 144 | + |
| 145 | + |
| 146 | + |
| 147 | +#### Option 3: Wait a bit... |
| 148 | +Wait until we release an open source version of this tool (pending positive feedback from the community) |
| 149 | + |
| 150 | +## All feedback / ideas welcome! |
| 151 | + |
| 152 | +Let us know the good, bad and ugly of your experience with AI-powered flamegraph analysis: |
| 153 | +- **On Twitter**: [@PyroscopeIO](https://twitter.com/PyroscopeIO) and let the world know how you're using AI in flamegraph analysis. [Placeholder for Twitter Handle] |
| 154 | +- **On Slack**: Join the conversation in our [community channel](https://grafana.slack.com/archives/C049PLMV8TB) |
| 155 | +- **On GitHub**: Feel free to add to [the discussion](https://github.com/grafana/pyroscope/discussions/2797) to share detailed feedback or suggest enhancements |
| 156 | + |
| 157 | +## Appendix |
| 158 | + |
| 159 | +In case you were wondering how we clasified peopel into groups we had them self categorize on the following scale: |
| 160 | +1. **Flamegraph Experts**: Comprising experienced software engineers and DevOps engineers who have used flamegraphs before |
| 161 | +2. **Flamegraph Advanced**: Junior developers, technical support staff, frontend engineers, or other engineers new to Pyroscope or continuous profiling tools |
| 162 | +3. **Non-Technical Professionals**: From sales, marketing, HR, and project management, know their way around observability tooling but seldom if ever directly involved in coding or debugging |
0 commit comments