No symbols in Node.JS application

[orgads]

Describe the bug
I'm running a Node.JS application (in a Docker container), and trying to profile it, but all I get are addresses.

I pass --perf-basic-prof and --interpreted-frames-native-stack to node.

Parca agent is running on the host.

To Reproduce
I don't know how to reproduce with a small app. The example in parca-demos works fine.

Expected behavior
Function names should appear.

Screenshots or Profiles (using https://pprof.me)

Logs
Not sure if it's related, but I get this frequently on the server side.

recovered stack trace:
 goroutine 191 [running]:
runtime/debug.Stack()
        runtime/debug/stack.go:24 +0x5e
github.com/parca-dev/parca/pkg/symbol/addr2line.(*DwarfLiner).PCToLines.func1()
        github.com/parca-dev/parca/pkg/symbol/addr2line/dwarf.go:107 +0x45
panic({0x3817a20?, 0xc00f8f1650?})
        runtime/panic.go:914 +0x21f
github.com/parca-dev/parca/pkg/symbol/elfutils.getFunctionName(...)
        github.com/parca-dev/parca/pkg/symbol/elfutils/debuginfofile.go:249
github.com/parca-dev/parca/pkg/symbol/elfutils.(*debugInfoFile).SourceLines(0xc01c1d7e60, 0x1c740)
        github.com/parca-dev/parca/pkg/symbol/elfutils/debuginfofile.go:121 +0x799
github.com/parca-dev/parca/pkg/symbol/addr2line.(*DwarfLiner).PCToLines(0x3752680?, 0xc001a425d0?)
        github.com/parca-dev/parca/pkg/symbol/addr2line/dwarf.go:112 +0x6c
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).pcToLines(0xc001775930, {0x574dfd0, 0xc01c1c4740}, {0xc039f17d70, 0x28}, 0x0?)
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:574 +0x53
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).symbolizeLocationsForMapping(0xc001775930, {0x574d6c0, 0xc000dd93e0}, 0xc01088fe80, {0xc010894d20, 0x4, 0xc01088f080?})
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:517 +0x1afb
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Symbolize(0xc001775930, {0x574d6c0, 0xc000dd93e0}, {0xc00cc3e000, 0x7312, 0x7?})
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:297 +0x971
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).runSymbolizationCycle(0xc001775930, {0x574d6c0, 0xc000dd93e0})
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:221 +0x152
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run.func1()
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:190 +0xf1
github.com/parca-dev/parca/pkg/runutil.Repeat(0xc0012d6020?, 0xc000e08000, 0xc000929dd8)
        github.com/parca-dev/parca/pkg/runutil/runutil.go:38 +0x82
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run(0xc001775930, {0x574d6c0, 0xc000dd93e0}, 0x37add20?)
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:188 +0x73
github.com/parca-dev/parca/pkg/parca.Run.func5.1({0x574d6c0?, 0xc000dd93e0?})
        github.com/parca-dev/parca/pkg/parca/parca.go:508 +0x3d
runtime/pprof.Do({0x574d6f8?, 0xc0017739f0?}, {{0xc0012d6020?, 0x0?, 0x0?}}, 0xc000dfff40)
        runtime/pprof/runtime.go:51 +0x9d
github.com/parca-dev/parca/pkg/parca.Run.func5()
        github.com/parca-dev/parca/pkg/parca/parca.go:507 +0x1ef
github.com/oklog/run.(*Group).Run.func1({0xc0017aed00?, 0xc0017aed20?})
        github.com/oklog/[email protected]/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
        github.com/oklog/[email protected]/group.go:37 +0x67

Software (please complete the following information):

• Parca Agent Version: parca-agent, version 0.28.0 (commit: 1a09d0149d53f7e8c60287da7ada11b8ac29bf4d, date: 2023-12-04T14:09:50Z), arch: amd64
• Parca Server Version (if applicable): parca, version 0.20.0 (commit: d2b3947b351823a13da1b29757b4c8901e70e90c)

Workload (please complete the following information):

• Runtime (if applicable): Node.JS

Environment (please complete the following information):

• Linux Distribution: Debian Linux 12 (the image is node:18.18.2-alpine3.18)
• Linux Version: 6.1.0-13-cloud-amd64
• Arch: x86_64
• Container Runtime (if applicable): Docker 24.0.7

[javierhonduco]

Could this be related to #2810?

[orgads]

Maybe. Any progress with it?

[orgads]

With version 0.19.0 I don't get these errors, but still there are no symbols.

[orgads]

Anything I can provide to help debugging this?

[kakkoyun]

@orgads Sorry for not getting back to you sooner. Could you check this document for troubleshooting: https://www.polarsignals.com/docs/nodejs#troubleshooting

We recently discovered some issues regarding some nodejs bugs.

Let us know if it doesn't help.

[orgads]

I'm really not sure what I did different, but it works now. Thanks!

[brancz]

For what it's worth we've done many improvements in this area recently, so maybe it was just fixed with that.

[orgads]

I use the same versions 😅

[brancz]

Ah ok, that's odd then. But glad it works now!

[orgads]

Ok, I found out what my problem was. My application is running in Docker, so it writes the perfmap file in /tmp inside the container. I need to map it to the host for perf/parca to work correctly.

[javierhonduco]

This should not be a problem, we should be reading the tmp of any process in a different mount namespace... @kakkoyun for your attention

[orgads]

Actually I did my current tests with plain perf, so maybe parca did work without it.

Is there a flag for perf that enables reading it from the container?

[PabloPie]

I'm currently having the same issue with Parca running in K8s, no error logs though. NodeJS is generating the perfmap file in the /tmp directory of the container, Parca agent is running in the host, the whole thing deployed with the Parca Helm chart. Parca only shows addresses and no symbols though.

[kakkoyun]

We need to double-check the symbolization https://discord.com/channels/877547706334199818/877547706334199821/1217440802322776094

I'm re-opening the issue to keep track of things.

[PabloPie]

Providing more details on my previous comment:

Application versions

Nodejs: 20.11.1
Parca agent: v0.28
Parca server: v0.21

We are running nodejs in a container based on node-20:alpine, and using dumb-init. I followed the troubleshooting docs and the process is correctly creating the perfmap.

After enabling --perf-basic-prof-only-functions and --interpreted-frames-native-stack I have JIT symbols but no Node runtime symbols