-
Notifications
You must be signed in to change notification settings - Fork 431
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Procfs improvements #4540
Merged
Merged
Procfs improvements #4540
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
geyslan
force-pushed
the
procfs-work
branch
2 times, most recently
from
January 21, 2025 19:34
86fdd6c
to
ce212f0
Compare
geyslan
force-pushed
the
procfs-work
branch
4 times, most recently
from
February 4, 2025 00:17
b417b05
to
fb1298c
Compare
Good ideia about replacing ReadFile :)
|
rscampos
reviewed
Feb 6, 2025
rscampos
reviewed
Feb 6, 2025
rscampos
reviewed
Feb 6, 2025
rscampos
approved these changes
Feb 6, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good work!! @geyslan, LGTM
Since the type of the converted primitive is already known, formatter helpers should be used to construct procfs file paths instead of relying on `fmt.Sprintf`. Using `fmt.Sprintf` is relatively costly due to its internal formatting logic, which is unnecessary for simple path construction.
`os.ReadFile` is not efficient for reading files in `/proc` because it attempts to determine the file size before reading. This step is unnecessary for `/proc` files, as they are virtual files with sizes that are often reported as unknown or `0`. `proc.ReadFile` is a new function designed specifically for reading files in `/proc`. It reads directly into a buffer and is more efficient than `os.ReadFile` because it allows tuning the initial buffer size to better suit the characteristics of `/proc` files. Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^BenchmarkReadFile$ github.com/aquasecurity/tracee/pkg/utils/proc -benchtime=10000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/utils/proc cpu: AMD Ryzen 9 7950X 16-Core Processor BenchmarkReadFile/ProcFSReadFile/Empty_File-32 10000000 3525 ns/op 408 B/op 4 allocs/op BenchmarkReadFile/OsReadFile/Empty_File-32 10000000 4070 ns/op 872 B/op 5 allocs/op BenchmarkReadFile/ProcFSReadFile/Small_File-32 10000000 3961 ns/op 408 B/op 4 allocs/op BenchmarkReadFile/OsReadFile/Small_File-32 10000000 4538 ns/op 872 B/op 5 allocs/op BenchmarkReadFile/ProcFSReadFile/Exact_Buffer_Size-32 10000000 4229 ns/op 920 B/op 5 allocs/op BenchmarkReadFile/OsReadFile/Exact_Buffer_Size-32 10000000 4523 ns/op 872 B/op 5 allocs/op BenchmarkReadFile/ProcFSReadFile_/proc/self/stat-32 10000000 4043 ns/op 408 B/op 4 allocs/op BenchmarkReadFile/OsReadFile_/proc/self/stat-32 10000000 4585 ns/op 872 B/op 5 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/utils/proc 334.751s
Remove the use of library functions to parse the stat file and instead parse it manually (on the fly) to reduce the number of allocations and improve performance. chore(proc): align parsing of stat field with the formats size This also align parsing sizes with the formats to avoid wrong parsing of the stat file. The internal fields are represented aligned with the actual kernel fields to avoid any confusion (signed/unsigned).
Propagate values based on its real size which in most cases is smaller than int (64-bit). This change reduces the memory footprint or at least the stress on the stack/heap.
Remove the use of library functions to parse the status file and instead parse it manually (on the fly) to reduce the number of allocations and improve performance.
Reduce ProcNS memory footprint by using the right member type sizes - namespace id is an uint32, since it is the inode number in struct ns_common. This change also improves the performance of GetAllProcNS(), GetProcNS() and GetMountNSFirstProcesses().
- NewProcStatFields() - NewThreadStatFields() - NewProcStatusFields() - NewThreadStatusFields()
Calling stat on /proc/<pid> would only increase the window for process termination between the stat call and the read of the file. This also replaces fmt.Sprintf with string concatenation and strconv.FormatInt for better performance.
/fast-forward |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ran these in
main
andprocfs-work
branches:dist/tracee --metrics --pyroscope --pprof -s tree=2609608 -e sched_process_exec,sched_process_fork,sched_process_exit --proctree source=both --proctree disable-procfs -o none
dist/tracee --metrics --pyroscope --pprof -s tree=2241901 -e sched_process_exec,sched_process_fork,sched_process_exit --proctree source=both -o none
Stressor details:
Details
8 threads with 1_000_000 ops each
running on:cpu: AMD Ryzen 9 7950X 16-Core Processor
MemTotal: 64923992 kB (64GB)
NOTE: You may notice that the
WITH procfs
Heap values have increased - it is a side effect of the CPU time improvement in the proc package. Since the process termination window has been reduced (from feed request to proc file reading), more child processes have been added to the Process still living in LRU.Close: #4547
1. Explain what the PR does
c5be62d perf(proctree): remove stat call
d474eeb chore: introduce builders with specific fields
c717863 perf(proc): improve ns
8fc7eb3 perf(proc): improve status file parsing
7471ae2 perf(proctree/proc): align fields to real size
764ba4a perf(proc): improve stat file parsing
2afe594 perf(proc): introduce ReadFile for /proc
d6818c1 perf(proc): use formatters for procfs file paths
c5be62d perf(proctree): remove stat call
d474eeb chore: introduce builders with specific fields
c717863 perf(proc): improve ns
8fc7eb3 perf(proc): improve status file parsing
7471ae2 perf(proctree/proc): align fields to real size
764ba4a perf(proc): improve stat file parsing
2afe594 perf(proc): introduce ReadFile for /proc
d6818c1 perf(proc): use formatters for procfs file paths
2. Explain how to test it
3. Other comments