Skip to content

chore: Add benchmark workflow and refactor benchmark projects #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 19, 2025

Conversation

filzrev
Copy link
Contributor

@filzrev filzrev commented Mar 19, 2025

What's changed this PR

1. Add .github/workflows/benchmark.yml workflow

Add workflow execute all benchmarks on GitHub actions.
This workflow execute ALL benchmarks and write results to GitHub Actions Summaries.

Example beunchmark summaries and artifacts
https://github.com/filzrev/ZLinq/actions/runs/13944270499

**How to invoke benchmark workflow by using gh command`

gh workflow run benchmark.yml --repo filzrev/ZLinq --ref "chore-modify-benchmarks2"

Note: It' available after benchmark.yml workflow is merged to main branch.

2. Reorganize Benchmark project.

Reoraganize Program.cs file contents to separated files.

  1. Move IterateBenchmark to separated file.
  2. Benchmark execution settings is migrated to Properties/launchSettings.json file.
  3. Change benchmark runner logics to use BenchmarkSwitcher.

@neuecc
Copy link
Member

neuecc commented Mar 19, 2025

Thank you, this makes me very happy!

@neuecc neuecc merged commit a821bd6 into Cysharp:main Mar 19, 2025
1 check passed
@neuecc
Copy link
Member

neuecc commented Mar 19, 2025

Regarding standard tests, LinqAf is showing faster results.
Honestly, I believe that ZLinq should be faster based on the code path, so this result is surprising and I don't understand the cause yet.
I'd like to find the reason for this.

@filzrev
Copy link
Contributor Author

filzrev commented Mar 19, 2025

BenchmarkReport: https://github.com/Cysharp/ZLinq/actions/runs/13950755415

Benchmark.LinqPerfBenchmarks.Count00 benchmark results show different results compared before.
It can be reproduced on my local environment too.

Method Mean Error StdDev Allocated
Linq 54.20 ms 5.407 ms 0.296 ms 40 B
ZLinq 573.06 ms 42.462 ms 2.327 ms 400 B

@neuecc
Copy link
Member

neuecc commented Mar 19, 2025

The previous result was simply a bug.
And the reason it's extremely slow now is because Span optimization is missing, which I implemented locally and it became faster than SystemLinq.

However, I'm concerned about the extreme slowness in normal iteration (the allocation numbers are also puzzling... even when measuring with a profiler, I can't determine where they're coming from).
There's a possibility that nested generics might be having some effect.
It seems we need to try various approaches to check performance in nested cases.
dotnet/runtime#77192

@neuecc
Copy link
Member

neuecc commented Mar 19, 2025

There have been known issues with MemoryDiagnoser reporting incorrect values.
Your code might also have elements that don't work well with benchmarking.
dotnet/BenchmarkDotNet#1542

@neuecc
Copy link
Member

neuecc commented Mar 19, 2025

It seems the constrained is the cause.
I will think of some countermeasures.
https://stackoverflow.com/questions/74117219/why-calls-with-nested-generic-type-has-worse-performance-in-c

@filzrev filzrev deleted the chore-modify-benchmarks2 branch March 21, 2025 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants