Skip to content

chore: Add micro benchmarks for ZLinq/SystemLinq perf comparison #149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 12, 2025

Conversation

filzrev
Copy link
Contributor

@filzrev filzrev commented May 3, 2025

This PR add ZLinq micro benchmarks for LINQ operations under Benchmark.ZLinq namespace.
And adding SystemLinqBenchmarkConfig that compare ZLinq/SystemLinq performance.

When SystemLinq benchmark config is used.
AsValueEnumerable() is replaced to stub implementation that returns IEnumerable<T>.
So it can compare ZLinq/SystemLinq performance with single benchmark method.

SystemLinq benchmark results example
See: https://github.com/filzrev/ZLinq/actions/runs/14808549467

Other changes

This PR contains following changes.
Sorry for the large diffs.

1..github/workflows/benchmark.yml

  • Add inputs to run benchmarks with custom config/filter
  • Increase timeout to 360 minutes. (Currently it takes about 120 minutes to run all benchmarks)

2. Benchmark.csproj

  • Move net10.0 to first element. (Because first TFM is used to generate benchmark lists)
  • Modify config that relating NuGetVersionsBenchmarkConfig/SystemLinqBenchmarkConfig
  • Add implicit namespace using settings.
  • Add custom MSBuild target to cleanup bin/artifacts directories when Clean operation executed.

3. BaseBenchmarkConfig.cs

Add TargetFrameworkFilter benchmark to exclude non compatible benchmarks by BenchmarkCategory.

4. NuGetVersionsBenchmarkConfig.cs

Modify to support ZLinq versions v1.0 or later with conditional build symbols(e.g. ZLINQ_1_4_0_OR_GREATER).

Note:
Currently multi NuGet versions benchmarks seems not works because there is problems on BenchmarkDotNet side.
It's expected to be resolved when next version is released. (ArtifactsPath is enabled by default)

5. SystemLinqBenchmarkConfig.cs
Add custom config to compare ZLinq/SystemLinq performance.
When SystemLinq benchmark job is used. ZLinq's AsValueEnumerable() is replaced to stub implementation that returns IEnumerable<T>

**6. TargetFrameworksBenchmarkConfig.cs
Add settings to suppress baseline related warnings that occurred.

7. BenchmarkEventProcessor.cs
Add logics to stop benchmark when benchmark projects failed to build.
And logs outputs.

8. ConsumerExtensions.*.cs

Add optimized version Consume methods that required for ZLinq micro benchmarks.

9. SummariesExtensions.*.cs
Refactor codes. and add following artifact outputs.

  • reports.md: Simple markdown reports.
  • reports.json: Contains results data that required to generate HTML report with client-side filter.

These files are intended to be used to publish HTML report to GitHub Pages
(Currently benchmark results page URL is changed every runs. So I want to publish result to fixed path)

10. TargetFrameworkFilter/ZLinqBenchmarkFilter.cs
Add custom benchmark filters.

These filters are required because #if preprocessor is not applied when running benchmark method.
Because BenchmarkDotNet listup benchmarks with .NET 9 (that defined at<TargetFrameworks> first element).

11. ZLinq/MicroBenchmarks/*
Add simple micro benchmark classes that use enumerable.

Almost benchmarks are inherited from EnumerableBenchmarkBase_WithBasicTypes<T>.
The following are exceptions.

  1. SumBenchmark: Use EnumerableBenchmarkBase<T> base type. Because INumber<T> based type required.
  2. Left/RightJoinBenchmark: Use EnumerableBenchmarkBase<T> base type. Because Join results generate too large data in some cases.
  3. Following benchmarks use Distinct because value must be unique value. So actual test items count is smaller.
    3.1. Single/SingleOrDefault
    3.2. ToDictionary
    3.3. ToFrozenDictionary
    3.4. ToImmutableDictionary

12. BenchmarkResultsReport
Add model class that represent benchmark results.
It'll be used to create HTML report from artifacts.

13. Program.cs

  • Add export logics for additional benchmark report.
  • Add --verbose extra parameter support to enable BenchmarkDotNet default progress logs.

14. ValueEnumerableExtensions.*.cs
Add extension methods that are used by SystemLinqBenchmarks

15. TestData/*
Add code that generate random test data for IEnumerable.

@filzrev filzrev force-pushed the chore-add-zlinq-systemlinq-benchmarks branch from e94bb64 to 3ff3c78 Compare May 3, 2025 08:26
@neuecc
Copy link
Member

neuecc commented May 12, 2025

Thank you! Sorry for the delayed check.

There doesn't seem to be any major issues, but there are some things that don't look good numerically.
The benchmarks are essential for improvement, so they're very helpful.

@neuecc neuecc merged commit 1b87d0a into Cysharp:main May 12, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants