Skip to content

Commit

Permalink
Add descriptive name to benchmarks, enable CI summaries (#1109)
Browse files Browse the repository at this point in the history
## Description of change

This change leverages our benchmark action's GitHub Action's job summary
integration to display a summary for throughput and latency benchmarks.
Before this change, we would need to check the webpage or the JSON
output to compare. For pull requests, only the JSON output would be
available. This change applies to both commits on `main` as well as pull
requests.

This should also address issues we had in the past where commit comments
did not describe which benchmark failed. See implementation:
https://github.com/benchmark-action/github-action-benchmark/blob/6bae118c112083251560ad8b3a1ff2e43aa23351/src/write.ts#L203-L239

More information on benchmark GH Action's integration:
https://github.com/benchmark-action/github-action-benchmark?tab=readme-ov-file#job-summary

Relevant issues: N/A

## Does this change impact existing behavior?

No impact to Mountpoint file system or client.

## Does this change need a changelog entry in any of the crates?

No, no customer-facing change.

---

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license and I agree to the terms of
the [Developer Certificate of Origin
(DCO)](https://developercertificate.org/).

---------

Signed-off-by: Daniel Carl Jones <[email protected]>
  • Loading branch information
dannycjones authored Nov 7, 2024
1 parent 50433e6 commit 011043e
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 0 deletions.
10 changes: 10 additions & 0 deletions .github/workflows/bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ jobs:
- name: Check benchmark results
uses: benchmark-action/github-action-benchmark@v1
with:
name: Throughput Benchmark (S3 Standard)
tool: 'customBiggerIsBetter'
output-file-path: results/output.json
# This is default value, just make it more explicit
Expand All @@ -86,9 +87,11 @@ jobs:
auto-push: ${{ inputs.publish }}
comment-on-alert: true
max-items-in-chart: 20
summary-always: true
- name: Check resource utilization
uses: benchmark-action/github-action-benchmark@v1
with:
name: Throughput Benchmark - Peak Memory Usage (S3 Standard)
tool: 'customSmallerIsBetter'
output-file-path: results/peak_mem_usage.json
benchmark-data-dir-path: dev/bench/peak_mem_usage
Expand All @@ -101,6 +104,7 @@ jobs:
comment-on-alert: false
max-items-in-chart: 20
skip-fetch-gh-pages: true
summary-always: true

latency-bench:
name: Benchmark (Latency)
Expand Down Expand Up @@ -148,6 +152,7 @@ jobs:
- name: Check benchmark results
uses: benchmark-action/github-action-benchmark@v1
with:
name: Latency Benchmark (S3 Standard)
tool: 'customSmallerIsBetter'
output-file-path: results/output.json
benchmark-data-dir-path: dev/latency_bench
Expand All @@ -159,6 +164,7 @@ jobs:
auto-push: ${{ inputs.publish }}
comment-on-alert: false
max-items-in-chart: 20
summary-always: true

cache-bench:
name: Benchmark (Cache)
Expand Down Expand Up @@ -208,6 +214,7 @@ jobs:
- name: Check benchmark results
uses: benchmark-action/github-action-benchmark@v1
with:
name: Cache Throughput Benchmark (S3 Standard)
tool: 'customBiggerIsBetter'
output-file-path: results/output.json
benchmark-data-dir-path: dev/cache_bench
Expand All @@ -219,9 +226,11 @@ jobs:
auto-push: ${{ inputs.publish }}
comment-on-alert: false
max-items-in-chart: 20
summary-always: true
- name: Check resource utilization
uses: benchmark-action/github-action-benchmark@v1
with:
name: Cache Throughput Benchmark - Peak Memory Usage (S3 Standard)
tool: 'customSmallerIsBetter'
output-file-path: results/peak_mem_usage.json
benchmark-data-dir-path: dev/cache_bench/peak_mem_usage
Expand All @@ -234,3 +243,4 @@ jobs:
comment-on-alert: false
max-items-in-chart: 20
skip-fetch-gh-pages: true
summary-always: true
6 changes: 6 additions & 0 deletions .github/workflows/bench_s3express.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ jobs:
- name: Check benchmark results
uses: benchmark-action/github-action-benchmark@v1
with:
name: Throughput Benchmark (S3 Express One Zone)
tool: 'customBiggerIsBetter'
output-file-path: results/output.json
benchmark-data-dir-path: dev/s3-express/bench
Expand All @@ -85,9 +86,11 @@ jobs:
auto-push: ${{ inputs.publish }}
comment-on-alert: true
max-items-in-chart: 20
summary-always: true
- name: Check resource utilization
uses: benchmark-action/github-action-benchmark@v1
with:
name: Throughput Benchmark - Peak Memory Usage (S3 Express One Zone)
tool: 'customSmallerIsBetter'
output-file-path: results/peak_mem_usage.json
benchmark-data-dir-path: dev/s3-express/bench/peak_mem_usage
Expand All @@ -100,6 +103,7 @@ jobs:
comment-on-alert: false
max-items-in-chart: 20
skip-fetch-gh-pages: true
summary-always: true

latency-bench:
name: Benchmark (Latency)
Expand Down Expand Up @@ -147,6 +151,7 @@ jobs:
- name: Check benchmark results
uses: benchmark-action/github-action-benchmark@v1
with:
name: Latency Benchmark (S3 Express One Zone)
tool: 'customSmallerIsBetter'
output-file-path: results/output.json
benchmark-data-dir-path: dev/s3-express/latency_bench
Expand All @@ -158,3 +163,4 @@ jobs:
auto-push: ${{ inputs.publish }}
comment-on-alert: false
max-items-in-chart: 20
summary-always: true

0 comments on commit 011043e

Please sign in to comment.