Add a workflow to run the committed benchmark on CI

benchmark/Makefile has always been local-only, so the numbers it produces
were never recorded anywhere. Wire it up to a manual workflow so a run can
be kicked off and its output kept in the job summary.

This does not gate anything: it reports absolute throughput for the
current ref, with Crow v1.3.1 alongside for reference. Absolute req/s is
only comparable against other runs on the same runner type, which is why
the ref, runner and load parameters are recorded next to the numbers.

Use benchmark-ab instead when the question is whether a specific change
made things faster; comparing absolute numbers across runs cannot answer
that.

Linux and macOS only. Windows needs benchmark/Makefile rewritten first,
since it relies on nc, & and kill.
This commit is contained in:
yhirose
2026-07-30 20:52:50 -04:00
parent 29ceccecd6
commit 60f285a301

69
.github/workflows/benchmark_run.yaml vendored Normal file
View File

@@ -0,0 +1,69 @@
name: benchmark-run
# Runs the committed benchmark (`just bench`) and records the numbers.
#
# This is a measurement, not a test: nothing here fails the build. Unlike
# benchmark-ab, which compares two refs inside one job, this just reports the
# absolute throughput of the current ref alongside Crow for reference.
#
# Absolute req/s is only meaningful against other runs on the same runner type,
# so compare like with like when reading the history.
#
# Non-SSL only. Windows is excluded: benchmark/Makefile depends on `nc`, `&`
# and `kill`, so it would need a PowerShell rewrite first.
on:
workflow_dispatch:
inputs:
duration:
description: "Load duration per server"
required: false
default: "5s"
connections:
description: "Concurrent connections"
required: false
default: "10"
crow:
description: "Also benchmark Crow v1.3.1 for reference"
type: boolean
required: false
default: true
permissions:
contents: read
jobs:
bench:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- name: checkout
uses: actions/checkout@v4
- name: install bombardier
run: go install github.com/codesenberg/bombardier@latest
- name: run benchmark
run: |
export PATH="$(go env GOPATH)/bin:$PATH"
if [ "${{ inputs.crow }}" = "true" ]; then TARGET=bench-all; else TARGET=bench; fi
make -C benchmark "$TARGET" \
BENCH="bombardier -c ${{ inputs.connections }} -d ${{ inputs.duration }} localhost:8080" \
2>&1 | tee /tmp/bench.txt
- name: record results
if: always()
run: |
{
echo "## Benchmark (${{ matrix.os }})"
echo ""
echo "- ref: \`${{ github.ref_name }}\` (${{ github.sha }})"
echo "- connections=${{ inputs.connections }} duration=${{ inputs.duration }}"
echo ""
echo '```'
cat /tmp/bench.txt
echo '```'
} >> "$GITHUB_STEP_SUMMARY"