104 lines
5.4 KiB
Markdown
104 lines
5.4 KiB
Markdown
# Performance benchmarks
|
|
|
|
Benchmarks measure time and Go allocations for repeatable operations. Run the
|
|
relevant cases before and after performance changes, alongside correctness tests.
|
|
Normal `go test` runs do not execute benchmarks; add `-bench` explicitly.
|
|
|
|
## Browser document benchmarks
|
|
|
|
Run from the repository root:
|
|
|
|
```sh
|
|
go test ./internal/review -run '^$' -bench '^BenchmarkDocument(Snapshot|HeadingNavigation|LineAssembly)$' -benchmem -count=10 > /tmp/cthulhu-bench-before.txt
|
|
```
|
|
|
|
Repeat after the change with the same command and
|
|
`/tmp/cthulhu-bench-after.txt` as the output file. Choose distinct filenames when
|
|
retaining earlier measurements. To check that the benchmarks execute quickly,
|
|
use `-benchtime=1x -count=1`; that is a smoke check, not a useful timing baseline.
|
|
|
|
The initial cases use 100, 1,000 and 10,000 content objects, plus one section per
|
|
twenty objects and a document root. Fixtures contain headings, links and Unicode
|
|
text, with two objects on each visual row. Each benchmark iteration performs:
|
|
|
|
- `BenchmarkDocumentSnapshot`: one document snapshot from an existing tree,
|
|
including traversal and derived metadata.
|
|
- `BenchmarkDocumentHeadingNavigation`: a forward/backward heading round trip
|
|
on an existing tree. The `adjacent` case visits two headings near the middle;
|
|
`wrap` moves between the last and first headings. Each iteration returns to
|
|
the same position, keeping the workload independent of the loop count.
|
|
Time and allocations are per round trip (two moves), including snapshot
|
|
construction and navigation bookkeeping.
|
|
- `BenchmarkDocumentLineAssembly`: one line lookup in the middle of a document,
|
|
combining the two text objects on that row. This includes snapshot construction.
|
|
- `BenchmarkDocumentCaretNavigation`: a character forward/backward round trip
|
|
within the same text object. Reusable caret ranges are warmed before timing.
|
|
- `BenchmarkDocumentWorkerHeadingNavigation`: a heading forward/backward round
|
|
trip through the review worker, including focus synchronization, diagnostics
|
|
and placement dispatch, with an in-memory provider. It performs no browser IPC.
|
|
|
|
Include the additional cases with
|
|
`-bench '^BenchmarkDocument(Snapshot|HeadingNavigation|LineAssembly|CaretNavigation|WorkerHeadingNavigation)$'`.
|
|
Navigation shares an immutable indexed view; `BenchmarkDocumentSnapshot`
|
|
continues to measure the independently owned node copy returned by `Document`.
|
|
The first view construction is included in snapshot and line measurements and
|
|
amortized over subsequent iterations. A short run can therefore overstate the
|
|
steady allocation rate. See [document navigation validation](browser-navigation-performance.md)
|
|
for cache ownership, invalidation and real-browser checks.
|
|
|
|
Fixture construction and initial focus synchronization are outside the timed
|
|
loops. The benchmarks fail on errors, missing work or invalid result shapes.
|
|
They use owned tree data and do not start a browser, access the desktop reader,
|
|
query native AT-SPI objects, or produce speech or Braille output. They do not
|
|
measure browser capture, native caret placement, input queues or audible latency.
|
|
Expand the suite when a reproduced bottleneck needs a different workload, such
|
|
as nested documents, tables, long text or event bursts.
|
|
|
|
Existing benchmarks also cover speech text preparation and Tentacle commands:
|
|
|
|
```sh
|
|
go test ./internal/speech -run '^$' -bench '^BenchmarkNormalizeSpeakText$' -benchmem -count=10
|
|
go test ./internal/tentacle -run '^$' -bench '^BenchmarkCommandRoundTrip$' -benchmem -count=10
|
|
```
|
|
|
|
The Tentacle benchmark starts disposable test peers and measures one completed
|
|
command at a time across 1, 8 or 16 processes; it does not measure concurrent
|
|
command saturation.
|
|
|
|
## Comparing changes
|
|
|
|
Record the revision, uncommitted changes, Go version, machine and benchmark
|
|
command with results. Compare repeated measurements on the same machine and
|
|
toolchain, under similar load and power settings. Do not run performance
|
|
comparisons with the race detector or coverage instrumentation enabled. Use
|
|
`benchstat` from `golang.org/x/perf/cmd/benchstat`, if installed, to compare files:
|
|
|
|
```sh
|
|
benchstat /tmp/cthulhu-bench-before.txt /tmp/cthulhu-bench-after.txt
|
|
```
|
|
|
|
`ns/op` is elapsed time per operation, `B/op` is Go bytes allocated per operation,
|
|
and `allocs/op` counts Go allocations. These are not total process memory or
|
|
native C allocation measurements. A benchmark's `PASS` means it executed without
|
|
failure; it does not mean its performance is acceptable or improved.
|
|
|
|
Investigate substantial, repeatable regressions before changing code. Profile
|
|
the affected operation when the cause is unclear. Preserve the full workload
|
|
and result checks when optimizing; do not improve scores by dropping required
|
|
content or bypassing validation. Initially keep timings informational rather
|
|
than adding timing gates to tests or push hooks. These commands can be run by
|
|
automation later; no scheduled job or CI timing gate is configured here.
|
|
|
|
For this benchmark-only change, run the existing affected correctness tests:
|
|
|
|
```sh
|
|
go test ./internal/review ./internal/atspi -count=1
|
|
```
|
|
|
|
Future browser performance fixes also need the relevant native browser harness
|
|
and human checks: navigate headings forward and backward through wrapping, read
|
|
lines with arrows, move through links and editable fields, return from browser
|
|
chrome, and verify fresh input interrupts old output. Check speech and physical
|
|
Braille where available. Automated timing does not replace those checks or the
|
|
human review required by `CONTRIBUTING.md`.
|