cthulhu/docs/testing/benchmarks.md

Performance benchmarks

Benchmarks measure time and Go allocations for repeatable operations. Run the relevant cases before and after performance changes, alongside correctness tests. Normal go test runs do not execute benchmarks; add -bench explicitly.

Browser document benchmarks

Run from the repository root:

go test ./internal/review -run '^$' -bench '^BenchmarkDocument(Snapshot|HeadingNavigation|LineAssembly)$' -benchmem -count=10 > /tmp/cthulhu-bench-before.txt

Repeat after the change with the same command and /tmp/cthulhu-bench-after.txt as the output file. Choose distinct filenames when retaining earlier measurements. To check that the benchmarks execute quickly, use -benchtime=1x -count=1; that is a smoke check, not a useful timing baseline.

The initial cases use 100, 1,000 and 10,000 content objects, plus one section per twenty objects and a document root. Fixtures contain headings, links and Unicode text, with two objects on each visual row. Each benchmark iteration performs:

• BenchmarkDocumentSnapshot: one document snapshot from an existing tree, including traversal and derived metadata.
• BenchmarkDocumentHeadingNavigation: a forward/backward heading round trip on an existing tree. The adjacent case visits two headings near the middle; wrap moves between the last and first headings. Each iteration returns to the same position, keeping the workload independent of the loop count. Time and allocations are per round trip (two moves), including snapshot construction and navigation bookkeeping.
• BenchmarkDocumentLineAssembly: one line lookup in the middle of a document, combining the two text objects on that row. This includes snapshot construction.
• BenchmarkDocumentCaretNavigation: a character forward/backward round trip within the same text object. Reusable caret ranges are warmed before timing.
• BenchmarkDocumentWorkerHeadingNavigation: a heading forward/backward round trip through the review worker, including focus synchronization, diagnostics and placement dispatch, with an in-memory provider. It performs no browser IPC.

Include the additional cases with -bench '^BenchmarkDocument(Snapshot|HeadingNavigation|LineAssembly|CaretNavigation|WorkerHeadingNavigation)$'. Navigation shares an immutable indexed view; BenchmarkDocumentSnapshot continues to measure the independently owned node copy returned by Document. The first view construction is included in snapshot and line measurements and amortized over subsequent iterations. A short run can therefore overstate the steady allocation rate. See document navigation validation for cache ownership, invalidation and real-browser checks.

Fixture construction and initial focus synchronization are outside the timed loops. The benchmarks fail on errors, missing work or invalid result shapes. They use owned tree data and do not start a browser, access the desktop reader, query native AT-SPI objects, or produce speech or Braille output. They do not measure browser capture, native caret placement, input queues or audible latency. Expand the suite when a reproduced bottleneck needs a different workload, such as nested documents, tables, long text or event bursts.

Existing benchmarks also cover speech text preparation and Tentacle commands:

go test ./internal/speech -run '^$' -bench '^BenchmarkNormalizeSpeakText$' -benchmem -count=10
go test ./internal/tentacle -run '^$' -bench '^BenchmarkCommandRoundTrip$' -benchmem -count=10

The Tentacle benchmark starts disposable test peers and measures one completed command at a time across 1, 8 or 16 processes; it does not measure concurrent command saturation.

Comparing changes

Record the revision, uncommitted changes, Go version, machine and benchmark command with results. Compare repeated measurements on the same machine and toolchain, under similar load and power settings. Do not run performance comparisons with the race detector or coverage instrumentation enabled. Use benchstat from golang.org/x/perf/cmd/benchstat, if installed, to compare files:

benchstat /tmp/cthulhu-bench-before.txt /tmp/cthulhu-bench-after.txt

ns/op is elapsed time per operation, B/op is Go bytes allocated per operation, and allocs/op counts Go allocations. These are not total process memory or native C allocation measurements. A benchmark's PASS means it executed without failure; it does not mean its performance is acceptable or improved.

Investigate substantial, repeatable regressions before changing code. Profile the affected operation when the cause is unclear. Preserve the full workload and result checks when optimizing; do not improve scores by dropping required content or bypassing validation. Initially keep timings informational rather than adding timing gates to tests or push hooks. These commands can be run by automation later; no scheduled job or CI timing gate is configured here.

For this benchmark-only change, run the existing affected correctness tests:

go test ./internal/review ./internal/atspi -count=1

Future browser performance fixes also need the relevant native browser harness and human checks: navigate headings forward and backward through wrapping, read lines with arrows, move through links and editable fields, return from browser chrome, and verify fresh input interrupts old output. Check speech and physical Braille where available. Automated timing does not replace those checks or the human review required by CONTRIBUTING.md.