Added some benchmarks.
This commit is contained in:
1 parent
37fdd4e114
commit
a763714fb4
3 files changed
+239
No files matched your search
@@ -38,6 +38,8 @@ Expose GUI controls through accessibility APIs. Keep outer preferences combo wid
|
|||||||
|
|
||||||
Use Go's `testing` package, `*_test.go`, and `TestBehavior` functions. Start with the affected package. Use deterministic synchronization and generic fixtures. Native checks need their relevant harness, such as `python3 tests/native_atspi.py`. No numeric coverage target is configured. Report automated, installed-binary, physical-input, audible-speech, and Braille-display checks separately.
|
Use Go's `testing` package, `*_test.go`, and `TestBehavior` functions. Start with the affected package. Use deterministic synchronization and generic fixtures. Native checks need their relevant harness, such as `python3 tests/native_atspi.py`. No numeric coverage target is configured. Report automated, installed-binary, physical-input, audible-speech, and Braille-display checks separately.
|
||||||
|
|
||||||
|
For performance work, run the relevant `BenchmarkBehavior` functions before and after changes using repeated runs and allocation reporting. Add or expand representative benchmarks when investigating a new bottleneck; keep fixture construction outside timed loops and check that the intended work actually happens. Preserve announcement content, ordering, interruption, focus, and Braille behavior. Benchmark timing alone does not establish a bug or live responsiveness, and noisy timing changes must not become automatic pass/fail thresholds. See [benchmark procedures](docs/testing/benchmarks.md) for commands and measurement limits. Plain `go test` does not run benchmarks.
|
||||||
|
|
||||||
## Commit & Pull Request Guidelines
|
## Commit & Pull Request Guidelines
|
||||||
|
|
||||||
Use short descriptive subjects consistent with history. Confirm the intended target branch; `legacy` preserves Python. Describe behavior, issues, regressions checked, skipped checks, and remaining live acceptance. Human diff review and testing precede submission under the policy in `CONTRIBUTING.md`.
|
Use short descriptive subjects consistent with history. Confirm the intended target branch; `legacy` preserves Python. Describe behavior, issues, regressions checked, skipped checks, and remaining live acceptance. Human diff review and testing precede submission under the policy in `CONTRIBUTING.md`.
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
# Performance benchmarks
|
||||||
|
|
||||||
|
Benchmarks measure time and Go allocations for repeatable operations. Run the
|
||||||
|
relevant cases before and after performance changes, alongside correctness tests.
|
||||||
|
Normal `go test` runs do not execute benchmarks; add `-bench` explicitly.
|
||||||
|
|
||||||
|
## Browser document benchmarks
|
||||||
|
|
||||||
|
Run from the repository root:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go test ./internal/review -run '^$' -bench '^BenchmarkDocument(Snapshot|HeadingNavigation|LineAssembly)$' -benchmem -count=10 > /tmp/cthulhu-bench-before.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
Repeat after the change with the same command and
|
||||||
|
`/tmp/cthulhu-bench-after.txt` as the output file. Choose distinct filenames when
|
||||||
|
retaining earlier measurements. To check that the benchmarks execute quickly,
|
||||||
|
use `-benchtime=1x -count=1`; that is a smoke check, not a useful timing baseline.
|
||||||
|
|
||||||
|
The initial cases use 100, 1,000 and 10,000 content objects, plus one section per
|
||||||
|
twenty objects and a document root. Fixtures contain headings, links and Unicode
|
||||||
|
text, with two objects on each visual row. Each benchmark iteration performs:
|
||||||
|
|
||||||
|
- `BenchmarkDocumentSnapshot`: one document snapshot from an existing tree,
|
||||||
|
including traversal and derived metadata.
|
||||||
|
- `BenchmarkDocumentHeadingNavigation`: a forward/backward heading round trip
|
||||||
|
on an existing tree. The `adjacent` case visits two headings near the middle;
|
||||||
|
`wrap` moves between the last and first headings. Each iteration returns to
|
||||||
|
the same position, keeping the workload independent of the loop count.
|
||||||
|
Time and allocations are per round trip (two moves), including snapshot
|
||||||
|
construction and navigation bookkeeping.
|
||||||
|
- `BenchmarkDocumentLineAssembly`: one line lookup in the middle of a document,
|
||||||
|
combining the two text objects on that row. This includes snapshot construction.
|
||||||
|
|
||||||
|
Fixture construction and initial focus synchronization are outside the timed
|
||||||
|
loops. The benchmarks fail on errors, missing work or invalid result shapes.
|
||||||
|
They use owned tree data and do not start a browser, access the desktop reader,
|
||||||
|
query native AT-SPI objects, or produce speech or Braille output. They do not
|
||||||
|
measure browser capture, native caret placement, input queues or audible latency.
|
||||||
|
Expand the suite when a reproduced bottleneck needs a different workload, such
|
||||||
|
as nested documents, tables, long text or event bursts.
|
||||||
|
|
||||||
|
Existing benchmarks also cover speech text preparation and Tentacle commands:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go test ./internal/speech -run '^$' -bench '^BenchmarkNormalizeSpeakText$' -benchmem -count=10
|
||||||
|
go test ./internal/tentacle -run '^$' -bench '^BenchmarkCommandRoundTrip$' -benchmem -count=10
|
||||||
|
```
|
||||||
|
|
||||||
|
The Tentacle benchmark starts disposable test peers and measures one completed
|
||||||
|
command at a time across 1, 8 or 16 processes; it does not measure concurrent
|
||||||
|
command saturation.
|
||||||
|
|
||||||
|
## Comparing changes
|
||||||
|
|
||||||
|
Record the revision, uncommitted changes, Go version, machine and benchmark
|
||||||
|
command with results. Compare repeated measurements on the same machine and
|
||||||
|
toolchain, under similar load and power settings. Do not run performance
|
||||||
|
comparisons with the race detector or coverage instrumentation enabled. Use
|
||||||
|
`benchstat` from `golang.org/x/perf/cmd/benchstat`, if installed, to compare files:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
benchstat /tmp/cthulhu-bench-before.txt /tmp/cthulhu-bench-after.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
`ns/op` is elapsed time per operation, `B/op` is Go bytes allocated per operation,
|
||||||
|
and `allocs/op` counts Go allocations. These are not total process memory or
|
||||||
|
native C allocation measurements. A benchmark's `PASS` means it executed without
|
||||||
|
failure; it does not mean its performance is acceptable or improved.
|
||||||
|
|
||||||
|
Investigate substantial, repeatable regressions before changing code. Profile
|
||||||
|
the affected operation when the cause is unclear. Preserve the full workload
|
||||||
|
and result checks when optimizing; do not improve scores by dropping required
|
||||||
|
content or bypassing validation. Initially keep timings informational rather
|
||||||
|
than adding timing gates to tests or push hooks. These commands can be run by
|
||||||
|
automation later; no scheduled job or CI timing gate is configured here.
|
||||||
|
|
||||||
|
For this benchmark-only change, run the existing affected correctness tests:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go test ./internal/review ./internal/atspi -count=1
|
||||||
|
```
|
||||||
|
|
||||||
|
Future browser performance fixes also need the relevant native browser harness
|
||||||
|
and human checks: navigate headings forward and backward through wrapping, read
|
||||||
|
lines with arrows, move through links and editable fields, return from browser
|
||||||
|
chrome, and verify fresh input interrupts old output. Check speech and physical
|
||||||
|
Braille where available. Automated timing does not replace those checks or the
|
||||||
|
human review required by `CONTRIBUTING.md`.
|
||||||
@@ -0,0 +1,148 @@
|
|||||||
|
// Copyright (c) 2026 Storm Dragon
|
||||||
|
// Authors: Storm Dragon, Angus Kola
|
||||||
|
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||||
|
|
||||||
|
package review
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"testing"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"git.stormux.org/storm/cthulhu/internal/atspi"
|
||||||
|
)
|
||||||
|
|
||||||
|
// benchmark_document_tree models an already captured page, not native AT-SPI
|
||||||
|
// capture. Each section has twenty content objects: a heading, a link, and
|
||||||
|
// ordinary text. Pairs of objects share a visual row, including Unicode text.
|
||||||
|
func benchmark_document_tree(b *testing.B, contentNodes int) (*atspi.AccessibleTree, []atspi.ObjectID) {
|
||||||
|
b.Helper()
|
||||||
|
rootID := webID("/benchmark/document")
|
||||||
|
root := atspi.NewSnapshot(rootID.Path, "Benchmark page", "document web", "", "")
|
||||||
|
root.BusName = rootID.BusName
|
||||||
|
nodes := []atspi.TreeNode{{Snapshot: root}}
|
||||||
|
contentIDs := make([]atspi.ObjectID, contentNodes)
|
||||||
|
sectionIndex := 0
|
||||||
|
for index := range contentNodes {
|
||||||
|
if index%20 == 0 {
|
||||||
|
sectionID := webID(fmt.Sprintf("/benchmark/section_%d", index/20))
|
||||||
|
section := atspi.NewSnapshot(sectionID.Path, "", "section", "", "")
|
||||||
|
section.BusName = sectionID.BusName
|
||||||
|
nodes[0].Children = append(nodes[0].Children, sectionID)
|
||||||
|
sectionIndex = len(nodes)
|
||||||
|
nodes = append(nodes, atspi.TreeNode{Snapshot: section})
|
||||||
|
}
|
||||||
|
id := webID(fmt.Sprintf("/benchmark/content_%d", index))
|
||||||
|
contentIDs[index] = id
|
||||||
|
role, text := "static", fmt.Sprintf("Café sample %d", index)
|
||||||
|
switch index % 20 {
|
||||||
|
case 0:
|
||||||
|
role, text = "heading", fmt.Sprintf("Heading %d", index/20)
|
||||||
|
case 1:
|
||||||
|
role, text = "link", fmt.Sprintf("Link %d", index/20)
|
||||||
|
}
|
||||||
|
snapshot := atspi.NewSnapshot(id.Path, text, role, "", text)
|
||||||
|
snapshot.BusName = id.BusName
|
||||||
|
snapshot.CharacterCount = utf8.RuneCountInString(text)
|
||||||
|
if role == "heading" {
|
||||||
|
snapshot.HeadingLevel = 2
|
||||||
|
}
|
||||||
|
bounds := atspi.Rectangle{X: int32(index%2) * 200, Y: int32(index/2) * 24, Width: 180, Height: 20}
|
||||||
|
nodes[sectionIndex].Children = append(nodes[sectionIndex].Children, id)
|
||||||
|
nodes = append(nodes, atspi.TreeNode{
|
||||||
|
Snapshot: snapshot,
|
||||||
|
Geometry: atspi.Geometry{Bounds: bounds, BoundsKnown: true, VisibilityKnown: true, Visible: true, Showing: true},
|
||||||
|
Text: atspi.VisibleText{Known: true, Ranges: []atspi.TextRange{{
|
||||||
|
Start: 0, End: snapshot.CharacterCount, Text: text, Bounds: bounds,
|
||||||
|
}}},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
tree, err := atspi.NewAccessibleTree(rootID, nodes)
|
||||||
|
if err != nil {
|
||||||
|
b.Fatal(err)
|
||||||
|
}
|
||||||
|
return tree, contentIDs
|
||||||
|
}
|
||||||
|
|
||||||
|
func BenchmarkDocumentSnapshot(b *testing.B) {
|
||||||
|
for _, contentNodes := range []int{100, 1000, 10000} {
|
||||||
|
b.Run(fmt.Sprintf("nodes-%d", contentNodes), func(b *testing.B) {
|
||||||
|
tree, _ := benchmark_document_tree(b, contentNodes)
|
||||||
|
root, err := tree.Root()
|
||||||
|
if err != nil {
|
||||||
|
b.Fatal(err)
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
wantNodes := contentNodes + contentNodes/20 + 1
|
||||||
|
b.ReportAllocs()
|
||||||
|
for b.Loop() {
|
||||||
|
document, err := tree.Document(ctx, root)
|
||||||
|
if err != nil || len(document.Nodes) != wantNodes {
|
||||||
|
b.Fatalf("document snapshot: nodes=%d, error=%v", len(document.Nodes), err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func BenchmarkDocumentHeadingNavigation(b *testing.B) {
|
||||||
|
for _, contentNodes := range []int{100, 1000, 10000} {
|
||||||
|
for _, scenario := range []string{"adjacent", "wrap"} {
|
||||||
|
b.Run(fmt.Sprintf("nodes-%d/%s", contentNodes, scenario), func(b *testing.B) {
|
||||||
|
tree, contentIDs := benchmark_document_tree(b, contentNodes)
|
||||||
|
cursor, err := NewDocumentCursor(tree)
|
||||||
|
if err != nil {
|
||||||
|
b.Fatal(err)
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
startIndex := contentNodes / 40 * 20
|
||||||
|
wantWrapped := scenario == "wrap"
|
||||||
|
if wantWrapped {
|
||||||
|
startIndex = contentNodes - 20
|
||||||
|
}
|
||||||
|
nextIndex := (startIndex + 20) % contentNodes
|
||||||
|
if _, err := cursor.SyncFocus(ctx, contentIDs[startIndex]); err != nil {
|
||||||
|
b.Fatal(err)
|
||||||
|
}
|
||||||
|
b.ReportAllocs()
|
||||||
|
for b.Loop() {
|
||||||
|
// One operation is a round trip. Returning to the same heading
|
||||||
|
// keeps the workload independent of the calibrated loop count.
|
||||||
|
move, err := cursor.Move(ctx, DocumentHeading, 2, false)
|
||||||
|
if err != nil || !move.Moved || move.Position.Object.ID() != contentIDs[nextIndex] || move.Wrapped != wantWrapped {
|
||||||
|
b.Fatalf("forward heading navigation: moved=%v, error=%v", move.Moved, err)
|
||||||
|
}
|
||||||
|
move, err = cursor.Move(ctx, DocumentHeading, 2, true)
|
||||||
|
if err != nil || !move.Moved || move.Position.Object.ID() != contentIDs[startIndex] || move.Wrapped != wantWrapped {
|
||||||
|
b.Fatalf("backward heading navigation: moved=%v, error=%v", move.Moved, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func BenchmarkDocumentLineAssembly(b *testing.B) {
|
||||||
|
for _, contentNodes := range []int{100, 1000, 10000} {
|
||||||
|
b.Run(fmt.Sprintf("nodes-%d", contentNodes), func(b *testing.B) {
|
||||||
|
tree, contentIDs := benchmark_document_tree(b, contentNodes)
|
||||||
|
middle := contentNodes / 2
|
||||||
|
reference, err := tree.Reference(contentIDs[middle])
|
||||||
|
if err != nil {
|
||||||
|
b.Fatal(err)
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
b.ReportAllocs()
|
||||||
|
for b.Loop() {
|
||||||
|
line, err := tree.DocumentLineAtOffset(ctx, reference, 1)
|
||||||
|
if err != nil || len(line.Contents) != 2 {
|
||||||
|
b.Fatalf("line assembly: contents=%d, error=%v", len(line.Contents), err)
|
||||||
|
}
|
||||||
|
if line.Contents[0].Reference.ID() != contentIDs[middle] || line.Contents[1].Reference.ID() != contentIDs[middle+1] {
|
||||||
|
b.Fatal("line assembly returned the wrong visual row")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in new issue
Block a user