Method
Mission, fault vector, oracle, verdict, library.
Breakbench does not modify the application under test. It changes inputs, resources, or the environment, then asks whether defined properties still held.
[1]
Mission
A mission fixes the initial pose, layout, goal region, and timeout. Repeatability under zero fault is measured first: nominal runs continue in batches until the spread of min clearance converges or a declared cap is hit. That spread is the noise floor, not a threshold.
[2]
Fault vector
Each axis has a clean value, a direction of increasing severity, and a hard limit. A fault vector is one point in that space. The origin is a clean run. Every run stores the requested vector and the delivered vector measured at the topic.
[3]
Oracle
Verdicts are predicates over ground truth the robot does not publish. In simulation that is the Gazebo pose stream. Safety-invariant violation and mission failure are evaluated and stored separately. Self-report is recorded alongside for comparison only.
[4]
Characterization
Single-axis curves come first, k/n with a Wilson interval per severity. Monotonicity is checked before any bisection. Then an exhaustive two-axis grid with repeats. The grid is the reference any search method must beat.
[5]
Confirmation, minimization, replay
Vectors that violate in exploration are rerun with fresh seeds. Confirmed vectors are reduced toward locally minimal or Pareto-minimal severities, stored with full parameters, seeds, and versions, and replayed on later builds. When no vector violates, the fallback is a zero-fault replay, recorded as such.
Recorded runs, oracle trajectories, and corpus counts are on the prototype. Numbers there are read from run manifests.