# Benchmark protocol

The benchmark suite is designed to improve the solver without turning the test into the product.

## Case anatomy

Each `benchmarks/cases/<id>/` directory contains:

- `case.yaml`: identity, title, kind, resolution, tags, metric contract, notes, and optional golden path;
- `run.py`: local setup, execution, reduction, and structured metrics.

`benchmarks/catalog.yaml` supplies ordering and global policy. `progress.json` records the latest completed result.

## Reference hierarchy

Analytical
: Closed-form physics, conservation, reciprocity, or a separately implemented numerical method.

Forward parity
: An FDTDX observable compared with a compact, frozen Tidy3D result. Goldens include provenance and hashes; cloud generation is outside the hot loop.

Inverse-design parity
: Best objective reached locally compared with Tidy3D's best objective. Geometry and wall-clock speed are not comparison targets.

## Resolution-aware acceptance

When a case uses a coarser grid than its golden, each metric declares a `resolution_power` $p$ and uses

$$
a_\text{eff}=a_0r^{-p},\qquad
t_\text{eff}=t_0r^{-p},
$$

where $r$ is the linear `resolution_factor`, $a$ is absolute tolerance, and $t$ is relative tolerance. The case notes must explain the coarsening. This policy acknowledges discretization error without allowing ad-hoc post-failure tolerance edits.

## Metric meanings

| Comparator | Question |
|---|---|
| `scalar` | Is one number within absolute + relative budget? |
| `rel_l2` | Does an array/spectrum have the right overall shape and scale? |
| `abs_max` | Is the worst pointwise error bounded? |
| `corr` | Is pattern correlation high enough? |
| `fom_ge` | Does best optimization performance meet or exceed a threshold? |

Completed outputs must include exactly the declared finite metrics. Pass/fail is derived from them by the harness.

## Inverse design policy

The default criterion is

$$
J_{\text{best,FDTDX}}\geq
J_{\text{best,Tidy3D}}\times\texttt{fom_ratio},
$$

within a generous cap of three times Tidy3D's iteration count. Converging two or three times slower is acceptable. Additional passive-power and parameter-bound checks prevent objective loopholes.

## Golden integrity

- goldens are never updated merely because a local case fails;
- generators require an explicit cloud-run switch;
- provenance links each artifact to builder, reducer, source, and hash;
- non-converged or irreproducible references are quarantined as blocked;
- tolerances and benchmark definitions remain reviewable YAML, not hidden runner constants.