# Benchmark command line

`fdtdx-bench` is the validation interface for this fork. It reads the ordered catalog, case-local YAML, runner output, frozen goldens, and `progress.json`.

## Status and discovery

```bash
uv run fdtdx-bench status
uv run fdtdx-bench list
uv run fdtdx-bench list --failing
```

`status` gives aggregate counts. `list` includes kind and resolution factor. Blocked cases are excluded from unattended selection until their underlying reason changes.

## Execute

```bash
uv run fdtdx-bench smoke
uv run fdtdx-bench run --case analytical_plane_wave
uv run fdtdx-bench run --case invdes_power_splitter
```

`smoke` is the mandatory regression gate after solver changes. A case run updates its entry in `progress.json` from structured runner output. Tidy3D is never invoked by these commands.

## Report

```bash
uv run fdtdx-bench report
```

This regenerates `docs/PROGRESS.md`, the human-readable record checked with the repository. The public example atlas is generated separately:

```bash
uv run python scripts/generate_public_docs.py
```

## Result schema

A completed runner returns a status, resolution factor, finite metric list, and optional diagnostics. Each metric includes a comparison type plus observed/reference values or an error/threshold pair. The harness rejects undeclared or non-finite metrics and derives pass/fail from their actual results.

## Golden safety

Golden-generation tools require an explicit `--run-cloud` gate. Do not use them in ordinary development or CI. All current compact artifacts have provenance and hashes; three non-converged artifacts are quarantined as blocked rather than accepted.