Benchmark command line

fdtdx-bench is the validation interface for this fork. It reads the ordered catalog, case-local YAML, runner output, frozen goldens, and progress.json.

Status and discovery

uv run fdtdx-bench status
uv run fdtdx-bench list
uv run fdtdx-bench list --failing

status gives aggregate counts. list includes kind and resolution factor. Blocked cases are excluded from unattended selection until their underlying reason changes.

Execute

uv run fdtdx-bench smoke
uv run fdtdx-bench run --case analytical_plane_wave
uv run fdtdx-bench run --case invdes_power_splitter

smoke is the mandatory regression gate after solver changes. A case run updates its entry in progress.json from structured runner output. Tidy3D is never invoked by these commands.

Report

uv run fdtdx-bench report

This regenerates docs/PROGRESS.md, the human-readable record checked with the repository. The public example atlas is generated separately:

uv run python scripts/generate_public_docs.py

Result schema

A completed runner returns a status, resolution factor, finite metric list, and optional diagnostics. Each metric includes a comparison type plus observed/reference values or an error/threshold pair. The harness rejects undeclared or non-finite metrics and derives pass/fail from their actual results.

Golden safety

Golden-generation tools require an explicit --run-cloud gate. Do not use them in ordinary development or CI. All current compact artifacts have provenance and hashes; three non-converged artifacts are quarantined as blocked rather than accepted.