Operating runbook

Install and verify

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
export OPENBLAS_NUM_THREADS=4
export OMP_NUM_THREADS=4
export MPLCONFIGDIR=/tmp/matplotlib-25d
python -m unittest discover -v
python validate_25d.py --quick --output outputs_quick_validation

Stop if either command fails.

Reproduce the accepted validation

python optimize_25d.py --profile validated --output outputs_reproduction
python validate_25d.py \
  --checkpoint outputs_reproduction/checkpoint.npz \
  --output outputs_reproduction_validation

Require accepted: true in the validation metrics.json.

Start a discovery campaign

python optimize_25d.py --profile discovery --steps 1 \
  --output outputs_discovery_step1

Measure one step's time and memory before requesting more steps. Continue from the accepted checkpoint:

python optimize_25d.py --profile discovery --steps 1 \
  --resume outputs_discovery_step1/checkpoint.npz \
  --output outputs_discovery_step2

Each restart treats the loaded topology as a new initial state. Retain every checkpoint directory; do not overwrite the previous accepted result.

Validate a discovery result

validate_25d.py automatically adds two x harmonics, one y harmonic, and two vertical nodes relative to the checkpoint configuration:

python validate_25d.py \
  --checkpoint outputs_discovery_stepN/checkpoint.npz \
  --output outputs_discovery_stepN_validation

If the field correlation or gain gate fails, reject the topology. Do not relax the thresholds to preserve a favorable Q.

Transfer to full-vector 3D

python export_3d_geometry.py \
  outputs_discovery_stepN/checkpoint.npz \
  --output outputs_discovery_stepN/density_3d_seed.npz \
  --nx 40 --ny 26 --nz 22

On the 3D branch:

1. fit the exported physical density to the 3D filtered/projected raw variables;
2. reset the protected strip exactly to air;
3. attach the exact semi-infinite periodic mirrors with the correct phase;
4. reacquire a broad source-coupled spectrum;
5. identify the transferred field by overlap and atom-gap localization;
6. run the full multi-probe spectral, PML, and grid gates;
7. never use the 2.5D Q as a tracking value or acceptance threshold.

Reproduce the included paired transfer

Set VECTOR_PROJECT to the full-vector handoff directory and select its controlled configuration. The transfer carries geometry only:

python transfer_to_vector3d.py \
  --checkpoint outputs_validated/checkpoint.npz \
  --vector-project "$VECTOR_PROJECT" \
  --vector-config "$VECTOR_PROJECT/outputs_bragg_q200_refined_validation/config.json" \
  --output-dir outputs_vector3d_ab

Use the full-vector project's frequency_search.py and multiprobe_sweep.py independently on outputs_vector3d_ab/{initial,optimized}/run_data.npz. The included evidence uses a 17-point 0.8--1.2 broad scan, a 41-point 1.066--1.086 seven-probe wide scan, and a 29-point local scan of half-span 0.0025 about the lower branch. Then run:

python validate_vector3d_ab.py \
  --vector-project "$VECTOR_PROJECT" \
  --initial-wide outputs_vector3d_ab/initial/wide/multiprobe_data.npz \
  --initial-local outputs_vector3d_ab/initial/local_lower/multiprobe_data.npz \
  --optimized-wide outputs_vector3d_ab/optimized/wide/multiprobe_data.npz \
  --optimized-local outputs_vector3d_ab/optimized/local_lower/multiprobe_data.npz \
  --output-dir outputs_vector3d_ab/paired_lower_validation

python analyze_vector3d_ab.py \
  --vector-project "$VECTOR_PROJECT" \
  --initial-warmstart outputs_vector3d_ab/initial/run_data.npz \
  --optimized-warmstart outputs_vector3d_ab/optimized/run_data.npz \
  --branch-names lower upper \
  --initial-frequencies 1.0731548063 1.079441324 \
  --optimized-frequencies 1.0731482886 1.079436400 \
  --output-dir outputs_vector3d_ab/field_match

Do not reinterpret accepted: false in the paired pole metrics as an accepted Q gain. The direct real-axis response and vector-field identity remain useful transfer diagnostics.

Run the controlled-grid full-vector linewidth optimizer

The required backend is included in vector_backend/; --vector-project can still select another compatible checkout. The baseline gradient sweep and withheld midpoint data are checkpointed, so the included accepted step can be reproduced without recomputing its baseline:

python fullvector_linewidth_optimizer.py \
  --warmstart outputs_vector3d_ab/optimized/run_data.npz \
  --baseline-local-data \
    outputs_fullvector_upper_diagnostic/_work/iteration_0_baseline_local.npz \
  --baseline-midpoint-data \
    outputs_fullvector_upper_diagnostic/local_midpoints/multiprobe_data.npz \
  --output-dir outputs_fullvector_linewidth_reproduced \
  --learning-rate 0.002 \
  --minimum-q-improvement 0.001

Require accepted: true and every entry in gates to be true. The trial sweep and withheld sweep are saved frequency by frequency and safely resume after interruption. For a new iteration, use the last accepted run_data.npz as the warm start and reacquire a gradient-bearing local baseline for that topology before proposing another step. Do not reuse response gradients after the topology changes.

Run the direct 780 nm atom-response proposal

The current immediate campaign uses the centered-emitter driven Green response at exactly 780 nm, not Q, as the proposal objective. The physical topology grid has sixteen exact cells across the one-wavelength gap:

python optimize_atom_driven_25d.py \
  --config configs/atom_780_25d_parity_scaleup.json \
  --output outputs_atom_780_driven \
  --step 0.035 \
  --minimum-improvement 0.0001

This command deliberately saves a driven proposal and reports scientifically_accepted_2_5d_step: false. It is useful for iterating the new objective while the nonlinear pole gate is made restartable. Do not promote the checkpoint until all pending_gates in metrics.json are resolved.

The strict combined driver is:

python optimize_atom_25d.py \
  --config configs/atom_780_25d_parity_scaleup.json \
  --output outputs_atom_780_25d \
  --steps 1 \
  --gradient-check smoke

The parity scale-up config restricts the scalar Fourier/FEM field to the exact centered-emitter sector that is even in x, y, and z. At the current launch basis this reduces the linear problem from 1,701 to 275 unknowns while reproducing the full-basis response, field, and atom gradient to roundoff. Run the parity/full equivalence, DtN, and nonlinear-Q gradient gates with:

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 \
python validate_25d.py --quick \
  --output outputs_scaleup_780/parity_25d_validation

Historical checkpoints keep their full coefficient dimensions because parity reduction is opt-in. Transfer density only when switching to the reduced basis, then reacquire every driven field and pole.

The strict reduced nonlinear mode attempt remains historical evidence, but the full-vector 780 nm path is no longer blocked. The scale-up qualification uses the retarded condensed lead, exact subpixel mirror/envelope fills, density-only transfer, and independently reacquired fields. Restart or reproduce its resolved spectra with:

export VECTOR3D_PARALLEL_SURFACES=1
export OPENBLAS_NUM_THREADS=1
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1

python scaleup_qualification.py sweep \
  --config configs/fullvector_atom_780_scaleup_y40.json \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --start 0.995 --stop 1.005 --points 21 --seed-existing
python scaleup_qualification.py sweep \
  --config configs/fullvector_atom_780_scaleup_y40.json \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --start 0.99525 --stop 1.00425 --points 10 \
  --sweep-name q_sweep_midpoints_a
python scaleup_qualification.py sweep \
  --config configs/fullvector_atom_780_scaleup_y40.json \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --start 0.99575 --stop 1.00475 --points 10 \
  --sweep-name q_sweep_midpoints_b
python scaleup_qualification.py pole \
  --config configs/fullvector_atom_780_scaleup_y40.json \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --derivative-step 0.0001 --max-iterations 8
python scaleup_qualification.py analyze \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --reference-real 0.998301156386388 \
  --reference-imag -0.0009486543641915186
python scaleup_qualification.py rational-analyze \
  --output-dir outputs_scaleup_780/source_consistent/y40

The first uniform 41-point spectrum rejected the fixed four-fold pole-spread gate despite predicting the held-out complex responses accurately. Its metrics and plots are preserved with the suffix 41_rejected. The follow-up evidence set keeps the same AAA order and acceptance thresholds and adds 16 midpoint samples at the largest fold-disagreement regions:

python scaleup_qualification.py sweep \
  --config configs/fullvector_atom_780_scaleup_y40.json \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --start 0.995125 --stop 0.995875 --points 4 \
  --sweep-name q_sweep_adaptive_low
python scaleup_qualification.py sweep \
  --config configs/fullvector_atom_780_scaleup_y40.json \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --start 0.997875 --stop 0.998625 --points 4 \
  --sweep-name q_sweep_adaptive_pole
python scaleup_qualification.py sweep \
  --config configs/fullvector_atom_780_scaleup_y40.json \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --start 0.999875 --stop 1.001625 --points 8 \
  --sweep-name q_sweep_adaptive_upper

The completed 57-point adaptive set still rejected the pole-displacement gate: its maximum holdout displacement was 0.14227 linewidth while the limit stayed 0.10. Because inserting nonuniform points also changes sorted interleaved fold membership, the final declared validation grid is the complete uniform 81-point grid from 0.995 through 1.005 at spacing 0.000125. The 24 missing samples are restartable shards:

python scaleup_qualification.py sweep \
  --config configs/fullvector_atom_780_scaleup_y40.json \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --start 0.996125 --stop 0.997625 --points 7 \
  --sweep-name q_sweep_uniform81_low
python scaleup_qualification.py sweep \
  --config configs/fullvector_atom_780_scaleup_y40.json \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --start 0.998875 --stop 0.999625 --points 4 \
  --sweep-name q_sweep_uniform81_middle
python scaleup_qualification.py sweep \
  --config configs/fullvector_atom_780_scaleup_y40.json \
  --output-dir outputs_scaleup_780/source_consistent/y40 \
  --start 1.001875 --stop 1.004875 --points 13 \
  --sweep-name q_sweep_uniform81_high

Do not rerun the analyzers or make a spectral decision until all 81 samples exist. rational-analyze now makes that fixed-grid completion an explicit gate in addition to the unchanged response, pole-spread, Q-spread, Q-agreement, and resonance thresholds.

The pole command atomically saves its field, center, and full history after every correction. It tracks the branch by complex-field overlap and uses a safeguarded secant center only after two correlated evaluations are available; otherwise it uses relaxed continuation. Its complete real-axis operator derivative includes both lead self-energies. Pole-field localization is source-free: gap and inner energy must each meet the uniform-grid fraction of the same x mask (0.40 and 0.80 on these grids), while PML energy must remain below 0.25. Do not copy the stronger 0.55/0.85 driven-field thresholds onto a quasinormal eigenvector; those include the centered point-source singularity.

A branch-targeted second-order real-axis Taylor companion was also tested as a possible correction to the first-order Q. Its untargeted form selected a different Q = 1076.94 root; the targeted form returned no localized root with even 0.95 same-basis complex-field overlap. Both attempts are rejected and archived under the y40 output directory. Do not substitute either quadratic Q for the accepted first-order pole. The independent uniform response is the declared cross-check.

After a density-only domain transfer, the accepted reference pole's real frequency may be supplied as --initial-center to avoid rediscovering the continuation window. This transfers no Maxwell field: the new discretization still assembles its own operators and reacquires the mode from scratch. A saved pole_checkpoint.npz always takes precedence over this starting hint.

The sweep skips every completed frequency exactly. Use --seed-existing only when intentionally replacing a locator sweep with a denser coincident grid; it archives the previous samples as q_sweep_locator.npz. Named q_sweep_* shards can run independently and the analyzers deterministically union every finished sample, with exact-frequency de-duplication.

The first expanded-domain comparison is retained as rejected evidence. Its mode fields correlate above 0.994, but Q changes by 36.63%; moreover, it changed the PML thickness from 0.30/0.24 to 0.40/0.32 wavelengths in y/z. The convergence command now refuses that confounded comparison. Rebuild the coarse and reference profiles with the same physical PML thickness as the expanded profile, carrying density only:

python scaleup_qualification.py prepare-vector \
  --config configs/fullvector_atom_780_scaleup_y36_pmlmatched.json \
  --source-config outputs_scaleup_780/source_consistent/y40/config.json \
  --source-state outputs_scaleup_780/source_consistent/y40/field_780.npz \
  --output-dir outputs_scaleup_780/pml_matched/coarse --fit-steps 500
python scaleup_qualification.py prepare-vector \
  --config configs/fullvector_atom_780_scaleup_y40_pmlmatched.json \
  --source-config outputs_scaleup_780/source_consistent/y40/config.json \
  --source-state outputs_scaleup_780/source_consistent/y40/field_780.npz \
  --output-dir outputs_scaleup_780/pml_matched/reference --fit-steps 500
python scaleup_qualification.py point \
  --config configs/fullvector_atom_780_scaleup_y36_pmlmatched.json \
  --output-dir outputs_scaleup_780/pml_matched/coarse --vacuum-reference
python scaleup_qualification.py point \
  --config configs/fullvector_atom_780_scaleup_y40_pmlmatched.json \
  --output-dir outputs_scaleup_780/pml_matched/reference --vacuum-reference
python scaleup_qualification.py pole \
  --config configs/fullvector_atom_780_scaleup_y36_pmlmatched.json \
  --output-dir outputs_scaleup_780/pml_matched/coarse \
  --derivative-step 0.0001 --max-iterations 8 \
  --initial-center 0.998408872394809
python scaleup_qualification.py pole \
  --config configs/fullvector_atom_780_scaleup_y40_pmlmatched.json \
  --output-dir outputs_scaleup_780/pml_matched/reference \
  --derivative-step 0.0001 --max-iterations 8 \
  --initial-center 0.998408872394809

The already completed expanded endpoint was independently solved with:

python scaleup_qualification.py point \
  --config configs/fullvector_atom_780_scaleup_y40_expanded.json \
  --output-dir outputs_scaleup_780/source_consistent/y40_expanded \
  --vacuum-reference
python scaleup_qualification.py pole \
  --config configs/fullvector_atom_780_scaleup_y40_expanded.json \
  --output-dir outputs_scaleup_780/source_consistent/y40_expanded \
  --derivative-step 0.0001 --max-iterations 8 \
  --initial-center 0.998301156386388

The starting center is a scalar frequency hint, not a transferred Maxwell field. After the PML-matched coarse/reference poles complete, rebuild the strict decision with:

python scaleup_qualification.py convergence \
  --coarse-dir outputs_scaleup_780/pml_matched/coarse \
  --reference-dir outputs_scaleup_780/pml_matched/reference \
  --expanded-dir outputs_scaleup_780/source_consistent/y40_expanded \
  --output outputs_scaleup_780/convergence_metrics.json
python scaleup_qualification.py summarize \
  --output-dir outputs_scaleup_780

Do not start an optimization unless qualification_metrics.json is accepted. Q is a required diagnostic, but the retained-step objective is still the centered z-dipole response at exactly 780 nm.

Stop conditions

• nonlinear residual above 2e-5;
• upper-half-plane pole or nonpositive Q;
• same-basis overlap, defect energy, or protected-gap energy below threshold;
• nonzero protected-strip density or reflection error above numerical noise;
• cross-basis field correlation below 0.95;
• Q gain fails to persist on the refined basis;
• topology crosses a Fourier-channel threshold far enough that the configured outgoing sheet is no longer the correct continuation.
• fitted linewidth improvement fails on the withheld interleaved frequencies.

Run the persistent qualified 780 nm production campaign

After outputs_scaleup_780/production_qualification/metrics.json reports all five launch gates accepted, start the unbounded restartable driver with:

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 \
python scripts/run_production_atom_campaign.py \
  --output-root outputs_atom_780_production \
  --learning-rate 0.004 \
  --minimum-learning-rate 0.00025 \
  --maximum-learning-rate 0.064 \
  --learning-rate-growth-factor 2 \
  --strong-model-agreement-ratio 0.75 \
  --armijo-fraction 0.25 \
  --maximum-backtracks 6 \
  --q-workers 4 \
  --q-every 10

The driver performs one topology update at a time and runs all direct gates on every proposal. The adaptive trust radius grows only after strong agreement between predicted and exact log-LDOS gain. A failed Armijo or scientific gate halves the step, and every attempted rate and rejection reason is archived. The exact 81-point, seven-probe Q diagnostic runs every tenth step; intervening retained checkpoints are explicitly provisional and remain available for backfilling. It has no iteration limit unless --max-iterations is supplied. Stop it with Ctrl-C; the last complete checkpoint and any partial spectrum remain restartable. A physical pole candidate pauses the loop for nonlinear mode reacquisition rather than assigning a Q from the real-axis scan. Optimizer and supervisor status files are atomically replaced; keep this property when adding new live fields because the supervisor polls them while a solve is running.

Publish the live state independently:

python scripts/publish_dashboard.py --watch --interval 60