Benchmark Inventory And Gap Matrix
Date: 2026-07-15
Origin: .agent/work/2026-05-22-performance-audit-harness-hardening.
This living inventory started with the Phase 3 harness audit and now includes
later measured benchmark and closure surfaces.
External OpenMM, LAMMPS, OpenBenchmarking, and MLX material is used only as
benchmark-design context, not as a pass/fail target for mlx_atomistic.
Tier Rules
Section titled “Tier Rules”| Tier | Purpose | Required availability | Result location |
|---|---|---|---|
| Fast developer | Smoke-check importable MLX benchmark modules and JSON/CSV shape. | uv run environment only; no mandatory OpenMM, LAMMPS, OpenCL, or large fixture. | Temporary pytest paths, stdout JSON, or optional local CSV. |
| Opt-in performance | Larger Apple Silicon runs, prepared production fixtures, and reference-engine context. | Local accelerator, optional OpenMM/LAMMPS/dev reference setup, and optional gitignored fixtures. | Raw JSON/CSV under gitignored results/; committed summaries under docs/benchmarks/. |
Historical Neutral-PME Branch
Section titled “Historical Neutral-PME Branch”feat/neutralized-dhfr-pme-validation was audited at commit 7584ec5 against
current main and is retired, not merged. Its still-useful evidence is a
controlled 24,488-atom neutral TIP3P/NaCl PME result: it is explicitly not a
DHFR protein or GPCRmd simulation. The branch recorded three OpenMM parity
configurations, bounded NVE/NVT checks, and a 0.06008 s median MLX LJ+PME
nonbonded evaluation with no sustained five-evaluation memory growth.
Current main’s charged 94,232-atom JAC and source-backed 92,001-atom GPCRmd
rows supersede that branch for production-path capability claims. No branch
code or deletion-heavy history was ported; the neutral row remains historical
context at 7584ec5, and NPT plus analytic virial remain open.
Current MLX Benchmark Modules
Section titled “Current MLX Benchmark Modules”| Module | Current coverage | Test/doc evidence | Result/raw-output location | Tier | Phase 3 gap |
|---|---|---|---|---|---|
src/mlx_atomistic/benchmarks/lj_md.py | LJ MD modes and CSV output. | tests/test_benchmarks.py::test_lj_benchmark_csv_smoke | Caller-provided --csv; stdout otherwise. | Fast developer | Does not cover virtual sites, TIP4P-Ew, GBSA/OBC, soft-core/lambda, or replica exchange. |
src/mlx_atomistic/benchmarks/md_performance.py | End-to-end synthetic LJ MD throughput, neighbor policy, cadence, synchronization, finite output, and MLX runtime metadata. | tests/test_benchmarks.py benchmark smoke and JSON-file output coverage. | Stdout JSON/CSV or caller-provided --json-out; raw outputs belong in results/. | Fast developer now; opt-in for larger sizes. | No Phase 3 feature-specific row. |
src/mlx_atomistic/benchmarks/neighbor_nonbonded_parity.py | Compact mlx_cell_pairs versus tiled all-pairs energy/force parity, topology semantics, candidate waste, and build/evaluation timing through caller-selected sizes. | tests/test_benchmarks.py::test_neighbor_nonbonded_parity_command_writes_validated_row; scalable-neighbor-nonbonded-runtime-m5max.md. | Caller-provided --out; at-scale raw JSON under results/scalable-neighbor-nonbonded-runtime/. | Fast smoke; opt-in at scale. | Validates the generic short-range neighbor axis; charged PME scale is covered by the separate JAC row. |
src/mlx_atomistic/benchmarks/md_acceleration.py | Neighbor build versus force evaluation split, backend policy, pair representation, and waste counters. | tests/test_benchmarks.py benchmark smoke coverage. | Stdout JSON/CSV when invoked by module. | Fast developer now; opt-in for larger sizes. | No virtual-site, GBSA/OBC, soft-core/lambda, or replica-exchange overhead row. |
src/mlx_atomistic/benchmarks/cadence_sensitivity.py | Reporter/evaluation cadence and synchronization/materialization counts. | tests/test_benchmarks.py benchmark smoke coverage. | Stdout JSON/CSV when invoked by module. | Fast developer | Can inform future replica exchange history/materialization timing, but does not cover replica exchange today. |
src/mlx_atomistic/benchmarks/mm_force_terms.py | Bonded autodiff, neighbor-list build, LJ pair eval, direct Coulomb, combined nonbonded, constraints, and TIP4P-Ew virtual-site reconstruction/force redistribution micro-rows. | tests/test_benchmarks.py::test_force_term_benchmark_includes_profile_rows | Caller-provided --csv; stdout JSON with --json. | Fast developer | Still a microbenchmark surface, not a production-scale advanced-water workload. |
src/mlx_atomistic/benchmarks/phase3_physics.py | Fast normalized rows for virtual sites, TIP4P-Ew M-site reconstruction, GBSA/OBC energy/forces and surface-area term, soft-core/lambda derivative grid, and two-replica exchange. | tests/test_benchmarks.py::test_phase3_physics_benchmark_covers_required_feature_rows | Caller-provided --csv; stdout JSON with --json; raw audit outputs under results/performance-audit-harness-hardening/. | Fast developer | Synthetic probes only; larger opt-in rows are still needed before optimization claims. |
src/mlx_atomistic/benchmarks/charged_pme.py | Deterministic prepared supercells plus bounded fixed-cell charged-PME NVT with plan, neighbor, throughput, finite-state, and memory diagnostics. | tests/test_prep_supercell.py; charged-PME tests in tests/test_benchmarks.py; scalable-charged-pme-runtime-m5max.md. | Caller-provided prepared/output paths; measured JAC outputs under results/scalable-charged-pme-runtime/. | Fast blocked-path smoke; opt-in preparation/runtime at scale. | Validated on the fixed orthorhombic JAC envelope, not GPCRmd, NPT, analytic virial, or triclinic PME. |
src/mlx_atomistic/benchmarks/pme_performance.py | Plan-aware PME stage profiling against legacy, charged-JAC, or GPCRmd parity reports; explicit prepared/report paths; fail-closed atom-count, manifest, PME-config, lazy-topology, NeighborBlocks, and no-fallback admission. | Profile and blocked-path coverage in tests/test_benchmarks.py; scalable-charged-pme-runtime-m5max.md; gpcrmd-729-pme-runtime-m5max.md. | Caller-provided fixture or explicit --parity-report/--prepared; measured profiles under results/scalable-charged-pme-runtime/ and results/gpcrmd-pme-runtime-closure/. | Opt-in performance; blocked-path smoke is fast. | Dense O(N²) reference is intentionally disabled at large scale; the production direct path must remain shared-block based. |
src/mlx_atomistic/benchmarks/ewald_reference.py | Small-system Ewald correctness/backend timing, explicitly not GPCRmd-scale PME. | tests/test_benchmarks.py::test_ewald_reference_benchmark_json_and_csv_smoke | Caller-provided --csv; stdout JSON with --json. | Fast developer | Does not cover soft-core/lambda derivatives or advanced water models. |
src/mlx_atomistic/benchmarks/stability.py | NVE/NVT stability diagnostics for small systems. | tests/test_benchmarks.py::test_stability_cli_json_and_csv_smoke | Caller-provided --csv; stdout JSON with --json. | Fast developer | Could catch stability impacts after Phase 3 rows exist, but no Phase 3 timing row today. |
src/mlx_atomistic/benchmarks/validation_gauntlet.py | Finite-difference force validation cases. | tests/test_benchmarks.py::test_validation_gauntlet_cli_json_and_csv | Caller-provided --csv; stdout JSON with --json. | Fast developer | Validation-oriented, not a performance row for Phase 3 features. |
src/mlx_atomistic/benchmarks/schema.py | Shared normalized benchmark fields and default command helpers. | Imported by current benchmark modules and exercised by tests/test_benchmarks.py. | N/A helper module. | Fast developer support | New benchmark rows should use this helper to keep report joins simple. |
src/mlx_atomistic/benchmarks/gpcrmd_runtime.py and mlx_atomistic.prep.gpcrmd_benchmark | Shared reporting plus source-manifest-driven warmup, measured NVT, trajectory/checkpoint, and restart orchestration. | GPCRmd/checkpoint coverage in tests/test_gpcrmd_registry.py and tests/test_checkpoint_restart.py; gpcrmd-729-pme-runtime-m5max.md. | Caller-provided prepared/output paths; measured 92,001-atom evidence under results/gpcrmd-pme-runtime-closure/. | Fast small-fixture regression; opt-in real-system execution. | Validated only for the selected fixed-cell orthorhombic NVT fixture; not production-length or NPT coverage. |
DFT benchmark modules: dft_scf.py, dft_operator.py, dft_pseudopotential.py, dft_geometry.py, dft_nonlocal.py, dft_solver.py, dft_spin_kpoints.py, dft_relaxation.py | DFT operation and solver smoke timings. | DFT smoke tests in tests/test_benchmarks.py. | Caller-provided --csv; stdout JSON with --json. | Fast developer | Out of the Phase 3 MD physics gap set. |
Reference Scripts And Docs
Section titled “Reference Scripts And Docs”| Surface | Current coverage | Test/doc evidence | Result/raw-output location | Tier | Boundary |
|---|---|---|---|---|---|
scripts/benchmark_openmm_opencl.py | Synthetic OpenMM/OpenCL LJ reference benchmark with blocked payload for unavailable platform. | tests/test_benchmarks.py::test_openmm_opencl_unavailable_platform_non_json_does_not_crash; docs under docs/benchmarks/openmm-opencl-*.md. | Caller-provided --csv; raw OpenMM report examples under results/openmm-opencl-*.json. | Opt-in reference; blocked smoke is fast. | openmm-reference, design context only. |
scripts/run_openmm_mlx_parity.py | OpenMM versus MLX parity workflow support. | Existing parity tests outside this slice; not a benchmark smoke in tests/test_benchmarks.py. | Local run artifacts under results/ when invoked. | Opt-in reference/context | Reference parity context, not product runtime dependency. |
scripts/run_charged_pme_parity.py | Independent AMBER/OpenMM replica construction, strict workload manifests, charged background gate, component energies, and complete-force comparison. | tests/test_openmm_mlx_parity.py, tests/test_pme_validation.py, and charged-PME benchmark coverage; scalable-charged-pme-runtime-m5max.md. | Caller-provided output root; measured JAC parity under results/scalable-charged-pme-runtime/. | Opt-in reference/correctness at scale. | OpenMM remains reference-only; unsupported force classes or manifest mismatches block metrics. |
scripts/run_openmm_charged_pme_runtime.py | Manifest-bound OpenMM fixed-cell NVT timing for the validated charged JAC workload, plus MLX/OpenMM protocol admission and ratio. | One-time at-scale execution and static admission through charged_pme.audit_openmm_runtime_artifacts; scalable-charged-pme-runtime-m5max.md. | Caller-provided output root; current matched evidence under results/larger-system-scaling/jac-2x2x1-modern/matched-runtime-v2/. | Opt-in reference/performance at scale. | OpenMM remains reference-only; topology, PME, dynamics, completion, or timing-boundary mismatch suppresses the ratio. |
scripts/run_gpcrmd_pme_parity.py | Independent source-built CHARMM/OpenMM comparator, canonical manifest comparison, component energies, resolved PME grid, and complete-force comparison for GPCRmd. | tests/test_gpcrmd_pme_parity.py; gpcrmd-729-pme-runtime-m5max.md. | Caller-provided source/cache/prepared/output paths; measured parity under results/gpcrmd-pme-runtime-closure/parity/. | Opt-in reference/correctness at scale. | OpenMM remains reference-only; partial force arrays, force-class drift, or manifest mismatch block metrics. |
scripts/run_openmm_mlx_npt_parity.py | NPT parity workflow support. | Existing NPT/parity tests outside this slice. | Local run artifacts under results/ when invoked. | Opt-in reference/context | Reference parity context, not product runtime dependency. |
scripts/run_openmm_production_md_reference.py | Production MD reference command surface. | Existing production reference tests outside this slice. | Local run artifacts under results/ when invoked. | Opt-in reference/context | OpenMM context only; not a pass/fail throughput target. |
scripts/run_mlx_production_md_probe.py | MLX production-probe command surface. | Existing production probe tests outside this slice. | Local run artifacts under results/ when invoked. | Opt-in performance | MLX product probe, but not a routine fast gate. |
scripts/benchmark_lammps_opencl.py | Synthetic LAMMPS/OpenCL reference benchmark with normalized ok or blocked payloads. | tests/test_benchmarks.py::test_lammps_opencl_reference_payload_is_normalized; docs command matrix and baseline audit. | Caller-provided --csv; stdout JSON with --json; raw audit output under results/performance-audit-harness-hardening/lammps-fast.json. | Opt-in reference; blocked smoke is fast. | lammps-reference, design context only. |
docs/benchmarks/README.md | Engine labels, file template, index, external input policy, raw-output policy. | This inventory is linked from README. | Committed Markdown in docs/benchmarks/. | Documentation | Defines mlx_atomistic, openmm-reference, and lammps-reference. |
docs/benchmarks/openmm-opencl-dhfr.md, openmm-opencl-apoa1.md, openmm-opencl-amber20.md | OpenMM OpenCL summary reports for DHFR, ApoA1, Cellulose, and STMV on Apple M5 Max. | Indexed from docs/benchmarks/README.md. | Raw JSON paths named under gitignored results/. | Opt-in reference documentation | External/reference comparison only. |
Phase 3 Coverage Gaps
Section titled “Phase 3 Coverage Gaps”| Feature | Current implementation/test evidence | Current benchmark placement | Remaining benchmark gap |
|---|---|---|---|
| virtual sites | tests/test_virtual_sites.py covers virtual-site position reconstruction, force redistribution, artifact round trip, runner propagation, and simulation configuration plumbing. | phase3_physics.py covers reconstruction and force redistribution; mm_force_terms.py adds synchronized TIP4P-Ew micro-rows. | Larger opt-in advanced-water workload. |
| TIP4P-Ew | tests/test_virtual_sites.py covers tip4p_ew_virtual_site, reference geometry, prepared-system round trip, and artifact build exposure. | phase3_physics.py emits a TIP4P-Ew M-site row; mm_force_terms.py emits TIP4P-Ew reconstruction and redistribution rows. | Couple the row to a larger nonbonded/PME workload before optimization claims. |
| GBSA/OBC | tests/test_gbsa.py covers GBSA surface-area energy, OBC force finite-difference behavior, OpenMM OBC reference energy, artifact loading, and save/load parameters. | phase3_physics.py emits OBC energy/force and surface-area rows. | Scaling row over larger atom counts. |
| soft-core/lambda | tests/test_soft_core.py covers finite overlap, endpoint equivalence, finite-difference energy_forces_dlambda, wrapper delegation, artifact metadata, and fail-closed non-cutoff electrostatics. | phase3_physics.py emits an energy_forces_dlambda lambda-grid row. | Larger lambda-grid sweep. |
| replica exchange | tests/test_replica_exchange.py covers Metropolis probability, adjacent swaps, lambda-scaled Hamiltonians, odd/even pairing, metadata validation, and unsupported runtime inputs. | phase3_physics.py reports per-replica throughput, swap counts, acceptance rate, and history materialization count. | Larger opt-in multi-replica workload. |
External Context Caveats
Section titled “External Context Caveats”| Context source | Useful design signal | Caveat |
|---|---|---|
| OpenMM | Public reports commonly use ns/day on named systems such as DHFR, ApoA1, Cellulose, and STMV, with platform, precision, timestep, constraints, hydrogen mass, ensemble, and cutoff/PME settings recorded. | OpenMM numbers are reference context only. They are not direct pass/fail targets for MLX because hardware, backend, precision, and engine semantics differ. |
| LAMMPS | Official benchmark families cover LJ liquid, polymers, metals/EAM, granular systems, and protein/rhodopsin-style systems with atom counts, timesteps, packages, precision/backend, and loop-time style metrics. | LAMMPS coverage should stay opt-in and fail-soft; historical public numbers are context for scaling behavior, not target thresholds. |
| OpenBenchmarking | The LAMMPS profile records repeatable run metadata and reports ns/day for rhodopsin/protein-sized systems with variance reporting. | OpenBenchmarking helps shape provenance and repetition fields; it is not an apples-to-apples MLX acceptance gate. |
| MLX | Public MLX benchmark practice emphasizes operation-level timing, Apple Silicon device/runtime metadata, synchronization-aware measurements, and CPU/GPU/backend distinctions. | MLX context informs measurement hygiene for mlx_atomistic; it does not provide atomistic throughput targets by itself. |
Slice 1 Findings
Section titled “Slice 1 Findings”- The fast benchmark gate is currently centered on
tests/test_benchmarks.py. - Current committed benchmark docs include OpenMM reference reports, the benchmark README, this inventory, and the baseline audit report.
- Raw benchmark outputs should remain under gitignored
results/; committed Markdown indocs/benchmarks/should summarize reproducible rows and cite raw output paths. - The Phase 3 named features now have normalized fast benchmark rows; remaining gaps are larger opt-in workloads for optimization validation.