Benchmark Ladder

Date: 2026-07-15

This ladder organizes mlx_atomistic benchmark rows by decision value. It is not a ranking across engines. Reference rows are used only where operation semantics and metric families line up.

Layer Rules

Layer	Purpose	Comparable rule
micro/kernel	Isolate local MLX costs such as force terms, neighbor work, virtual-site reconstruction, and synchronization.	Diagnostic unless a reference engine exposes the same operation and metric.
controlled MD	Run tiny same-workload MD rows with matching atom and step semantics.	Comparable only when both sides are `ok` and use the same metric family.
feature physics	Time MLX feature rows for GBSA/OBC, TIP4P-Ew, soft-core/lambda, and replica exchange.	Comparable only after a matching OpenMM controlled case exists; otherwise diagnostic.
scaling	Sweep sizes or policies to tell fixed overhead from force-evaluation, neighbor-list, or memory-pressure costs.	Usually MLX-only diagnostic; reference parity is separate.
reference parity	Map controlled OpenMM rows to MLX rows and allow ratios only for matching metrics.	Comparable, diagnostic, or blocked from normalized payload status.
stretch	Track real-system rows such as DHFR and bounded at-scale PME.	Comparable only for manifest-matched operations; runtime-only rows stay diagnostic until metric semantics match.

Must-Ship Rows

Row	Layer	MLX command	OpenMM command	LAMMPS mapping	Metric family	Raw output	Comparability status	Decision value
`lj-synthetic-loop`	controlled MD; reference parity	`uv run python -m mlx_atomistic.benchmarks.md_performance --sizes 32 --steps 1 --sample-interval 1 --diagnostic-interval 1 --evaluation-interval 1 --json`	`uv run python scripts/benchmark_openmm_opencl.py --platform OpenCL --particles 32 --steps 1 --warmup-steps 0 --spacing-nm 1.0 --json`	Simple LJ-like smoke mapping may use `uv run python scripts/benchmark_lammps_opencl.py --particles 32 --steps 1 --json`; first-class LAMMPS parity is deferred.	`steps/s`; `ns/day` only when timestep and step semantics are aligned	`results/same-workload-openmm-comparison/mlx-lj-synthetic-loop.json`; `results/same-workload-openmm-comparison/openmm-lj-synthetic-loop.json`; optional `results/performance-audit-harness-hardening/lammps-fast.json`	`comparable` only for MLX/OpenMM rows with matching atom count, step count, and timing metric; LAMMPS is diagnostic/deferred	Decide whether tiny full-loop overhead deserves optimization before larger-system work.
`gbsa-obc-small`	feature physics; reference parity	`uv run python -m mlx_atomistic.benchmarks.phase3_physics --evaluations 1 --waters 1 --atoms 4 --replica-steps 1 --json`	`uv run python scripts/benchmark_openmm_opencl.py --case gbsa-obc-small --platform Reference --particles 4 --steps 1 --json`	deferred; no LAMMPS mapping in this slice	`ms/eval` latency; no mixed unit ratio	MLX combined raw source: `results/same-workload-openmm-comparison/mlx-phase3-controlled.json`; MLX row extract: `results/same-workload-openmm-comparison/mlx-gbsa-obc-small.json`; OpenMM: `results/same-workload-openmm-comparison/openmm-gbsa-obc-small.json`; summary: `results/same-workload-openmm-comparison/summary.json`; audit source `results/performance-audit-harness-hardening/phase3-physics-fast.json`	`comparable` for the refreshed MLX/OpenMM Reference latency row; OpenMM reports `fixture: gbsa_obc_small`, `status: ok`, and `obc_force_setup`	Decide whether GBSA/OBC force evaluation is an optimization target from the controlled latency row.
`tip4p-ew-water`	micro/kernel; feature physics; reference parity	`uv run python -m mlx_atomistic.benchmarks.phase3_physics --evaluations 1 --waters 1 --atoms 4 --replica-steps 1 --json`; micro row: `uv run python -m mlx_atomistic.benchmarks.mm_force_terms --evaluations 1 --particles 16 --json`	`uv run python scripts/benchmark_openmm_opencl.py --case tip4p-ew-water --platform Reference --particles 4 --steps 1 --json`	deferred; no LAMMPS mapping in this slice	`ms/eval` latency for virtual-site reconstruction; no mixed unit ratio	MLX combined raw source: `results/same-workload-openmm-comparison/mlx-phase3-controlled.json`; MLX row extract: `results/same-workload-openmm-comparison/mlx-tip4p-ew-water.json`; OpenMM: `results/same-workload-openmm-comparison/openmm-tip4p-ew-water.json`; summary: `results/same-workload-openmm-comparison/summary.json`; audit sources `results/performance-audit-harness-hardening/phase3-physics-fast.json` and `results/performance-audit-harness-hardening/mm-force-terms-fast.json`	`comparable` for the refreshed MLX/OpenMM Reference virtual-site reconstruction latency row; OpenMM reports `operation_semantics: virtual_site_reconstruction` and `openmm_operation: Context.computeVirtualSites`	Decide whether TIP4P-Ew overhead is reconstruction-specific or part of a broader water-workload cost.
`soft-core-lambda`	feature physics	`uv run python -m mlx_atomistic.benchmarks.phase3_physics --evaluations 1 --waters 1 --atoms 4 --replica-steps 1 --json`	deferred mapping	deferred	`ms/eval` for lambda-grid energy/force derivative work	`results/performance-audit-harness-hardening/phase3-physics-fast.json`; future same-workload path under `results/same-workload-openmm-comparison/` when mapped	`diagnostic`; reference parity deferred	Decide whether lambda derivative work needs a larger opt-in sweep before optimization.
`replica-exchange`	feature physics	`uv run python -m mlx_atomistic.benchmarks.phase3_physics --evaluations 1 --waters 1 --atoms 4 --replica-steps 1 --json`	deferred mapping	deferred	`ms/eval`, per-replica throughput, aggregate replica throughput, swap/history counters	`results/performance-audit-harness-hardening/phase3-physics-fast.json`	`diagnostic`; reference parity deferred	Decide whether replica execution, history materialization, or swap bookkeeping is the next MLX bottleneck.
`scaling-sweep`	scaling	`uv run python -m mlx_atomistic.benchmarks.md_acceleration --include-large --evaluations 10 --json`; `uv run python -m mlx_atomistic.benchmarks.md_performance --include-large --steps 100 --json`	deferred until a controlled OpenMM sweep is specified	deferred except simple LJ-like smoke notes	`ms/eval`, neighbor-build timing, steps/s; keep units separate by row	`results/mlx-md-acceleration.json`; `results/mlx-md-performance.json`; audit smoke sources under `results/performance-audit-harness-hardening/`	`diagnostic` for MLX scaling; reference parity deferred	Decide whether overhead is fixed, neighbor-list dominated, memory-pressure dominated, or force-evaluation dominated.
`dhfr-implicit`	stretch; reference parity	`uv run python -m mlx_atomistic.benchmarks.dhfr --case dhfr-implicit --steps 1 --json`	`uv run python scripts/benchmark_openmm_dhfr.py --case dhfr-implicit --platform Reference --steps 1 --json`	deferred	`ns/day` for matching one-step GBSA/OBC rows	MLX: `results/same-workload-openmm-comparison/mlx-dhfr-implicit.json`; OpenMM Reference: `results/same-workload-openmm-comparison/openmm-dhfr-implicit.json`; summary: `results/same-workload-openmm-comparison/summary.json`; OpenMM OpenCL context: `results/openmm-opencl-dhfr-m5max.json`	`comparable` for the one-step MLX/OpenMM Reference smoke row; broader OpenCL context remains separate	Track the smaller DHFR real-system path and harden MLX artifact/runtime behavior before broad performance claims.
`dhfr-explicit-pme`	stretch; reference parity	`uv run python -m mlx_atomistic.benchmarks.dhfr --case dhfr-explicit-pme --steps 1 --amber-topology results/inputs/Amber20_Benchmark_Suite/PME/Topologies/JAC.prmtop --amber-coordinates results/inputs/Amber20_Benchmark_Suite/PME/Coordinates/JAC.inpcrd --json`	`uv run --with openmm python scripts/run_charged_pme_parity.py --mlx-prepared results/dhfr-artifacts/dhfr-explicit-pme --amber-prmtop results/inputs/Amber20_Benchmark_Suite/PME/Topologies/JAC.prmtop --amber-coordinates results/inputs/Amber20_Benchmark_Suite/PME/Coordinates/JAC.inpcrd --replicas 1,1,1 --platform OpenCL --out results/scalable-charged-pme-runtime/jac-1x`	deferred	manifest-matched energy/complete-force parity; MLX `ns/day` is diagnostic until a matching OpenMM runtime row exists	`results/scalable-charged-pme-runtime/jac-1x/charged_pme_parity_report.json`; `results/scalable-charged-pme-runtime/jac-1x/runtime-smoke.json`; OpenMM OpenCL context: `results/openmm-opencl-dhfr-m5max.json`	parity passed and the explicit-plasma MLX step is runnable; no throughput ratio because runtime schemas do not match	Keep the charged-system convention explicit and improve the measured runtime without weakening parity or fail-closed policy.
`jac-charged-pme-94k`	stretch; scaling; reference parity	`uv run python -m mlx_atomistic.benchmarks.charged_pme runtime --prepared results/larger-system-scaling/jac-2x2x1-modern/prepared --warmups 10 --steps 75 --seed 17 --out results/larger-system-scaling/jac-2x2x1-modern/matched-runtime-v2/mlx_runtime.json`	`uv run --no-sync python scripts/run_openmm_charged_pme_runtime.py ... --replicas 2,2,1 --platform OpenCL --precision single --warmups 10 --steps 75 --out results/larger-system-scaling/jac-2x2x1-modern/matched-runtime-v2`	deferred	manifest-matched energy/complete-force parity and fixed-cell NVT throughput	parity/profile JSON under `results/scalable-charged-pme-runtime/jac-2x2x1/`; matched runtime under `results/larger-system-scaling/jac-2x2x1-modern/matched-runtime-v2/`; scalable-charged-pme-runtime-m5max.md	comparable on the exact 75-step protocol; MLX/OpenMM = `9.7586x`	Preserve the bounded charged-PME envelope and target the remaining direct-nonbonded cost without weakening parity.
`gpcrmd-729-pme-92k`	stretch; reference parity	`uv run python -m mlx_atomistic.prep.gpcrmd_benchmark --target-id gpcrmd-729-beta1-5f8u-cyanopindolol --prepared results/gpcrmd-pme-runtime-closure/prepared --protocol-manifest results/gpcrmd-pme-runtime-closure/prepared/mlx-workload-manifest.json --warmups 1 --measured-steps 2 --checkpoint-restart --out results/gpcrmd-pme-runtime-closure/runtime --force --json`	`uv run --with openmm python scripts/run_gpcrmd_pme_parity.py --source-manifest results/gpcrmd-pme-runtime-closure/source/fixture-manifest.json --cache notebooks/ligand-receptor-motion/data/gpcrmd-cache/729 --mlx-prepared results/gpcrmd-pme-runtime-closure/prepared --platform OpenCL --out results/gpcrmd-pme-runtime-closure/parity`	deferred	manifest-matched total/component energy and complete-force parity; MLX bounded fixed-cell NVT, restart, and PME-stage diagnostics	source, parity, runtime, profile, and blocker-matrix evidence under `results/gpcrmd-pme-runtime-closure/`; gpcrmd-729-pme-runtime-m5max.md	parity passed at 92,001 atoms and bounded MLX runtime/restart passed; no throughput ratio without a matching OpenMM NVT runtime manifest	Close the selected fixture’s stale topology/PME blockers while retaining the fixed-cell NVT and short-run claim boundary.

Reference Coverage Notes

OpenMM parity rows belong under scripts/ and emit normalized ok, diagnostic, or blocked payloads. Ratios are valid only for matching ok rows.
LAMMPS is deferred for this ladder except for simple LJ-like smoke notes. Do not map LAMMPS materials, protein, or package-specific benchmark families to MLX rows without a separate same-workload plan.
DHFR now has explicit same-workload row IDs. dhfr-implicit is runnable as a one-step GBSA/OBC smoke row. dhfr-explicit-pme, the 94,232-atom JAC supercell, and GPCRmd 729 pass manifest-matched energy/complete-force parity under their declared charge policies. Their MLX runtime rows do not receive an OpenMM throughput ratio without matching OpenMM runtime manifests. ApoA1, Cellulose, and STMV OpenMM reports remain reference context until matching MLX workloads exist.