How we benchmark quantum against classical

1. The classical baseline is the best available, not a strawman

The easiest way to make a quantum method look good is to compare it against a weak classical one. We use the strongest baseline that fits the problem: exhaustive enumeration where the instance is small enough that it returns the provable optimum, exact diagonalisation for molecular ground states, and established solvers such as OR-Tools CP-SAT or CVXPY where the problem is too large to enumerate.

Where enumeration is used, the classical result is not an estimate. It is the optimum, and the quantum method is being measured against the best answer that exists.

2. Instances are generated from a published seed

A benchmark on a hand-picked instance proves nothing, because the instance can be chosen after the fact to produce the desired result. Every problem instance on these pages is generated by a documented procedure from a stated random seed, so anyone can regenerate the exact same instance and check the numbers.

3. The variational loop is counted in full

Variational algorithms such as QAOA and VQE do not run once. They run inside an optimiser that evaluates the circuit hundreds of times while it searches for parameters. A benchmark that reports the cost of the final circuit, and quietly omits the search that found it, understates the true cost by two orders of magnitude.

Every page reports the total number of circuit evaluations and the total wall-clock time for the whole optimisation, not for the last run.

4. The quantum method is given a fair optimiser

This cuts the other way, and it matters just as much. A QAOA circuit at higher depth contains the lower-depth circuit as a special case, so a deeper run scoring worse is evidence that the classical optimiser failed to converge, not that the quantum method is worse. Under-optimising the quantum side produces a flattering result for classical computing and a benchmark that does not survive scrutiny.

We use multiple restarts from different initial parameters and check that results improve monotonically with circuit depth. Where they do not, the run is treated as a bug in the benchmark rather than as a finding.

5. Scoring is scale-free

Percentage gap against the optimum is the obvious metric and it breaks whenever the optimum sits near zero, where a negligible absolute miss reads as a thousand-percent error. We report instead where the answer sits between the best and worst possible solution, and what fraction of all feasible solutions it beats. Both stay meaningful regardless of scale.

6. Approximate results carry a bound

Where the quantum result comes from an approximate simulation method, it is accompanied by a certified error statement under the ZCC-v0.1 protocol, and hardware runs carry ZHF-v0.1. An approximate number without a bound is an assertion rather than a measurement, and it has no place in a comparison.

7. Results are published whichever side wins

We publish benchmarks where the classical method wins outright, and we expect to keep publishing them. Classical computing is extremely good at most of the problems industry actually has, and a page claiming otherwise would be false.

This is the point of the exercise rather than an embarrassment to be managed. A vendor who only publishes wins tells you nothing, because you cannot tell which results were discarded. The pages where quantum does not win are what make the pages where it does worth believing.

8. Every run is dated, and re-run

Quantum hardware improves and classical solvers improve. A benchmark without a date is a claim about an unspecified moment. Every result states when it was measured, on what hardware, and with which software versions.

What we are not claiming

None of these pages assert that quantum computing is ready to replace a classical method in production today. At the sizes current hardware can run, it generally is not. What the pages provide is the comparison at the size you can actually test, the point at which the classical method stops being tractable, and what hardware would have to look like to change the answer.

The conclusion is yours to draw. Ours is only that the numbers should be real.