Six benchmark tasks through the lens of Varun Harbola’s research on freestanding oxide membranes. See where the agents were repeatable, where they struggled, and what might transfer to the lab.
Each number counts official verifier passes across three attempts. One pass shows the workflow was possible in that setup; three passes show it repeated in this small sample.
Official passes out of three trials for each task and model
Across all 70 tasks: GPT-6 Astra 143/210 (68.1%) · Claude Opus 5.5 133/210 (63.3%) · Claude Fable 5.1 84/210 (40.0%). All three systems used maximum reasoning effort. These are selected official runs, not a ranking of every current model. Official result matrix.
02 / Case studies
What the agents actually did
Each task grades an output such as a solver, a numerical estimate, or a prediction. Open the trial evidence to inspect individual verdicts.
01
Diffraction
Stacking disorder from neutron scattering
Recover how the stacking of ice layers changes during annealing.
Inputs
Time-resolved neutron count histograms and instrument geometry. The development packet exposes the shape of the data; actual counts arrive in a hidden packet.
What the agent submitted
Write a reusable solver that fits a disorder parameter at each of 100 time points using a forward model for the binned scattering intensity.
What counted as correct
The trajectory must have RMSE of at most 0.02, stay in the physical range [0, 1], and run within both 30 minutes and 1.5 times the reference solver’s runtime.
Connection to the lab
The strongest analogue for fitting structural parameters to diffraction data. A thin-film workflow would need its own geometry, calibration and suitable reference scans.
Identify and quantify phases in mixtures with incomplete reference libraries.
Inputs
Powder XRD patterns, candidate crystal structures in CIF format, instrument settings, and a known crystalline spike to anchor the mass balance.
What the agent submitted
Report the mass fraction of each identified phase, plus separate fractions for unidentified crystalline material and amorphous material when present.
What counted as correct
Every sample must meet a weight-fraction RMSE of 0.02 and separately identify the components above the detection limit. Fractions must sum to one within 0.01.
Connection to the lab
Directly relevant to using diffraction to distinguish phases. Powder phase quantification uses different assumptions from epitaxial thin-film HRXRD and reciprocal-space mapping.
Recover intrinsic properties from measurements distorted by the instrument and contact geometry.
Inputs
Time, load and depth records, spherical or Berkovich tip geometry, and two reference samples. Tip area and frame compliance are unknown; drift changes with time.
What the agent submitted
Calibrate against the reference samples, then estimate modulus and other applicable properties such as hardness, adhesion, fracture toughness or creep parameters. Separate the coating response from substrate influence where needed.
What counted as correct
The grader checks the applicable property set and estimates against intrinsic reference properties. Both missing and spurious property rows count against the submission.
Connection to the lab
Closest to extracting material properties from AFM or indentation curves. A suspended membrane bending or stretching under a tip requires different physics from indentation into a supported solid.
Infer which physical mechanisms are active and predict behavior under new loading protocols.
Inputs
Uniaxial stress and stretch records for 18 soft solids, covering different rates, relaxation, cyclic loading and oscillation.
What the agent submitted
Identify nonlinear viscoelasticity, Mullins softening and the Payne effect separately for each material. Predict nominal stress point by point on held-out loading programs.
What counted as correct
Labels are checked against reference mechanisms and stress errors are normalized by each protocol’s maximum absolute reference stress. Predictions must start at zero stress and meet the task’s net-work condition.
Connection to the lab
Useful as a pattern for membrane mechanics: fit loading histories, then validate on a different history. The soft-solid constitutive laws themselves do not transfer directly.
Optimize a neural representation of a quantum many-body state.
Inputs
A disordered spin-½ Heisenberg model on a 6 × 4 periodic lattice, with specified couplings and a fixed restricted Boltzmann machine architecture.
What the agent submitted
Find numerical weights that drive the variational energy below the grader’s threshold.
What counted as correct
The verifier recomputes energy from the weights over all 2,704,156 basis states in the zero-magnetization sector. Passing establishes a low-energy approximation within the specified architecture.
Connection to the lab
A strong computational condensed-matter example. Its connection is to modeling quantum materials, farther from the diffraction and AFM workflows above.
Compute the loading range where a crack remains trapped by the crystal lattice.
Inputs
A prescribed Mode-I silicon crack geometry, the Stillinger–Weber potential, and a specification for coupling the crack-tip region to the surrounding elastic field.
What the agent submitted
Implement the flexible boundary model and sublattice correction, then compute the lattice-trapping interval.
What counted as correct
The task checks consistency of the coupled atomistic and continuum implementation as well as the predicted crack behavior. None of the 17 entries in the official result matrix recorded a pass.
Connection to the lab
Relevant to fracture in crystalline materials, although the task models silicon rather than an oxide membrane. Included as a counterexample to the successful cases.
Use an oxide thin-film scan with known peak positions or an independently measured structural parameter. Give the agent one reference export and the instrument geometry. Ask for a script that reads the export, fits the relevant peaks, and saves residuals and a table of estimates.
Set the units and acceptance tolerances before running the script.
Freeze the script, then run it on an unseen export from the same measurement setup.
Compare with the lab’s reference values and inspect where the fit misses.
For an AFM pilot, use an independently analyzed force curve or membrane dataset. Harbola and colleagues showed that the inferred modulus can depend on whether the membrane mainly bends or stretches. Read the study.