Skip to main content
Agent benchmark

Stim agent benchmarks

Compare how coding agents launch the same React Native app with Stim and the local Expo/native toolchain. Platforms and JavaScript/native tasks are measured separately, and every published time links to its command-level audit and Settings-screen proof.

iOS performance across models​

Each bar is one valid run; missing or invalid cells are labeled. Lower time is better.

Android performance across models​

Each bar is one valid run; missing or invalid cells are labeled. Lower time is better.

Launch failure diagnosis​

A deterministic root-render exception is committed before dispatch. The agent must launch first, diagnose from captured errors, repair the source, and prove the unchanged Settings screen. Each comparison shows the latest validated run per arm, not an average or a best-of selection. Stim runs use optional app readiness logs; retained control runs do not. The crash and recovery task are unchanged, but this compares integrated Stim with the standard toolchain, not identical instrumentation.

Methodology

How the comparisons work​

We compare Stim with standard Expo and native build tools on the same app. The goal is to measure how long an agent takes to complete a task and check the result in a running app.

Read the full protocol
Same task, same agent
Each comparison uses the same app task, AI model, and model settings. The agent works with Stim in one run and without it in the other. Launch-error Stim reruns add optional readiness logs; the retained controls use the earlier fixture without those logs.
Same hardware
Runs use the same Mac mini and run one at a time, so they do not compete for resources. The device model and OS version are matched within each comparison.
Both setups start prepared
Dependencies are installed and build caches are warmed before timing starts. Stim can reuse saved builds and, on iOS, an existing simulator. We are measuring reuse during development, not first-time setup.
Consistent timing
Each clock runs from the agent's first recorded action until it has checked the result and saved a screenshot. The same start and finish rules apply with and without Stim.
Different kinds of work
We test JavaScript changes, native changes, and fixing an app that fails to launch. Results stay separate by task, model, and platform, so a quick JavaScript change is not compared with a native rebuild.
Results you can inspect
Each bar represents the latest checked run, not an average or the fastest of several attempts. New validated runs replace the previous result in place. Open it to see the commands, logs, and screenshots. Runs that fail the protocol checks are excluded; the full protocol explains those rules.
Archived replay

Watch the Luna JavaScript run unfold​

A continuous, silent replay of the Luna JavaScript run. This archived video retains the original dispatch clock; the interactive audit now starts at first recorded agent activity. Commands, output, and proof come from the recorded run; matching simulator interaction was recreated with agent-device and plays at 2x speed.

Command-level evidence

Inspect every benchmark run​

Compare environments, play each command timeline, inspect terminal output, and open proof images and simulator recordings.

Explore detailed audits