SRE-0.1: Spatial Reasoning Eval
Spatial reasoning is a fundamental cognitive capability for motor skills. A robot needs it to understand where objects are, how they are going to transform, and how they co-exist in the space, before robots can make wise physical decisions. Today, many researchers and hobbiests are using frontier models to drive robots, so the question emerges: Are these frontier models good at spatial reasoning?
SRE-0.1 puts that question through interactive 3D tasks in a physics simulator. A model sees only camera images, moves things one step at a time, and is scored on where they end up. Here is one task, attempted by GPT-6 Astra, Gemini Argon, and Deepseek V4.1 flash:
Preliminary results already show significant capability gaps between 3 frontier models:
Side by Side Task Replay
Pick a task to see its replays.
Finding: Marked Goal v.s. Inferred Goal
One question splits the twelve tasks: is the goal marked in the scene, or does it have to be inferred?
When the goal is marked, Argon keeps pace with Astra, and it solves the rope task that Astra misses. When the goal has to be inferred, Astra solves all six and Argon solves none. Argon fails to understand 3D transformations as well as Astra.
| Task | Astra | Argon | DeepSeek |
|---|---|---|---|
| Goal marked in the scene | |||
| Q16 Four-pin insertion | ✓1.00 14 st | ✓1.00 10 st | ◐0.25 100 st* |
| Q28 L-block, two doors | ✓1.00 11 st | ✓1.00 13 st | ✕0.00 100 st |
| Q43 Blind keyed insertion | ✕0.00 80 st | ✕0.00 14 st | ✕0.00 35 st |
| Q46 Rope through two rings | ✕0.00 34 st | ✓1.00 50 st | ✕0.00 54 st |
| Q48 Blocked-move recovery | ✓1.00 9 st | ◐0.67 11 st | ◐0.75 15 st |
| Q50 Withdraw the support | ✓1.00 11 st | ✓1.00 13 st | ✓1.00 53 st |
| Goal inferred through a 3D transformation | |||
| Q10 Rotation transfer | ✓1.00 10 st | ✕0.00 9 st | ✕0.00 19 st |
| Q16H Four shapes, tilted start | ✓1.00 66 st | ✕0.00 28 st | ✕0.00 35 st |
| Q18 Articulated chain | ✓1.00 10 st | ✕0.00 2 st | ✕0.00 70 st |
| Q24 Recover the camera pose | ✓1.00 4 st | ✕0.00 99 st | ✕0.00 98 st |
| Q27 Multi-view assembly | ✓1.00 48 st | ✕0.00 37 st | ✕0.00 39 st |
| Q29 Bent-rod insertion | ✓1.00 58 st | ✕0.00 100 st | ✕0.00 98 st |
| Mean · solved | 0.83 · 10/12 | 0.39 · 4/12 | 0.17 · 1/12 |
Scores are in [0, 1]. "st" is environment steps used, out of 100. One recorded episode per model and task. *DeepSeek's Q16 ran on an earlier harness and is not directly comparable.
How the tasks are designed: Taxonomy of spatial reasoning
Spatial reasoning is too broad to test as a single skill, so we break it into 33 atomic capabilities in eight families. An atom is one computation: a declared kind of evidence goes in, one spatial quantity comes out. Each task is built as a chain of atoms, and together the current tasks have covered 19 of the 33. More tasks are to be built to provide a higher coverage.
filled used by an SRE-0.1 task (hover for which) outlined not yet covered. Tags mark the atoms a solution relies on. They are not isolated atom measurements.
How an episode runs
Every task follows the same loop.
- ObserveImages from a few fixed cameras, plus a step counter. That's all.
- ActLook through a camera, or move an object by a 6-DoF delta in a fixed world frame. Some tasks swap in their own tool: turn a joint, move a camera, grasp a rope at an image point.
- PhysicsMuJoCo, finite forces. Moves stop on contact, and nothing snaps or auto-aligns.
- Budget100 steps shared by looks, moves and invalid calls. Submitting is free.
- ScoreGeometry of the final state against the goal: pin depths, joint angles, tip pose, camera pose, rope route, stability after release.