JustRUA

1.1Codex and full history

The official L3 harness (L3 Inspect EEF, RoboProbe) calls the model API directly and keeps images from the last two observations only. We run the same loop in a Codex session^* that keeps every image, with Codex's own tools off. Codex alone lowered the first-round score, so we credit the gain to the two together. All comparisons keep the default cap of 100 decisions per episode, which only the final JustRUA lifts.

1.2Action design

Motion length. In the official loop a whole motion runs before the model sees anything, so an unstable CuRobo plan can carry the arm far. We stop each motion after 20 control steps. Letting the model choose 3 to 32 steps, as in action chunking (ACT, Astranomy), did no better, so JustRUA keeps the fixed cap.

Alternative interfaces. Two alternatives from recent systems did not beat absolute targets. Relative targets, offsets from the measured pose as in RoboDawn, scored lower, as Astranomy also found, and the arm struck objects almost three times as often. Every coordinate the model reads is in the world frame, so an offset likely asks for one more subtraction before each move.

Semantic operations replace raw targets with named skills, as in SayCan and Code as Policies. Ours are built on move_eef and add no new control code or learned policy. They changed what the model did more than how well it did. It moved its camera more than three times as often and dropped more of what it held, and the score did not rise.

Move sequences. Up to four moves per decision raised the score, mostly by easing the 100-decision budget. The final configuration of JustRUA removes the budget instead.

1.3Reasoning effort

Medium effort, the default, scored highest overall. Low effort gave up most of the precision score, and high effort traded precision for memory at a higher cost.

2.1Demonstrations

Demonstrations in context are a common way to help GPT-6 Astra (SenseNova). Both versions draw on one expert-curated successful episode per task from RoboDawn, inserted once before the first decision. The visual demo shows its 16 keyframes, and the text trajectory gives the same episode stage by stage, as poses, commands and outcomes in our coordinates.

Evaluation set	Default	Visual demo	Text trajectory
Generalization	42.5	41.0	40.3
Precision	25.0	22.8	11.6
Long-horizon	24.7	28.9	28.3
Memory	49.6	70.0	55.4
Open	48.1	36.6	40.3
Overall	38.0	39.9	35.2
Cost per episode	$4.94	$5.11	$5.84
Behaviour, first round
Handovers through the table	9	15	13
Dropped objects	26	20	17
Arm-to-arm collisions	6	0	0
Empty grasps	47	50	53

Cost per first-round episode.

Visual demo Text trajectory

The 16 keyframes the model sees before its first decision in make_kong, head camera.

1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10
11. 11
12. 12
13. 13
14. 14
15. 15
16. 16

Three tasks also include the author's wrist-camera views. No action trace or robot state is attached.

Five of the 40 stages of the same episode. The handover appears only as coordinates.

Stage 26 (source turn 28, control step 1522):
left: pose=[-0.4,-0.05,0.9079,0.0,0.0,90.0]; opening=0.78
right: pose=[0.3,-0.2019,0.922,90.0,0.0,0.0]; opening=0.18
Source event: LEFT arm closed on an object with the fingertips at z = 81.3 (still holding it).
1. left move x +20.0 => nominal left target_pose=[-0.2,-0.05,0.9079,0.0,0.0,90.0]; result=ok
2. left move x +20.0 => nominal left target_pose=[0.0,-0.05,0.9079,0.0,0.0,90.0]; result=FAILED
3. left move y -18.0 => nominal left target_pose=[0.0,-0.23,0.9079,0.0,0.0,90.0]; result=ok
Source failed commands: ["left move x +20.0"]

Stage 27 (source turn 31, control step 1642):
left: pose=[-0.001,-0.23,0.7789,0.0,0.0,90.0]; opening=0.78
right: pose=[0.3,-0.2019,0.922,90.0,0.0,0.0]; opening=0.18
Source event: LEFT arm closed on an object with the fingertips at z = 81.3 (still holding it).
1. left gripper 1.00 => left_gripper=1; result=ok

Stage 28 (source turn 32, control step 1656):
left: pose=[-0.001,-0.23,0.7789,0.0,0.0,90.0]; opening=1
right: pose=[0.3,-0.2019,0.922,90.0,0.0,0.0]; opening=0.18
1. left move z +12.0 => nominal left target_pose=[-0.001,-0.23,0.8989,0.0,0.0,90.0]; result=ok
2. left home => nominal left target_pose=[-0.3,-0.2019,0.922,90.0,0.0,0.0] (return to the demonstration initial pose; source joint fallback is not available); result=ok

Stage 29 (source turn 33, control step 1892):
left: pose=[-0.3,-0.2019,0.922,90.0,0.0,0.0]; opening=1
right: pose=[0.3,-0.2019,0.922,90.0,0.0,0.0]; opening=0.18
1. right gripper 1.00 => right_gripper=1; result=ok
2. right point down => nominal right target_pose=[0.3,-0.207,0.9169,0.0,0.0,0.0]; result=ok
3. right move x -20.0 => nominal right target_pose=[0.1,-0.207,0.9169,0.0,0.0,0.0]; result=ok
4. right move x -10.1 => nominal right target_pose=[-0.001,-0.207,0.9169,0.0,0.0,0.0]; result=ok

Stage 30 (source turn 36, control step 2052):
left: pose=[-0.3,-0.2019,0.922,90.0,0.0,0.0]; opening=1
right: pose=[-0.001,-0.208,0.7759,0.0,0.0,0.0]; opening=1
1. right gripper 0.00 => right_gripper=0; result=ok

About 16,000 characters per task on average, injected once and kept in the conversation.

2.2Self-improvement loop

Several RoboDojo entries improve themselves by writing their own skills (EmbodiedEvo, SenseNova-RoboRSI). Here a Codex agent writes extra operations on top of move_eef, with a guide on when to use them, starting from the RoboDawn demonstrations. The 42 tasks are split into five groups. Each round the model runs the library on one group, and the agent reviews the scores, logs and videos and revises it. Nothing is edited by hand, and nothing is stored or looked up by task.

place(..., release_after_lowering=false)
grasp(..., close_after_approach=false)
insert(arm, entry_point, tip_offset, axis, standoff, depth, advance)

place(..., retreat_direction={x, y, z})
pivot(arm, feature_offset, target_orientation, feature_shift)

tap(arm, target_point, feature_offset, axis, clearance, activate)
guide: "record attempted contact separately from verified activation"

transport(..., max_segment_length=<metres>, carry_after_lift=false)

hold(cycles, note)

The library changed how the model grasped, with fewer empty grasps and no arm-to-arm collisions in the first round. With each task practised once, it scored slightly higher than the loop's own runs on the tasks it had seen and stayed level with the default on the evaluation set. Systems that report clear gains practise each task many more times (EmbodiedEvo, PhysRSI-eval). The loop helps, but slowly.

3.1Robot readings

Robot readings are each gripper's basic sensor values, finger positions and forces, contact force and wrist load, with no privileged information such as the true pose of an object.

They help most in grasping. The default interface reports only the jaw opening the model asked for, while the readings give the opening the fingers actually reached and the force at each fingertip.

insert_tubes, after closing	Default	With readings
Jaw opening, 0 to 1	0.24	0.44 13.6 mm
Contact force, N	not reported	24.9 / 24.5

The fingers stopped at 0.44, not the commanded 0.24, and press on both sides, so the tube is held.

How the readings are written matters too. As full JSON they crowded the context and pushed many episodes into compaction, so JustRUA writes them in a short text format.

3.2Depth

Many recent agent systems use depth to place objects in 3D (RoboHarness, GPT-GEAR, RAX), but the RoboDojo leaderboard does not accept simulator depth for agent entries, as a maintainer stated in review. We test both simulator depth and depth that Depth Pro predicts from RGB alone.

The predicted distances are far off, by tens of centimetres even with the true focal length. Scaling each frame to the robot's geometry helps, but the scale drifts between frames and cannot be fitted for the wrist cameras, so JustRUA passes the raw predictions on.

0 m2 m or more

Head camera, table area	Mean absolute error	Pearson r	Spearman ρ
Raw prediction	73.7 cm	0.993	0.992
With the true focal length	40.1 cm	0.993	0.992
Scaled to robot geometry, per frame	7.3 cm	0.993	0.992

Images from make_kong. The table averages six head-camera frames from make_kong and fill_egg_holder against simulator depth. Scaling a frame leaves both correlations unchanged.

3.3Affordances

Depth shows where surfaces are, but not where an object can be held or which arm can reach it. That knowledge is an affordance. Recent agent systems compute affordances with dedicated models (CLARO, ORA), and since many episodes go wrong at the grasp, we tested giving them to the model directly.

For each camera view, GraspGen proposes up to eight grasp poses per object and inverse kinematics marks which surfaces each arm can reach. Both are drawn onto the image, and the text lists the grasp poses. The hard part is presentation, since anything added to an observation either takes more space or crowds what is already there.

Affordances drawnSimulator depth · 3 images

Drawn onto each camera view, with no depth images. The marks cover the pixels they point at.

Drawn with RGB and depthSimulator depth · 9 images

Clean RGB and depth go back in, so each camera view appears three times.

Drawn with depthPredicted depth · 6 images

Depth predicted from RGB, with the clean RGB left out again.

Affordance panelPredicted depth · 4 images

Clean RGB at full size. The six other views shrink to 256 × 192 pixels in one panel, with at most four grasp candidates.

Head camera at the first decision of insert_tubes. Click an image to enlarge it.

None of the four versions grasped more reliably or scored higher than depth images from the same source. Affordances could matter in few moments, since handovers stayed rare and grasp candidates only matter when a grasp starts, while their cost comes with every observation. We left them out of JustRUA.

3.4Coordinate solving

In the last step the model may also write and run code in its session¹, as in Code as Policies and URAI, and the 100-decision cap is lifted.

Almost every run uses code to solve for coordinates, casting rays from pixels through the camera calibration and triangulating a point seen by two cameras or meeting a known surface height. The code never reads the depth images. Code thus acts as one more source of scene geometry, which is why it sits on this axis.

In insert_tubes the model locates the tubes and the holes of the rack from pixels and places all three tubes.

¹ Web search stays off, so that the model cannot look up the benchmark or its solutions.

4.1Failure classification

4.2Potential fixes

Each of the first three causes points to a general fix, but each needs knowledge that is costly to supply.

Revising the task prompts fixes the prompt and scoring cases and raises the first round from 54.0 to 55.6. The rest would need trial and error on each task, close to tuning on the test tasks.