Learned policy
Cross-Arm · two arms
Bowl stacking
- Generation
- 66.7%
- Policy
- 80%
ScheduleGen: Demonstration Generation via Skill Scheduling for Multi-Arm Imitation Learning
1 NVIDIA 2 The University of Texas at Austin
† Equal advising
From a handful of source demonstrations to collision-aware, coordinated policy-training data.
ScheduleGen overview
Overview and task videos: 3× playback
- Simulation variants
- 28
- Mean generation success
- 79.6%
- Mean policy success
- 65.5%
- Real-world success
- 50–80%
At a glance
One pipeline, multiple embodiments
Object-relative skills, collision-aware motion, and multi-arm timing are solved together.
-
01
Integrated generation
Adapts contact skills, plans the connecting motions, and schedules every arm in one generator.
-
02
Beyond bimanual
Covers shared workspaces, clutter, long-horizon contact, four-arm systems, and a fixed-waist humanoid.
-
03
Simulation to deployment
Evaluated on 28 simulation variants and eight real-world variants through sim-to-real and real-to-sim-to-real transfer.
Method
Plan paths. Schedule time.
Per-arm paths preserve contact-rich behavior; a joint schedule resolves task, collision, and synchronization constraints.
- 01
Adapt skills
Transfer object-relative contact segments to sampled scenes.
- 02
Plan motion
Connect skills with feasible per-arm joint-space paths.
- 03
Schedule arms
Solve a MILP over collision, ordering, and synchronization constraints.
- 04
Validate data
Replay schedules in parallel simulation and retain successful trajectories.
Simulation
Generated trajectories and learned policies
Representative results across bimanual and humanoid systems. The four-arm experiments are shown separately below.
Generated trajectory
Cross-Arm · two arms
Dishware stacking
- Generation
- 68%
- Policy
- 76%
Learned policy
Humanoid · fixed waist
Nut pouring
- Layouts
- 2
- Policy range
- 48–96%
Four-arm system
Concurrent cube stacking with four arms
Table and bin tasks across normal and cross-reach layouts.
01
Cross-reach layouts
Synchronized scene and top views
Table Distribution I
Table Distribution II
Bin Distribution I
Bin Distribution II
02
Normal layouts
Side and top views of successful policy rollouts
Table Side view
Table Top view
Bin Side view
Bin Top view
Qualitative comparison
DexMimicGen vs. ScheduleGen
Side-by-side data-generation rollouts across seven task variants.
Dishware stacking
Cross-Arm layoutShared-source evaluation
Hard layouts, without new source data
ScheduleGen and the baselines use the same demonstrations within each task family.
Success (%)
Bowl
Cross-Arm
Dishware
Cross-Arm
Two-arm cube
Cross-Arm
Two-arm cube
Cross-Bin
Humanoid
Cross
Policies are evaluated over 50 episodes. Data-generation success is measured while collecting 1,000 accepted demonstrations per variant.
RoboDojo data generation
Five demonstrations become 500 per task
Input5 released demonstrationsfor each task
→
ReconstructReplayable simulation seedvalidated by task success
→
GenerateScheduleGencollision-aware coordination
→
Output500 demonstrationsfor each of 7 tasks
Shown below Successful ScheduleGen-generated trajectories. Cards report policy success over 20 held-out Nominal episodes per task.
Swap T
78.0% generation · 25% Nominal policy
Build Tower
91.7% generation · 45% Nominal policy
Insert Tubes
94.2% generation · 45% Nominal policy
Arrange Largest Number
96.5% generation · 50% Nominal policy
Press by Number
100% generation · 40% Nominal policy
Sort Nesting Dolls
88.0% generation · 65% Nominal policy
Classify Objects
75.5% generation · 65% Nominal policy
Scheduling effect
More parallel motion, shorter trajectories
Across 3,500 generated trajectories, scheduling increases simultaneous arm motion and reduces execution length relative to the released demonstrations.
Concurrent arm motion
Released15.5%
ScheduleGen31.5%
7.5%
shorter trajectories
on average
Real-world transfer
Two routes to the real robot
Every policy is evaluated over 20 hardware episodes and trained in simulation without real-world fine-tuning. 3× playback
01
Zero-shot
Sim2Real
77.5% mean success across four simulation-only policies.
Bowl stacking
Normal layout75%Bowl stacking
Cross-Arm layout80%Dishware stacking
Normal layout80%Dishware stacking
Cross-Arm layout75%02
Demonstration-seeded
Real2Sim2Real
Real demonstrations seed randomized simulation datasets; the learned policies return to the same robot.
Cube stacking
Five real seeds · 20 episodes65%Throwaway trash
Normal layout75%Throwaway trash
Cup Cross layout70%Throwaway trash
Cup + Bin Cross layout50%Paper
Abstract
Multi-arm robot platforms, including bimanual and humanoid robots, have the capability to complete tasks with lower cycle times than single-arm systems, but they are more difficult to safely control due to the presence of multi-arm constraints. The paradigm of training generalist robot policies from massive datasets is a promising approach to address general-purpose multi-arm manipulation, but the acquisition of large multi-arm manipulation datasets remains a bottleneck.
Automated demonstration generation can alleviate this burden by synthesizing new demonstrations from a handful of source demonstrations, but existing algorithms ignore multi-arm constraints and as a result often induce unsafe and unsuccessful behaviors. We address this limitation by introducing ScheduleGen, an integrated planning & scheduling algorithm for multi-arm demonstration generation.
We evaluate ScheduleGen on 28 simulation task variants, including seven externally authored RoboDojo tasks and settings inaccessible to prior systems; it achieves 79.6% mean data-generation success and trains policies with 65.5% mean success. ScheduleGen policies outperform those trained on a state-of-the-art data generation baseline, and achieve 50–80% success across eight real-world variants through zero-shot sim-to-real and real-to-sim-to-real transfer.