Generate once, perform many: AI motion on any character in Isaac Sim

Short versionOne retargeter, any rig

Text-to-motion models can now generate a full-body animation from a sentence in about a second. That's a huge source of varied human motion for robotics simulation. The catch is that every clip comes out on the model's own skeleton, and simulation needs it on your characters.

I built a retargeter that puts that generated motion onto any rigged character in NVIDIA Isaac Sim: seven MetaHumans, a chibi, and a digitigrade (z-leg) hyena, all through one code path. Then I scored it against Isaac Sim's built-in retargeter on the same clips.

Then I put both on a flight of stairs. Mine plants every foot on its step. Isaac's ends up knee-deep in them.

SourceGenerated motion

The motion comes from ARDY, an open text-to-motion model. It runs locally on one RTX 4090:

• Speed: a one-time model load of about 40 seconds, then about 1 second per 8-second clip, so motion can be generated on demand rather than picked from a fixed library.
• Clips: 11 everyday scenarios (walks, a doorbell ring and wait, a knock, carrying a box and setting it down, a jog) plus a stair climb.

ARDY outputs motion on its own 77-joint skeleton, with no knowledge of the scene around it. Getting it onto real characters, correctly, is the job.

BuiltHow the retargeter works

The design comes from the way rigging teams retarget in Maya: every skeleton describes itself, and the solver works from what joints are, not what they're called. It runs as a stack of small layers, each fixing a specific failure.

1. Markup, not names. Each rig carries a joint markup file (body region, side, and the start and end of each chain). Every joint gets a role (pelvis, spine, upper arm, calf, foot...), so a rig whose joints are named Bip01_L_UpperArm goes through the same code as a MetaHuman. As a test, renaming every joint gave bit-identical output.
2. Pose alignment by direction. Each target bone is aligned to its source bone's direction, which does the job of a hand-matched retarget pose (A-pose vs T-pose) automatically.
3. A proportional spine. Pelvis, spine and neck are treated as one curve and matched by arc length, so a 2-joint chibi spine and a 5-joint MetaHuman spine follow the same motion.
4. Foot lock. Plants come from the motion itself. Each planted foot is pinned and solved with two-bone leg IK, with a knee direction learned from the animation, so knees never flip. Soft IK near full extension stops the knee snapping straight. The hips drop only when a foot is truly out of reach.
5. Hand goals. On a thicker body, copying arm angles pushes forearms through the torso. Hands instead keep the source's clearance from the body, measured against the target's own mesh. Real contacts (a hand on a hip) map to the same spot on the new body, and two-handed holds keep their spacing, so a carried box doesn't change size.
6. A per-character profile. A small JSON file next to a character tunes the arm layer. For example, the chibi's oversized jacket gets a lighter push.

BodiesDifferent bodies, one pipeline

Across 11 clips on all 7 MetaHumans:

StairsWhen the ground isn't flat

Stairs are the real test of foot IK. ARDY can generate a convincing climb, but it doesn't know where any steps are: its own feet drift up to 21 cm while planted and clip up to 18 cm into the treads. So I fit a staircase to where its feet actually land (15 cm rise, 26 cm run) and built a terrain-aware mode:

• Planted feet go on their steps, kept inside each tread so the toes stay off the riser in front.
• The hips follow the legs, placed so each planted leg is as bent (or as straight) as the source's leg.
• Swinging feet follow a planned arc from lift-off to touchdown, just high enough to clear each step edge, eased at both ends so the knee never snaps.

Result: planted feet within 0.5 cm of their step, zero slide, no toe clipping, and foot and knee motion as smooth as the source.

One honest finding: the chibi can't use the same staircase. Each step is about half her leg length, so she climbs the whole flight in a deep squat. Foot IK adapts to the body, but the environment has to suit it too.

Isaac SimWhat I learned about the built-in retargeter

Getting the built-in retargeter to produce a usable animation at all took four undocumented steps, found by reading Isaac's own test scenes and log warnings:

1. Launch with the Animation Graph runtime enabled.
2. Tag the source clip with the skeleton it was authored on.
3. Add a root joint above the hips. Without it, Isaac only logs a warning and outputs a collapsed pose.
4. Correct the source's facing axis, which auto-setup picks wrong for this skeleton.

Even then, its walkers travel on a 4.4° downward slope and end up knee-deep after a few meters. The likely cause is that root motion is carried through the MetaHuman pelvis, whose rest pose is pitched about 5°. It also does no foot planting, so feet slide. It's fine for a quick preview, but not for training data where contact matters.

Two more Isaac 6.0.1 issues, both logged with workarounds:

• Playback hangs on a SkelAnimation with non-unit joint scales.
• Playback hangs on a time-sampled custom attribute.

LimitsWhat I'd do next

• Live streaming: generate a clip on request and retarget it inside a running simulation. Generation already runs several times faster than real time.
• Interaction layer: drive hand goals from scene objects (door handles, boxes), not just the source body.
• Secondary motion: no follow-through yet for tails, ears or cloth on stylized characters.
• Stairs down, and stairs scaled to each character.
• Measured impact: feed the cleaner motion into a perception test (like Porch Watch) and measure whether it changes detector results.

WhyPhysically plausible, not just good-looking

Synthetic data for robots that work around people needs people who move like people, with feet on the floor, hands that don't pass through bodies, and variety. Generative motion supplies the variety. Retargeting is where it either holds up or quietly breaks, and that's 16 years of rigging work pointed at a new problem: not "does it look right?" but "is it physically plausible enough to train on?"

Scripted in MCA Editor. Tools: NVIDIA Isaac Sim 6.0.1, OpenUSD (UsdSkel), ARDY text-to-motion, Python/NumPy, Omniverse Replicator, Unreal Engine 5 (MetaHuman), Maya .skl joint markup.