Skip to content

Add kitless teleoperation and replay with SO101 support - #8788

Open
hougantc-nvda wants to merge 2 commits into
isaac-sim:mainfrom
hougantc-nvda:hougantc/kitless-live-recorded-teleop
Open

hougantc-nvda wants to merge 2 commits into
isaac-sim:mainfrom
hougantc-nvda:hougantc/kitless-live-recorded-teleop

Conversation

@hougantc-nvda

@hougantc-nvda hougantc-nvda commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds live headset teleoperation, demonstration recording and replay without Kit, using the OVXR renderer and session from #8787. The SO101 arm supports the kitless path, including CPU OvPhysX. Live, recording and MCAP replay share launch validation, presentation updates and shutdown helpers.

Review stack · 2/5: #8787 → #8788 → #8789 → #8790 → #8791.

Review only this layer on the fork mirror. The upstream PRs target main; please leave comments here. This layer depends on #8787.

  • Kit XR uses --xr; kitless XR uses --ovxr, with one environment and kitless physics. --ovxr_session_mode {ar,vr} selects the OpenXR session mode. record_demos.py accepts the same options and records through the same OVXR session as live teleoperation.
  • OvxrPresentation owns the presentation path shared by live, recording and replay, including paused updates that leave physics unchanged. --ovxr_step_hz caps the host-control rate with --ovxr, while XR rendering runs independently. A task's anchor can move the headset view with the robot; an OVXR session uses the new ExternalXrAnchor.
  • Initial and reset publications must be admitted before the next input is consumed. Termination or truncation resets the anchor, retargeter and action history without resetting the environment a second time, and operator START/PAUSE/RESET events remain available. IsaacTeleopDevice.notify_reset() reports an environment reset to the input pipeline. TeleopSessionLifecycle delivers it by overriding the root pipeline's execution methods, for every Isaac Teleop session; test_host_reset_pipeline.py runs the override against the real graph and asynchronous runner.
  • MCAP replay follows recorded control events, with --start_on_replay for recordings without START. A recorded RESET resets the replay and waits for the next START instead of ending it. Without Kit, replay retargets synchronously, one recorded frame per step. A kitless backend also replays headless without --ovxr and reports each run's outcome without render timing; in that mode a task that sets target_frame_prim_path is rejected with an error that points to --ovxr or Kit.
  • replay_demos.py replays HDF5 demonstrations on a kitless backend selected with --physics, also when Isaac Sim is installed.
  • Tasks can supply an experimental IsaacTeleopCfg.prepare_environment hook. The live, replay and recording launchers run it once before creating simulation resources.
  • When the environment or the teleop device cannot be created, the live command exits non-zero, with or without --ovxr.
  • A retargeting failure in a live OVXR session ends the run with the original error, because the borrowed XR session cannot attach input again.
  • The entrypoints close the environment on the host thread once the XR session has closed. OVXR renderer startup is bounded at 20 minutes, which covers a first start that compiles shaders.

For existing Kit users (each is in the changelog):

  • The live agent and the recorder report environment resets to the input pipeline with notify_reset: after an operator reset, and when environment 0 terminates or is truncated during a step.
  • MCAP replay ends a run as a failure when a retargeting step fails; on main it restarts the input session, which replays the recording from its start. It rebuilds the simulation with env.sim.reset() only for Isaac Sim PhysX, and it raises when the CloudXR runtime it launched fails to stop.
  • run_teleoperation owns its cleanup: it no longer takes a cleanup stack and requires the keyword-only use_isaac_teleop.
  • Teleoperation cleanup no longer stops at its first failure, replay_demos.py --validate_states no longer raises a shape mismatch, and the recorder logs its instructions when there is no Kit window.
  • SO101 IK-Abs uses a full-pose clutch home. Its teleop and recording paths use nominal joint resets through the prepare_environment hook.
  • XrAnchorSynchronizer computes the same anchor pose through a policy class it shares with OVXR, and the helper that looks up an existing anchor prim reads the stage through get_current_stage().

Outside source/isaaclab_teleop/, each change sits with its owner:

  • isaaclab_ov: three optional contact settings on OvPhysxCfg, which keep the stage's values when unset.
  • isaaclab_newton: capture_graph captures on a non-blocking stream, so graph capture no longer conflicts with concurrent renderer or input uploads. Torch work that the caller runs on Warp's current stream, as MDP2 does, moves to the capture stream with it and is still recorded.
  • isaaclab_tasks: the SO101 IK-Abs task gains the ovphysx preset, the tuned OvPhysX configuration for physx without Kit, the clutch home and the reset hook; test/core/test_preset_cli.py gains two public-selector tests.
  • Docs: the kitless XR guide and its links.

Dependencies: #8787 and Isaac Capture 1.7.118rc1 through the locked ovxr extra. See the runtime guide and the installation guide.

Type of change

  • New feature
  • Bug fix
  • Documentation update

Testing

Tested as a stack on main at 6980e3d088a.

  • The changelog gate and the teleop unit tests pass at this layer's head. The tests cover entrypoints, replay bootstrap, reset publication, cleanup ordering, startup-failure exits, the anchor split and the SO101 pipeline.
  • A headless kitless SO101 replay without --ovxr exits 1 with the error that names isaac_teleop.target_frame_prim_path.

Validation of the full stack (#8787 to #8791) without Kit, on one GPU (RTX 5090 Laptop) and on two (2x RTX 6000 Ada):

  • The changelog gate passes on every PR of the stack, and ruff passes on every file the stack changes.

  • Unit tests, one file per process: the teleop tests pass on every PR (467, 631, 672, 755 and 815 tests on Add a teleop-owned OVXR renderer and session adapter #8787 to Add GPU selection for OVXR rendering and scene updates #8791), with isaaclab_ov's OVRTX renderer tests (164), the MDP2 and Newton capture tests (76) and the task, Mimic and OvPhysX tests (84, 183, 185 and 185 on Add kitless teleoperation and replay with SO101 support #8788 to Add GPU selection for OVXR rendering and scene updates #8791). The test files that need Kit were not run in this environment (test_teleop_scripts_smoke.py, test_openxr_device_constructors.py, test_oxr_device.py, test_retargeters.py, test_xr_camera_feed_kit_scene_ui_integration.py).

  • The native tests on the real libraries (the owner-transfer harness and marker visibility) pass on every PR that has them. With two GPUs, test_ovxr_renderer.py passes 98 tests with none skipped, test_check_cuda_peer.py passes 5, and the native transfer harness passes 6 of 6.

  • Live sessions with an emulated headset (IWER) and the CloudXR runtime launched by the session, on the two-GPU machine. Every session exits 0 with no dropped frames, with camera feeds, mirror and markers where the task has them:

    Session FPS over three client connections
    GR1T2, Newton, one GPU 26.3 / 26.2 / 25.6
    GR1T2, Newton, physics on GPU 1 and rendering on GPU 0 33.5 / 33.0 / 33.5
    GR1T2, Newton, physics on GPU 0 and rendering on GPU 1 33.9 / 33.8 / 34.5
    GR1T2, OvPhysX, one GPU 31.2 / 30.4 / 30.2 (26.7 / 26.6 / 26.8 with debug markers)
    GR1T2, CPU OvPhysX, rendering on GPU 0 32.9 / 32.6 / 32.1
    GR1T2, CPU OvPhysX, rendering on both GPUs 47.7 / 47.0 / 47.9
    Fixed-base G1, Newton 33.8 / 33.2 / 32.7
    SO101, OvPhysX 43.9 / 44.3 / 44.0
    G1 locomanipulation, Newton 27.1 / 27.0 / 27.0
    G1 locomanipulation, OvPhysX 28.7 / 29.2 / 29.5
  • On the one-GPU machine, GR1T2 on CPU OvPhysX and on Newton (camera feed, mirror, markers) and SO101 on OvPhysX start, stream both eyes and shut down cleanly.

  • record_demos.py --ovxr writes HDF5 and MCAP on one GPU and with physics and rendering on separate GPUs. Replaying the HDF5 of the split-GPU recording matches all 1394 recorded states exactly.

  • Ctrl-C during recording, during replay and while renderer attach is held, a forced render-worker failure and forced start-up failures all exit with every native resource closed on the main thread and no process or port left.

  • MCAP replay of a GR1T2 pick-and-place capture with --start_on_replay --num_success_steps 1000 --require_hand_replay_data exits 0 on Newton and on CPU OvPhysX (also when selected as --physics physx), with 455 active steps and 457 recorded hand frames. This checks that the replay consumes the whole recording; it is not a task-success check.

  • Not run: a physical headset (the sessions above use the emulator), a Kit XR session and Kit record_demos.py --xr. The Kit tests and the docs build are left to CI.

Known issues: renderer startup stalled twice in native ovrtx.attach_ovstage during development; identical retries passed, and the root cause is unknown. CloudXR logs a cuMemGetInfo error when a client disconnects.

Checklist

  • Contribution guidelines read
  • Public documentation and focused contract coverage updated
  • Repository formatting and changelog checks passed
  • No new warnings

@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team infrastructure labels Oct 23, 2026
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch 2 times, most recently from e933899 to 4d7c54f Compare October 26, 2026 17:12
@hougantc-nvda hougantc-nvda changed the title Add live teleoperation and MCAP replay through OVXR Add kitless teleoperation and replay with SO101 support Oct 26, 2026
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch 2 times, most recently from fe8d8d2 to 8d56932 Compare October 29, 2026 02:05
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch 2 times, most recently from 293a54c to bb8944d Compare October 29, 2026 16:56
@hougantc-nvda

Copy link
Copy Markdown
Contributor Author

@greptile Please review the latest changes in this PR.

@greptile-apps

greptile-apps Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The optional OVXR route connects teleoperation, simulation, and an external render worker.

Update the publication-failure test double before merging; both parameter cases otherwise fail at the new material lookup.

Findings

  1. P1 Publication tests fail early ▶

Summary

Adds kitless live teleoperation and MCAP replay, SO101 support, and ordered XR cleanup.

Since the last review:

  • Live tracking and recorded input can drive scenes shown through OVXR.
  • OVRTX resources can be leased to an external renderer owner.
  • The SO101 absolute-IK task supports CPU OVPhysX and a full gripper home pose.
  • CPU arrays, Newton graph capture, and simulation cleanup support the kitless runtime.
  • Headless Kit applications keep updating the scene while XR is active.

Previous unnumbered threads were rechecked without posting duplicates. The callback uses validate_live(), and Kit updates both anchors from the filtered pose. hougantc-nvda disputed the replay Reset finding because supported Capture forced requests only defer during shutdown; no retry change was added.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Task changes material] --> B[Mark channel dirty]
    B --> C[Check whether publication is needed]
    C --> D[Reserve scene ordinal]
    D --> E[Write materials and scene]
    E --> F[Wait for material reads]
    F --> G[Seal scene ordinal]
    G --> H[XR worker renders]
Loading

Reviews (4) · Last reviewed commit: "Merge updated prerequisite for #8788" · Reviewed by Greptile

@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch from bb8944d to 62a1200 Compare October 29, 2026 18:18
@hougantc-nvda

Copy link
Copy Markdown
Contributor Author

@greptile Please review the current head (62a1200) again. Recheck the fixes and remaining issues against the published code, including the actual interfaces and callers at this PR boundary.

@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch from 72a3b68 to 01fc065 Compare October 30, 2026 01:00
@hougantc-nvda

Copy link
Copy Markdown
Contributor Author

@greptile please review this PR again at the latest head (01fc06561c0d65c5dbe75fe56e5e18b7172dcd91), including the changes since your previous review.

@hougantc-nvda
hougantc-nvda marked this pull request as ready for review October 30, 2026 01:48
@hougantc-nvda
hougantc-nvda requested a review from a team October 30, 2026 01:48

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Needs attention: Scope and over-engineering (4), Compatibility and deprecation (3), Implementation (6), Style consistency (3), Test quality (4). Other reviewed categories look good.

  • PR description: Describe the included lease/adapter, consolidate validation totals, and benchmark publication/readback latency with baseline/candidate revisions, command/workload, hardware, backend, environment count, warmup, and metric; distinguish combined-stack qualification from this head.
  • Design and architecture: Architecture looks good.
  • Scope and over-engineering: Minimal implementation: Extend OVRTXRenderer and OvLibrariesXrSession; keep render_lease.py for thread-affine borrowing, ov_libraries_xr.py for Capture integration, ovxr_cfg.py for pre-runtime configuration, and _ovxr_launch.py for shared checks; merge ovxr_session.py and _ovxr_presentation.py into the session owner, preserving prepare/start/publish/drain. Suggested PR breakdown: renderer/CPU publication with native validation; Capture/reset integration with graph tests; entrypoints/docs with workflow tests; SO101 with preset/calibration tests; headless Kit with visualizer tests.
  • API: API changes look good.
  • Compatibility and deprecation: Breaking changes: run_teleoperation rebinds its cleanup argument, SO101 changes established reset/home defaults, and graph execution overwrites Capture simulation time; preserve old behavior through complete transitions where required.
  • Implementation: Operator resets can resume retargeting with pre-reset baselines because reset() skips an already-observed control event; notify retargeters after resetting the environment.
  • Style consistency: Clone-routing documentation incorrectly says a lease changes every renderer’s default; describe renderer-local OVStage selection.
  • Test quality: CPU publication assertions use prefilled outputs and a success-only transform mock, so broken transform production still passes; use SceneDataProvider for independent value assertions.

Needs rework. Posted 20 inline findings.

The full PR diff was reviewed; some supplemental surrounding file context was omitted.

Automated review; human maintainers own approval decisions.

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Needs attention: Design and architecture (3), Scope and over-engineering (4), Compatibility and deprecation (5), Implementation (4), Style consistency (11), Test quality (5). Other reviewed categories look good.

  • PR description: Checked live/replay scope and validation; condense repeated combined-stack results and benchmark per-step transform synchronization using baseline/candidate revisions, command, hardware, backend, environment count, warmup, and latency.
  • Design and architecture: Failed seals and unguarded Stage commits undermine exclusive publication ownership; poison the pair after sealing failure and prevent launcher context release until Capture input drain is proved.
  • Scope and over-engineering: Minimal implementation: Extend OVRTXRenderer; keep render_lease.py’s public contract but merge its controller into that owner; keep ov_libraries_xr.py and runtime-independent ovxr_cfg.py; merge ovxr_session.py and _ovxr_presentation.py into the session; merge _ovxr_launch.py into existing entrypoint preparation; bind invariant anchor attributes once. Essential flow: resolve, lease, bind, publish, input, drain. Suggested PR breakdown: (1) renderer lifecycle, native tests; (2) live/replay reset integration, graph tests; (3) SO101 configuration, task tests; (4) Kit XR updates, visualizer tests.
  • API: API changes look good.
  • Compatibility and deprecation: Breaking changes: run_teleoperation argument binding, SO101 clutch calibration and reset distribution, existing-anchor preservation, and cached-array return ownership lack functional deprecation bridges; retain old behavior with targeted warnings, migration guidance, removal timing, and transition coverage.
  • Implementation: Replay cleanup can replace the primary failure; preserve it with cleanup notes, retain pre-start cancellation, keep failed context retirement retryable, and supply or reject CUSTOM anchor head-pose input.
  • Style consistency: Renderer-selection and publication documentation contradict the implementation; correct those contracts, repair class references and clock descriptions, and remove redundant prose while restoring prescribed member ordering and direct field access.
  • Test quality: Added tests include self-derived rotation expectations, inactive parameter axes, unreachable Kit configurations, and direct validation disconnected from the changed caller; use independent expectations, exercise owning boundaries, consolidate cases, and add OV markers.

Needs rework. Posted 32 inline findings.

The full PR diff was reviewed; some supplemental surrounding file context was omitted.

Automated review; human maintainers own approval decisions.

Comment thread source/isaaclab_ov/isaaclab_ov/stage.py Resolved
@hougantc-nvda

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Nov 13, 2026
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch from bba22cd to 10b0b36 Compare November 16, 2026 14:33
@hougantc-nvda

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Nov 16, 2026
hougantc-nvda added a commit to hougantc-nvda/IsaacLab that referenced this pull request Nov 16, 2026
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch from 10b0b36 to 2251f68 Compare November 16, 2026 17:13
hougantc-nvda added a commit to hougantc-nvda/IsaacLab that referenced this pull request Nov 16, 2026

@rwiltz rwiltz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer pass on this layer. The kitless path's cleanup and failure semantics are carefully reasoned. The main ask is structural: pick the kitless/Kit strategy once at the boundary, instead of branching through the entrypoints, XrAnchorManager and TeleopSessionLifecycle. Doing it now keeps #8789 to #8791 from adding more branches to the same files. Comments prefixed nit: are optional.

lifecycle = self
completed_caches: weakref.WeakSet[ExecutionCache] = weakref.WeakSet()

class _HostResetOutputCombiner(OutputCombiner):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This subclass overrides Isaac Teleop's execute_pipeline_with_cache, imports ExecutionCache/GraphTime from retargeter_core_types, tracks cache identity in a WeakSet, and reuses GraphTime.real_time_ns as a submission fence. It runs for every Isaac Teleop user (Kit CloudXR and standalone too), not just kitless, and depends on internals we don't own. Please get a public host-reset entry point upstream, for example TeleopSession.notify_host_reset() or step(host_reset=True) ordered against queued submissions, and pin to that release. Then this combiner and the cutoff/frame-id fences can go.

signal.signal(signum, handler)


def _close_teleop_resources(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both entrypoints now orchestrate OVXR internals directly: about 29 presentation is None/using_ovxr branches here, about 20 in replay, and 17 reach-throughs to presentation.session.*. Teardown and signal protocols are duplicated and have already drifted (see replay L1696). Please pick the frame strategy once, as a small presenter protocol with Kit/null and OVXR implementations (idle(), after_step(), admit_reset(), advance_input(), should_stop, pace()), plus one shared close_teleop_owners()/install_shutdown_handlers() used by both agents. The loops then lose the is None branches, and the unit tests need far less monkeypatching of module globals.

@maxkra15 maxkra15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we carry the nonzero startup exit from #9030 into OVXR? If environment or input-device creation fails, the current CLI logs the error and exits successfully.

@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch from 4b42440 to fa38bea Compare November 18, 2026 03:16
hougantc-nvda added a commit to hougantc-nvda/IsaacLab that referenced this pull request Nov 18, 2026
hougantc-nvda added a commit to hougantc-nvda/IsaacLab that referenced this pull request Nov 18, 2026
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch from fa38bea to b56d857 Compare November 18, 2026 03:44
@hougantc-nvda

Copy link
Copy Markdown
Contributor Author

Could we carry the nonzero startup exit from #9030 into OVXR?

Done in this PR. The live entrypoint now re-raises environment and teleop-device creation failures after cleanup; the cleanup path keeps the first error. So isaaclab teleop run exits non-zero both with and without --ovxr.

It is the same change #9030 makes at those two handlers, so whichever lands second resolves trivially.

@hougantc-nvda

Copy link
Copy Markdown
Contributor Author

@matthewtrepte No deprecation warning for now. We'll keep both --xr (Kit) and --ovxr (kitless) until Kit is fully deprecated.

hougantc-nvda added a commit to hougantc-nvda/IsaacLab that referenced this pull request Nov 18, 2026
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch 2 times, most recently from 6cbc326 to 6bac479 Compare November 18, 2026 06:28
hougantc-nvda added a commit to hougantc-nvda/IsaacLab that referenced this pull request Nov 18, 2026
Lets an OVXR headset session render an Isaac Lab simulation without Kit.
The session's renderer, OvxrRenderer, implements BaseRenderer in
isaaclab_teleop. It lends its own OVRTX engine and OVStage to Isaac
Capture's render worker and publishes complete scene updates to them.
Adds live headset teleoperation, demonstration recording and replay
without Kit, using the OVXR renderer and session. The SO101 arm supports
the kitless path, including CPU OvPhysX. Live, recording and MCAP replay
share launch validation, presentation updates and shutdown helpers.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants