Repository navigation
Add kitless teleoperation and replay with SO101 support - #8788
hougantc-nvda wants to merge 2 commits into
Conversation
e933899 to
4d7c54f
Compare
fe8d8d2 to
8d56932
Compare
293a54c to
bb8944d
Compare
|
@greptile Please review the latest changes in this PR. |
[Medium impact] The optional OVXR route connects teleoperation, simulation, and an external render worker. Update the publication-failure test double before merging; both parameter cases otherwise fail at the new material lookup. Findings
SummaryAdds kitless live teleoperation and MCAP replay, SO101 support, and ordered XR cleanup. Since the last review:
Previous unnumbered threads were rechecked without posting duplicates. The callback uses Diagram%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[Task changes material] --> B[Mark channel dirty]
B --> C[Check whether publication is needed]
C --> D[Reserve scene ordinal]
D --> E[Write materials and scene]
E --> F[Wait for material reads]
F --> G[Seal scene ordinal]
G --> H[XR worker renders]
Reviews (4) · Last reviewed commit: "Merge updated prerequisite for #8788" · Reviewed by Greptile |
bb8944d to
62a1200
Compare
72a3b68 to
01fc065
Compare
|
@greptile please review this PR again at the latest head ( |
There was a problem hiding this comment.
Isaac Lab Review Bot
Needs attention: Scope and over-engineering (4), Compatibility and deprecation (3), Implementation (6), Style consistency (3), Test quality (4). Other reviewed categories look good.
- PR description: Describe the included lease/adapter, consolidate validation totals, and benchmark publication/readback latency with baseline/candidate revisions, command/workload, hardware, backend, environment count, warmup, and metric; distinguish combined-stack qualification from this head.
- Design and architecture: Architecture looks good.
- Scope and over-engineering: Minimal implementation: Extend OVRTXRenderer and OvLibrariesXrSession; keep render_lease.py for thread-affine borrowing, ov_libraries_xr.py for Capture integration, ovxr_cfg.py for pre-runtime configuration, and _ovxr_launch.py for shared checks; merge ovxr_session.py and _ovxr_presentation.py into the session owner, preserving prepare/start/publish/drain. Suggested PR breakdown: renderer/CPU publication with native validation; Capture/reset integration with graph tests; entrypoints/docs with workflow tests; SO101 with preset/calibration tests; headless Kit with visualizer tests.
- API: API changes look good.
- Compatibility and deprecation: Breaking changes: run_teleoperation rebinds its cleanup argument, SO101 changes established reset/home defaults, and graph execution overwrites Capture simulation time; preserve old behavior through complete transitions where required.
- Implementation: Operator resets can resume retargeting with pre-reset baselines because reset() skips an already-observed control event; notify retargeters after resetting the environment.
- Style consistency: Clone-routing documentation incorrectly says a lease changes every renderer’s default; describe renderer-local OVStage selection.
- Test quality: CPU publication assertions use prefilled outputs and a success-only transform mock, so broken transform production still passes; use SceneDataProvider for independent value assertions.
Needs rework. Posted 20 inline findings.
The full PR diff was reviewed; some supplemental surrounding file context was omitted.
Automated review; human maintainers own approval decisions.
There was a problem hiding this comment.
Isaac Lab Review Bot
Needs attention: Design and architecture (3), Scope and over-engineering (4), Compatibility and deprecation (5), Implementation (4), Style consistency (11), Test quality (5). Other reviewed categories look good.
- PR description: Checked live/replay scope and validation; condense repeated combined-stack results and benchmark per-step transform synchronization using baseline/candidate revisions, command, hardware, backend, environment count, warmup, and latency.
- Design and architecture: Failed seals and unguarded Stage commits undermine exclusive publication ownership; poison the pair after sealing failure and prevent launcher context release until Capture input drain is proved.
- Scope and over-engineering: Minimal implementation: Extend OVRTXRenderer; keep render_lease.py’s public contract but merge its controller into that owner; keep ov_libraries_xr.py and runtime-independent ovxr_cfg.py; merge ovxr_session.py and _ovxr_presentation.py into the session; merge _ovxr_launch.py into existing entrypoint preparation; bind invariant anchor attributes once. Essential flow: resolve, lease, bind, publish, input, drain. Suggested PR breakdown: (1) renderer lifecycle, native tests; (2) live/replay reset integration, graph tests; (3) SO101 configuration, task tests; (4) Kit XR updates, visualizer tests.
- API: API changes look good.
- Compatibility and deprecation: Breaking changes: run_teleoperation argument binding, SO101 clutch calibration and reset distribution, existing-anchor preservation, and cached-array return ownership lack functional deprecation bridges; retain old behavior with targeted warnings, migration guidance, removal timing, and transition coverage.
- Implementation: Replay cleanup can replace the primary failure; preserve it with cleanup notes, retain pre-start cancellation, keep failed context retirement retryable, and supply or reject CUSTOM anchor head-pose input.
- Style consistency: Renderer-selection and publication documentation contradict the implementation; correct those contracts, repair class references and clock descriptions, and remove redundant prose while restoring prescribed member ordering and direct field access.
- Test quality: Added tests include self-derived rotation expectations, inactive parameter axes, unreachable Kit configurations, and direct validation disconnected from the changed caller; use independent expectations, exercise owning boundaries, consolidate cases, and add OV markers.
Needs rework. Posted 32 inline findings.
The full PR diff was reviewed; some supplemental surrounding file context was omitted.
Automated review; human maintainers own approval decisions.
|
run-ci |
bba22cd to
10b0b36
Compare
|
run-ci |
10b0b36 to
2251f68
Compare
rwiltz
left a comment
There was a problem hiding this comment.
Maintainer pass on this layer. The kitless path's cleanup and failure semantics are carefully reasoned. The main ask is structural: pick the kitless/Kit strategy once at the boundary, instead of branching through the entrypoints, XrAnchorManager and TeleopSessionLifecycle. Doing it now keeps #8789 to #8791 from adding more branches to the same files. Comments prefixed nit: are optional.
| lifecycle = self | ||
| completed_caches: weakref.WeakSet[ExecutionCache] = weakref.WeakSet() | ||
|
|
||
| class _HostResetOutputCombiner(OutputCombiner): |
There was a problem hiding this comment.
This subclass overrides Isaac Teleop's execute_pipeline_with_cache, imports ExecutionCache/GraphTime from retargeter_core_types, tracks cache identity in a WeakSet, and reuses GraphTime.real_time_ns as a submission fence. It runs for every Isaac Teleop user (Kit CloudXR and standalone too), not just kitless, and depends on internals we don't own. Please get a public host-reset entry point upstream, for example TeleopSession.notify_host_reset() or step(host_reset=True) ordered against queued submissions, and pin to that release. Then this combiner and the cutoff/frame-id fences can go.
| signal.signal(signum, handler) | ||
|
|
||
|
|
||
| def _close_teleop_resources( |
There was a problem hiding this comment.
Both entrypoints now orchestrate OVXR internals directly: about 29 presentation is None/using_ovxr branches here, about 20 in replay, and 17 reach-throughs to presentation.session.*. Teardown and signal protocols are duplicated and have already drifted (see replay L1696). Please pick the frame strategy once, as a small presenter protocol with Kit/null and OVXR implementations (idle(), after_step(), admit_reset(), advance_input(), should_stop, pace()), plus one shared close_teleop_owners()/install_shutdown_handlers() used by both agents. The loops then lose the is None branches, and the unit tests need far less monkeypatching of module globals.
4b42440 to
fa38bea
Compare
fa38bea to
b56d857
Compare
Done in this PR. The live entrypoint now re-raises environment and teleop-device creation failures after cleanup; the cleanup path keeps the first error. So It is the same change #9030 makes at those two handlers, so whichever lands second resolves trivially. |
|
@matthewtrepte No deprecation warning for now. We'll keep both |
6cbc326 to
6bac479
Compare
Lets an OVXR headset session render an Isaac Lab simulation without Kit. The session's renderer, OvxrRenderer, implements BaseRenderer in isaaclab_teleop. It lends its own OVRTX engine and OVStage to Isaac Capture's render worker and publishes complete scene updates to them.
Adds live headset teleoperation, demonstration recording and replay without Kit, using the OVXR renderer and session. The SO101 arm supports the kitless path, including CPU OvPhysX. Live, recording and MCAP replay share launch validation, presentation updates and shutdown helpers.
Description
Adds live headset teleoperation, demonstration recording and replay without Kit, using the OVXR renderer and session from #8787. The SO101 arm supports the kitless path, including CPU OvPhysX. Live, recording and MCAP replay share launch validation, presentation updates and shutdown helpers.
Review stack · 2/5: #8787 → #8788 → #8789 → #8790 → #8791.
Review only this layer on the fork mirror. The upstream PRs target
main; please leave comments here. This layer depends on #8787.--xr; kitless XR uses--ovxr, with one environment and kitless physics.--ovxr_session_mode {ar,vr}selects the OpenXR session mode.record_demos.pyaccepts the same options and records through the same OVXR session as live teleoperation.OvxrPresentationowns the presentation path shared by live, recording and replay, including paused updates that leave physics unchanged.--ovxr_step_hzcaps the host-control rate with--ovxr, while XR rendering runs independently. A task's anchor can move the headset view with the robot; an OVXR session uses the newExternalXrAnchor.IsaacTeleopDevice.notify_reset()reports an environment reset to the input pipeline.TeleopSessionLifecycledelivers it by overriding the root pipeline's execution methods, for every Isaac Teleop session;test_host_reset_pipeline.pyruns the override against the real graph and asynchronous runner.--start_on_replayfor recordings without START. A recorded RESET resets the replay and waits for the next START instead of ending it. Without Kit, replay retargets synchronously, one recorded frame per step. A kitless backend also replays headless without--ovxrand reports each run's outcome without render timing; in that mode a task that setstarget_frame_prim_pathis rejected with an error that points to--ovxror Kit.replay_demos.pyreplays HDF5 demonstrations on a kitless backend selected with--physics, also when Isaac Sim is installed.IsaacTeleopCfg.prepare_environmenthook. The live, replay and recording launchers run it once before creating simulation resources.--ovxr.For existing Kit users (each is in the changelog):
notify_reset: after an operator reset, and when environment 0 terminates or is truncated during a step.mainit restarts the input session, which replays the recording from its start. It rebuilds the simulation withenv.sim.reset()only for Isaac Sim PhysX, and it raises when the CloudXR runtime it launched fails to stop.run_teleoperationowns its cleanup: it no longer takes acleanupstack and requires the keyword-onlyuse_isaac_teleop.replay_demos.py --validate_statesno longer raises a shape mismatch, and the recorder logs its instructions when there is no Kit window.prepare_environmenthook.XrAnchorSynchronizercomputes the same anchor pose through a policy class it shares with OVXR, and the helper that looks up an existing anchor prim reads the stage throughget_current_stage().Outside
source/isaaclab_teleop/, each change sits with its owner:isaaclab_ov: three optional contact settings onOvPhysxCfg, which keep the stage's values when unset.isaaclab_newton:capture_graphcaptures on a non-blocking stream, so graph capture no longer conflicts with concurrent renderer or input uploads. Torch work that the caller runs on Warp's current stream, as MDP2 does, moves to the capture stream with it and is still recorded.isaaclab_tasks: the SO101 IK-Abs task gains theovphysxpreset, the tuned OvPhysX configuration forphysxwithout Kit, the clutch home and the reset hook;test/core/test_preset_cli.pygains two public-selector tests.Dependencies: #8787 and Isaac Capture 1.7.118rc1 through the locked
ovxrextra. See the runtime guide and the installation guide.Type of change
Testing
Tested as a stack on
mainat6980e3d088a.--ovxrexits 1 with the error that namesisaac_teleop.target_frame_prim_path.Validation of the full stack (#8787 to #8791) without Kit, on one GPU (RTX 5090 Laptop) and on two (2x RTX 6000 Ada):
The changelog gate passes on every PR of the stack, and ruff passes on every file the stack changes.
Unit tests, one file per process: the teleop tests pass on every PR (467, 631, 672, 755 and 815 tests on Add a teleop-owned OVXR renderer and session adapter #8787 to Add GPU selection for OVXR rendering and scene updates #8791), with
isaaclab_ov's OVRTX renderer tests (164), the MDP2 and Newton capture tests (76) and the task, Mimic and OvPhysX tests (84, 183, 185 and 185 on Add kitless teleoperation and replay with SO101 support #8788 to Add GPU selection for OVXR rendering and scene updates #8791). The test files that need Kit were not run in this environment (test_teleop_scripts_smoke.py,test_openxr_device_constructors.py,test_oxr_device.py,test_retargeters.py,test_xr_camera_feed_kit_scene_ui_integration.py).The native tests on the real libraries (the owner-transfer harness and marker visibility) pass on every PR that has them. With two GPUs,
test_ovxr_renderer.pypasses 98 tests with none skipped,test_check_cuda_peer.pypasses 5, and the native transfer harness passes 6 of 6.Live sessions with an emulated headset (IWER) and the CloudXR runtime launched by the session, on the two-GPU machine. Every session exits 0 with no dropped frames, with camera feeds, mirror and markers where the task has them:
On the one-GPU machine, GR1T2 on CPU OvPhysX and on Newton (camera feed, mirror, markers) and SO101 on OvPhysX start, stream both eyes and shut down cleanly.
record_demos.py --ovxrwrites HDF5 and MCAP on one GPU and with physics and rendering on separate GPUs. Replaying the HDF5 of the split-GPU recording matches all 1394 recorded states exactly.Ctrl-C during recording, during replay and while renderer attach is held, a forced render-worker failure and forced start-up failures all exit with every native resource closed on the main thread and no process or port left.
MCAP replay of a GR1T2 pick-and-place capture with
--start_on_replay --num_success_steps 1000 --require_hand_replay_dataexits 0 on Newton and on CPU OvPhysX (also when selected as--physics physx), with 455 active steps and 457 recorded hand frames. This checks that the replay consumes the whole recording; it is not a task-success check.Not run: a physical headset (the sessions above use the emulator), a Kit XR session and Kit
record_demos.py --xr. The Kit tests and the docs build are left to CI.Known issues: renderer startup stalled twice in native
ovrtx.attach_ovstageduring development; identical retries passed, and the root cause is unknown. CloudXR logs acuMemGetInfoerror when a client disconnects.Checklist