Skip to content

Add a neural physics backend for world models and camera frame history - #9109

Draft
temiroff wants to merge 11 commits into
isaac-sim:mainfrom
temiroff:atemirov/world-model-backend
Draft

temiroff wants to merge 11 commits into
isaac-sim:mainfrom
temiroff:atemirov/world-model-backend

Conversation

@temiroff

@temiroff temiroff commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds isaaclab_neural, a neural physics backend in which a learned world model replaces the physics engine and renderer of a manager-based task, and an optional frame history on cameras. This is the backend of #9087 by Yun Liu, split out so it can be reviewed without a specific model; a Cosmos3 client and a task follow in separate PRs.

What is added:

  • NeuralManager and NeuralCfg (backend_name = "neural"), named like the other physics backends (PhysxManager, NewtonManager). The manager owns a WorldModel, selected with NeuralCfg(model_cfg=...) as a Newton solver is with solver_cfg, and each environment's joint state. The model predicts a chunk of frames from a chunk of joint targets: the manager buffers one target per step and calls the model on the last step of the chunk. The joint state is the reset record's state at reset and the last commanded target after that; the model never outputs joint state.
  • A virtual Articulation that shares the manager's joint state and targets, so joint_pos observations, JointPositionAction and the standard reset events work unchanged. Joints and limits come from the USD asset. Anything that needs a simulated body raises NotImplementedError.
  • NeuralRenderer / NeuralRendererCfg, which copy the predicted frames into cameras. A camera selects one of the model's views with NeuralRendererCfg.view, and NeuralRendererCfg.outputs declares the layout of each output (RGB by default).
  • The WorldModel interface: spec, reset, write, step, close. Before each step the backend writes the inputs a model lists in WorldModelSpec.inputs, named by WorldModelAttribute (the chunk's joint targets and the joint state before it), and rejects a model that needs one it does not provide. A model names its views and outputs in its spec (WorldModelSpec.views, WorldModelSpec.outputs, such as "rgb" or "depth"), and frames pass as {view: {output: tensor}}.
  • Reset records (start frames, joint state, prompt, split) that episodes start from. A record's start image is a path, one per view, or one per view and output (.npy for outputs other than RGB). envs_per_reset_record lets consecutive environments share a record, for group-relative algorithms.
  • at_chunk_boundary, which makes a termination fire only where environments can reset, and checks the episode timing once.
  • In isaaclab: CameraCfg.frame_history_length and CameraData.frame_history / frame_history_mask. The history keeps the most recent captures of every requested output, keyed like CameraData.output. The length defaults to 0, so existing cameras are unchanged. A renderer that predicts several frames per update fills the history itself and sets provides_frame_history on its configuration.

Design points:

  • No host reads in steps or resets. Joint state, targets, frames and the start frames of the records live on the simulation device, and masks are applied with torch.where. A reset shows the record's start frames at once; the step that ends the next chunk, which calls the model anyway, reads which environments reset (the prompts are host strings) and restarts them in the model. The tests check steps and a reset with torch.cuda.set_sync_debug_mode("error").
  • Resets only between chunks. The model predicts a whole chunk at once, so a reset inside a chunk raises when the chunk ends. The README explains how to set the episode length, decimation and terminations.
  • Cameras bind once. Each camera binds a reader to the frames of its view and outputs when its render data is created and is checked against the model there; read_output only copies the environments whose frames changed.
  • Model-agnostic. Any model that predicts camera outputs from joint targets plugs in by implementing WorldModel; each maps the targets to its own action space. The tests use a fake model with RGB and depth: no weights and no simulation app.

Suggested review order: the camera frame history (camera.py, camera_cfg.py, camera_data.py), world_model.py, frame_buffer.py, physics/neural_manager.py, the articulation, the renderer, mdp/terminations.py, then docs/README.md.

Type of change

  • New feature (non-breaking change which adds functionality)

Validation

  • source/isaaclab_neural/test and source/isaaclab/test/sensors/test_camera_frame_history.py pass on Windows (CPU and CUDA) and on Linux.
  • On Linux, a Cosmos3 client and a task built on this branch ran end to end with a real checkpoint: a replay, a two-environment rollout through a reset and a three-iteration rsl_rl PPO run.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with uv run isaaclab --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

A learned world model can replace the physics engine of a manager-based task. WorldModelManager is a physics backend that keeps each environment's joint state on the simulation device and calls a WorldModel once per chunk of frames; WorldModelRenderer copies the generated frames into cameras; a virtual Articulation shares the manager's joint state and starts each episode from a reset record, which becomes the default joint position, so standard joint observations, actions and reset events work unchanged. at_chunk_boundary makes terminations fire only where environments can reset.

Steps and resets do not read the device. A reset takes an environment mask, samples records on the device and shows the record's start image at once; the step that ends the next chunk, which waits for the model anyway, restarts the model's environments from their records.

Cameras gain an optional frame history (CameraCfg.frame_history_length, CameraData.frame_history and its mask). A renderer that predicts several frames per update fills it itself; otherwise the camera pushes each capture, without waiting for the GPU.

The tests drive the backend with a fake world model.

Co-authored-by: KumoLiu <55491388+KumoLiu@users.noreply.github.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team infrastructure labels Nov 19, 2026
class BaseRenderer(ABC):
"""Abstract base class for renderer implementations."""

provides_frame_history: ClassVar[bool] = False

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Abstract interfaces shouldn't carry any state.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. BaseRenderer is back to what it is on main. The world-model renderer's configuration now declares provides_frame_history, and the camera reads it from renderer_cfg as it reads async_rendering. Fixed in c60a71f.

…enderer

The abstract renderer no longer carries a provides_frame_history attribute. A renderer that fills the camera's frame history itself declares it on its configuration, which the camera reads as it reads async_rendering; the world-model renderer configuration sets it.
WorldModel stored cfg, num_envs and device in __init__, so implementations reused state through inheritance. It now only declares the contract; each implementation keeps what it needs of class_type(cfg, num_envs, device), and the manager and renderer share the rest by holding the model.
… new tensor

read_output concatenated the newest frame with an alpha channel into a new tensor whenever frames arrived. It now writes the RGB channels and the alpha channel straight into the camera's rgba buffer.
The renderer reached through the simulation into the manager's frame tensors and counters, and a model could predict only one camera view. A model now names its views in WorldModelSpec.views, step() returns frames per view, and reset() takes start images per view; a reset record's image is a path, or {view: path} for a model with several views.

The manager keeps one FrameBuffer per view and publishes into it after a chunk or a reset. In create_render_data() the renderer checks that the simulation runs the world-model backend and that the camera matches the model, then binds a FrameReader for the view in WorldModelRendererCfg.view (None for a model with one view). read_output() only asks that reader for the environments whose frames changed.
return self._intrinsic_matrices

@property
def frame_history(self) -> ProxyArray | None:

@pbarejko pbarejko Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assumes that the frame history will only have single modality - rgb. This could change in the future to support multiple modalities.

I would recommend to switch this to:

def frame_history(self, output: str) -> ProxyArray

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0de412e: frame_history is now dict[str, ProxyArray] keyed like output, with one history per requested data type.

"""

frame_history_length: int = 0
"""Number of most recent RGB captures kept in :attr:`CameraData.frame_history`. Defaults to 0 (disabled).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to have frame history only for RGB in the future? This seems quite constraining, it would be better to leave it generic.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0de412e: the history keeps every output in data_types, not only RGB.

)
if self.frame_history_length < 0:
raise ValueError(f"frame_history_length must be 0 (off) or positive, got {self.frame_history_length}")
if self.frame_history_length and "rgb" not in self.data_types:

@pbarejko pbarejko Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make it more generalized not only working with RGB? This seems like a world model feature what it can produce.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0de412e: the RGB requirement is gone; every requested output gets a history.

cfg: WorldModelRendererCfg

def supported_output_types(self) -> dict[RenderBufferKind, RenderBufferSpec]:
return self.cfg.supported_output_types()

@pbarejko pbarejko Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have a look at this https://github.com/isaac-sim/IsaacLab/pull/9109/changes#r4646006574 they are somehow are connected.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0de412e: NeuralRendererCfg.outputs declares the output layouts (RGB by default), and each camera output is checked against the model's WorldModelSpec.outputs when the camera binds. They can't come from the model itself, because the camera config is validated before the model is built.

# Output image buffers — allocated in allocate()
self._output: dict[str, ProxyArray] | None = None

self._frame_history: ProxyArray | None = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably should be a dictionary str -> ProxyArray

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0de412e: it is a dict keyed by output name, like output.

import torch


class FrameBuffer:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The frame buffer doesn't seem to belong to physics sub-module.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to the package top level in 4c52b62 (isaaclab_neural/frame_buffer.py): the manager writes it and the renderer reads it.

"""Initial condition of an episode: a start image, the joint state at that image and a task prompt."""

image: Path | dict[str, Path]
"""RGB start image, or one per view for a model that predicts several views."""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make it aov agnostic?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0de412e: a reset record's image can be {view: {output: path}}, with .npy for outputs other than RGB.

Args:
env_ids: Environments to restart.
images: For each of the model's views, the start RGB image of each selected environment, shape

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make it modality agnostic? this is a base class.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0de412e: reset() takes {view: {output: tensor}}, and the spec lists the outputs (WorldModelSpec.outputs).

start_joint_state: Joint positions before the chunk, shape ``(N, num_joints)``.
Returns:
For each of the model's views, RGB frames, shape ``(N, chunk_length, H, W, 3)``, ``uint8``, on the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an we make it modality agnostic? this is a base class.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0de412e: step() returns {view: {output: tensor}}.



class WorldModel(ABC):
"""Interface of a learned model that predicts camera frames from joint targets.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

predicts camera frames from joint target

is this true for every world model that we can and can not think of?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. Reworded in 0de412e: the interface covers action-conditioned models that predict camera outputs, each mapping the joint targets to its own action space. Models that predict state, and policies, are out of scope.

"""

@abstractmethod
def step(self, joint_targets: torch.Tensor, start_joint_state: torch.Tensor) -> dict[str, torch.Tensor]:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This API is quite constraining, because you bind this to very specific use case

  • joints in
  • frames out

can we replace it with

write(str: name, value)

Then in your implementation you can

enum Attributes:
   JOINT_TARGETS
   START_JOINT_STATE

world_model.write(Attributes.JOINT_TARGETS, joint_values)
world_model.write(Attributes.START_JOINT_STATE, start_joint_state_values)

This design allows you to extend the world model for any attribute, with implementation driven names.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e4d955c: the model lists the inputs it needs in WorldModelSpec.inputs, named by WorldModelAttribute (JOINT_TARGETS and START_JOINT_STATE for now), and the manager writes each with write(name, value) before step(). A new input only needs the backend to provide it; a model that needs one it does not provide is rejected at bind. reset() stays explicit, since start frames, prompts and seeds are per episode.

…ackends

Each physics backend is named <Backend>Manager and <Backend>Cfg after its backend_name (PhysxManager, NewtonManager, OvPhysxManager), so the neural backend's manager, configuration and renderer become NeuralManager, NeuralCfg, NeuralRenderer and NeuralRendererCfg, with renderer_type "neural". One backend serves every world model, which NeuralCfg.model_cfg selects as NewtonCfg.solver_cfg selects a solver. WorldModel stays the name of the model interface.
The manager writes the predicted frames and the renderer reads them, so the frame buffer sits next to world_model.py instead of in physics.
…edict any of them

The frame history and the world-model interface assumed RGB. CameraData.frame_history is now keyed by output like CameraData.output and keeps every requested data type, so depth or segmentation history works with any renderer. A world model names its outputs in WorldModelSpec.outputs, step() returns and reset() takes {view: {output: tensor}}, and reset records may give {view: {output: path}}, with .npy arrays for outputs other than RGB. The manager keeps one FrameBuffer per view and output, laid out like the start frames, and NeuralRendererCfg.outputs declares the camera layouts, which the camera checks against the model when it binds, since the model is built after the camera configuration is checked. The WorldModel docstring now says which models it covers.
@temiroff temiroff changed the title Add a world-model physics backend and camera frame history Add a neural physics backend for world models and camera frame history Nov 20, 2026
WorldModel.step took joint targets and the start joint state, which bound every model to those two inputs. A model now lists the inputs it needs in WorldModelSpec.inputs, from the WorldModelAttribute names the backend provides, and the manager writes each with write(name, value) before step(), which predicts from what was written. A new input only needs the backend to provide it; a model that needs one it does not provide is rejected when it binds. reset() keeps its start frames, prompts and seeds, which are per episode rather than per step.
WorldModelSpec.max_envs was declared but only the Cosmos3 service enforced it, when its session opened. The manager now checks it when the articulation binds, so any model's limit fails before the first step with the number of environments to change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants