Repository navigation
Conversation
A learned world model can replace the physics engine of a manager-based task. WorldModelManager is a physics backend that keeps each environment's joint state on the simulation device and calls a WorldModel once per chunk of frames; WorldModelRenderer copies the generated frames into cameras; a virtual Articulation shares the manager's joint state and starts each episode from a reset record, which becomes the default joint position, so standard joint observations, actions and reset events work unchanged. at_chunk_boundary makes terminations fire only where environments can reset. Steps and resets do not read the device. A reset takes an environment mask, samples records on the device and shows the record's start image at once; the step that ends the next chunk, which waits for the model anyway, restarts the model's environments from their records. Cameras gain an optional frame history (CameraCfg.frame_history_length, CameraData.frame_history and its mask). A renderer that predicts several frames per update fills it itself; otherwise the camera pushes each capture, without waiting for the GPU. The tests drive the backend with a fake world model. Co-authored-by: KumoLiu <55491388+KumoLiu@users.noreply.github.com>
| class BaseRenderer(ABC): | ||
| """Abstract base class for renderer implementations.""" | ||
|
|
||
| provides_frame_history: ClassVar[bool] = False |
There was a problem hiding this comment.
Abstract interfaces shouldn't carry any state.
There was a problem hiding this comment.
Agreed. BaseRenderer is back to what it is on main. The world-model renderer's configuration now declares provides_frame_history, and the camera reads it from renderer_cfg as it reads async_rendering. Fixed in c60a71f.
…enderer The abstract renderer no longer carries a provides_frame_history attribute. A renderer that fills the camera's frame history itself declares it on its configuration, which the camera reads as it reads async_rendering; the world-model renderer configuration sets it.
WorldModel stored cfg, num_envs and device in __init__, so implementations reused state through inheritance. It now only declares the contract; each implementation keeps what it needs of class_type(cfg, num_envs, device), and the manager and renderer share the rest by holding the model.
… new tensor read_output concatenated the newest frame with an alpha channel into a new tensor whenever frames arrived. It now writes the RGB channels and the alpha channel straight into the camera's rgba buffer.
The renderer reached through the simulation into the manager's frame tensors and counters, and a model could predict only one camera view. A model now names its views in WorldModelSpec.views, step() returns frames per view, and reset() takes start images per view; a reset record's image is a path, or {view: path} for a model with several views.
The manager keeps one FrameBuffer per view and publishes into it after a chunk or a reset. In create_render_data() the renderer checks that the simulation runs the world-model backend and that the camera matches the model, then binds a FrameReader for the view in WorldModelRendererCfg.view (None for a model with one view). read_output() only asks that reader for the environments whose frames changed.
| return self._intrinsic_matrices | ||
|
|
||
| @property | ||
| def frame_history(self) -> ProxyArray | None: |
There was a problem hiding this comment.
This assumes that the frame history will only have single modality - rgb. This could change in the future to support multiple modalities.
I would recommend to switch this to:
def frame_history(self, output: str) -> ProxyArrayThere was a problem hiding this comment.
Done in 0de412e: frame_history is now dict[str, ProxyArray] keyed like output, with one history per requested data type.
| """ | ||
|
|
||
| frame_history_length: int = 0 | ||
| """Number of most recent RGB captures kept in :attr:`CameraData.frame_history`. Defaults to 0 (disabled). |
There was a problem hiding this comment.
Do we want to have frame history only for RGB in the future? This seems quite constraining, it would be better to leave it generic.
There was a problem hiding this comment.
Done in 0de412e: the history keeps every output in data_types, not only RGB.
| ) | ||
| if self.frame_history_length < 0: | ||
| raise ValueError(f"frame_history_length must be 0 (off) or positive, got {self.frame_history_length}") | ||
| if self.frame_history_length and "rgb" not in self.data_types: |
There was a problem hiding this comment.
Can we make it more generalized not only working with RGB? This seems like a world model feature what it can produce.
There was a problem hiding this comment.
Done in 0de412e: the RGB requirement is gone; every requested output gets a history.
| cfg: WorldModelRendererCfg | ||
|
|
||
| def supported_output_types(self) -> dict[RenderBufferKind, RenderBufferSpec]: | ||
| return self.cfg.supported_output_types() |
There was a problem hiding this comment.
Have a look at this https://github.com/isaac-sim/IsaacLab/pull/9109/changes#r4646006574 they are somehow are connected.
There was a problem hiding this comment.
Done in 0de412e: NeuralRendererCfg.outputs declares the output layouts (RGB by default), and each camera output is checked against the model's WorldModelSpec.outputs when the camera binds. They can't come from the model itself, because the camera config is validated before the model is built.
| # Output image buffers — allocated in allocate() | ||
| self._output: dict[str, ProxyArray] | None = None | ||
|
|
||
| self._frame_history: ProxyArray | None = None |
There was a problem hiding this comment.
This probably should be a dictionary str -> ProxyArray
There was a problem hiding this comment.
Done in 0de412e: it is a dict keyed by output name, like output.
| import torch | ||
|
|
||
|
|
||
| class FrameBuffer: |
There was a problem hiding this comment.
The frame buffer doesn't seem to belong to physics sub-module.
There was a problem hiding this comment.
Moved to the package top level in 4c52b62 (isaaclab_neural/frame_buffer.py): the manager writes it and the renderer reads it.
| """Initial condition of an episode: a start image, the joint state at that image and a task prompt.""" | ||
|
|
||
| image: Path | dict[str, Path] | ||
| """RGB start image, or one per view for a model that predicts several views.""" |
There was a problem hiding this comment.
Can we make it aov agnostic?
There was a problem hiding this comment.
Done in 0de412e: a reset record's image can be {view: {output: path}}, with .npy for outputs other than RGB.
| Args: | ||
| env_ids: Environments to restart. | ||
| images: For each of the model's views, the start RGB image of each selected environment, shape |
There was a problem hiding this comment.
Can we make it modality agnostic? this is a base class.
There was a problem hiding this comment.
Done in 0de412e: reset() takes {view: {output: tensor}}, and the spec lists the outputs (WorldModelSpec.outputs).
| start_joint_state: Joint positions before the chunk, shape ``(N, num_joints)``. | ||
| Returns: | ||
| For each of the model's views, RGB frames, shape ``(N, chunk_length, H, W, 3)``, ``uint8``, on the |
There was a problem hiding this comment.
an we make it modality agnostic? this is a base class.
There was a problem hiding this comment.
Done in 0de412e: step() returns {view: {output: tensor}}.
|
|
||
|
|
||
| class WorldModel(ABC): | ||
| """Interface of a learned model that predicts camera frames from joint targets. |
There was a problem hiding this comment.
predicts camera frames from joint target
is this true for every world model that we can and can not think of?
There was a problem hiding this comment.
No. Reworded in 0de412e: the interface covers action-conditioned models that predict camera outputs, each mapping the joint targets to its own action space. Models that predict state, and policies, are out of scope.
| """ | ||
|
|
||
| @abstractmethod | ||
| def step(self, joint_targets: torch.Tensor, start_joint_state: torch.Tensor) -> dict[str, torch.Tensor]: |
There was a problem hiding this comment.
This API is quite constraining, because you bind this to very specific use case
- joints in
- frames out
can we replace it with
write(str: name, value)
Then in your implementation you can
enum Attributes:
JOINT_TARGETS
START_JOINT_STATE
world_model.write(Attributes.JOINT_TARGETS, joint_values)
world_model.write(Attributes.START_JOINT_STATE, start_joint_state_values)This design allows you to extend the world model for any attribute, with implementation driven names.
There was a problem hiding this comment.
Done in e4d955c: the model lists the inputs it needs in WorldModelSpec.inputs, named by WorldModelAttribute (JOINT_TARGETS and START_JOINT_STATE for now), and the manager writes each with write(name, value) before step(). A new input only needs the backend to provide it; a model that needs one it does not provide is rejected at bind. reset() stays explicit, since start frames, prompts and seeds are per episode.
…ackends Each physics backend is named <Backend>Manager and <Backend>Cfg after its backend_name (PhysxManager, NewtonManager, OvPhysxManager), so the neural backend's manager, configuration and renderer become NeuralManager, NeuralCfg, NeuralRenderer and NeuralRendererCfg, with renderer_type "neural". One backend serves every world model, which NeuralCfg.model_cfg selects as NewtonCfg.solver_cfg selects a solver. WorldModel stays the name of the model interface.
The manager writes the predicted frames and the renderer reads them, so the frame buffer sits next to world_model.py instead of in physics.
…edict any of them
The frame history and the world-model interface assumed RGB. CameraData.frame_history is now keyed by output like CameraData.output and keeps every requested data type, so depth or segmentation history works with any renderer. A world model names its outputs in WorldModelSpec.outputs, step() returns and reset() takes {view: {output: tensor}}, and reset records may give {view: {output: path}}, with .npy arrays for outputs other than RGB. The manager keeps one FrameBuffer per view and output, laid out like the start frames, and NeuralRendererCfg.outputs declares the camera layouts, which the camera checks against the model when it binds, since the model is built after the camera configuration is checked. The WorldModel docstring now says which models it covers.
WorldModel.step took joint targets and the start joint state, which bound every model to those two inputs. A model now lists the inputs it needs in WorldModelSpec.inputs, from the WorldModelAttribute names the backend provides, and the manager writes each with write(name, value) before step(), which predicts from what was written. A new input only needs the backend to provide it; a model that needs one it does not provide is rejected when it binds. reset() keeps its start frames, prompts and seeds, which are per episode rather than per step.
WorldModelSpec.max_envs was declared but only the Cosmos3 service enforced it, when its session opened. The manager now checks it when the articulation binds, so any model's limit fails before the first step with the number of environments to change.
Description
Adds
isaaclab_neural, a neural physics backend in which a learned world model replaces the physics engine and renderer of a manager-based task, and an optional frame history on cameras. This is the backend of #9087 by Yun Liu, split out so it can be reviewed without a specific model; a Cosmos3 client and a task follow in separate PRs.What is added:
NeuralManagerandNeuralCfg(backend_name = "neural"), named like the other physics backends (PhysxManager,NewtonManager). The manager owns aWorldModel, selected withNeuralCfg(model_cfg=...)as a Newton solver is withsolver_cfg, and each environment's joint state. The model predicts a chunk of frames from a chunk of joint targets: the manager buffers one target per step and calls the model on the last step of the chunk. The joint state is the reset record's state at reset and the last commanded target after that; the model never outputs joint state.Articulationthat shares the manager's joint state and targets, sojoint_posobservations,JointPositionActionand the standard reset events work unchanged. Joints and limits come from the USD asset. Anything that needs a simulated body raisesNotImplementedError.NeuralRenderer/NeuralRendererCfg, which copy the predicted frames into cameras. A camera selects one of the model's views withNeuralRendererCfg.view, andNeuralRendererCfg.outputsdeclares the layout of each output (RGB by default).WorldModelinterface:spec,reset,write,step,close. Before each step the backend writes the inputs a model lists inWorldModelSpec.inputs, named byWorldModelAttribute(the chunk's joint targets and the joint state before it), and rejects a model that needs one it does not provide. A model names its views and outputs in its spec (WorldModelSpec.views,WorldModelSpec.outputs, such as"rgb"or"depth"), and frames pass as{view: {output: tensor}}..npyfor outputs other than RGB).envs_per_reset_recordlets consecutive environments share a record, for group-relative algorithms.at_chunk_boundary, which makes a termination fire only where environments can reset, and checks the episode timing once.isaaclab:CameraCfg.frame_history_lengthandCameraData.frame_history/frame_history_mask. The history keeps the most recent captures of every requested output, keyed likeCameraData.output. The length defaults to 0, so existing cameras are unchanged. A renderer that predicts several frames per update fills the history itself and setsprovides_frame_historyon its configuration.Design points:
torch.where. A reset shows the record's start frames at once; the step that ends the next chunk, which calls the model anyway, reads which environments reset (the prompts are host strings) and restarts them in the model. The tests check steps and a reset withtorch.cuda.set_sync_debug_mode("error").read_outputonly copies the environments whose frames changed.WorldModel; each maps the targets to its own action space. The tests use a fake model with RGB and depth: no weights and no simulation app.Suggested review order: the camera frame history (
camera.py,camera_cfg.py,camera_data.py),world_model.py,frame_buffer.py,physics/neural_manager.py, the articulation, the renderer,mdp/terminations.py, thendocs/README.md.Type of change
Validation
source/isaaclab_neural/testandsource/isaaclab/test/sensors/test_camera_frame_history.pypass on Windows (CPU and CUDA) and on Linux.Checklist
pre-commitchecks withuv run isaaclab --formatsource/<pkg>/changelog.d/for every touched package (do not editCHANGELOG.rstor bump package versions — CI handles that)CONTRIBUTORS.mdor my name already exists there