Skip to content

Add isaaclab_fovi foveated camera sensors and active-gaze camera tasks - #9074

Open
nblauch wants to merge 71 commits into
isaac-sim:mainfrom
nblauch:feat/isaaclab-fovi-sensors
Open

nblauch wants to merge 71 commits into
isaac-sim:mainfrom
nblauch:feat/isaaclab-fovi-sensors

Conversation

@nblauch

@nblauch nblauch commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds isaaclab_fovi, a foveated camera for Isaac Lab built on the fovi library, active-gaze manager terms, and an active-gaze Cartpole task. A foveated camera samples the scene densely where it looks and sparsely in the periphery, on a retina-like grid, instead of on a uniform pixel grid. Try it with uv run --extra fovi isaaclab demo foveated-camera.

Camera (isaaclab_fovi.sensors, depends only on isaaclab)

  • FoveatedCamera / FoveatedCameraCfg work with the newton_warp, newton_optix, isaac_rtx and ovrtx renderers. Newton casts one ray per retinal sample; RTX and OVRTX render through a lens lookup texture.
  • The retina places its samples by visual angle inside the field of view of a physical camera: the one spawn describes (an ideal pinhole, or the OpenCV model in spawn.distortion), or any calibrated camera given as camera_model.
  • The gaze turns the retina relative to the physical camera's optical axis: camera.set_gaze_angles(angles_deg). The gaze stays relative to the mount when the mount moves, and get_world_poses reports the mount on every renderer. mask_to_sensor_fov zeroes samples outside the physical camera's image.
  • cmf_a_deg sets the strength of foveation (default 0.591 deg, an estimate from human data); a large value gives a matched non-foveated baseline.
  • With the default grid style the camera publishes an upright (num_envs, H, W, C) image, as ProxyArray values like the core Camera. Sample-set styles publish (num_envs, N, 1, C); they are experimental on RTX.
  • Optional two-layer pooling (pool_enabled, pool_mode); segmentation pools by modal label in its own dtype (pool_int_mode="modal"). With mask_to_sensor_fov, pooling leaves out the photoreceptors the physical camera does not see, so samples at the edge of its image are not darkened.
  • set_intrinsic_matrices leaves a foveated camera as configured and warns, as the core camera does for OpenCV lens models.

Active gaze

  • isaaclab_fovi.mdp: GazeActionCfg (two bounded gaze actions, with a fixed or tracked mount), gaze_state, and gaze_teacher_labels, a teacher that labels the gaze action toward a point on an asset (never a policy input). They run as Warp kernels with recorded launches.
  • IsaacContrib-Cartpole-Camera-Fovi in isaaclab_tasks.contrib.active_gaze: the stock Cartpole camera task with a 32×32 foveated retina, two gaze actions and a teacher that looks at the pole's tip, with the stock image and critic observations, CNN and PPO settings. It is registered when the fovi extra is installed. Its GazeSupervisedPPO adds a mean-only gaze supervision loss, and its GazeGaussianDistribution head holds the gaze actions' noise at a fixed 0.1: the gaze step clamps its actions, so a learned gaze noise meets no resistance and PPO's entropy bonus would grow it.

Docs and demo. A concise Foveated Camera page under Sensors: an overview figure of a retina behind a 180° fisheye lens among sim-ready YCB and Isaac Lab objects, path-traced with OVRTX (one fixation far to the side shows how samples outside the lens render black), the sampling math, a configuration example, and two interactive viewers (foveation-viewer.js): one draws the sampling grid for a pinhole and a fisheye camera as the style, CMF constant, resolution, field of view and gaze change, the other shows which photoreceptors each pooled sample averages. Their geometry matches fovi's SamplingCoords and GridPoolingLayer exactly. Also a section in Configure Rendering, the foveated-camera demo (examples/demos/foveated_camera.py, on the Demos page), the fovi extra in the installation extras table and the troubleshooting import table, the task in the environment browser, and API pages for isaaclab_fovi.mdp and isaaclab_fovi.sensors. The docs build passes with warnings as errors.

Simplification. nblauch#5 (by ooctipus, merged here) shares the camera setup across the Newton and RTX backends, records the sensor mask once per update and reuses it for every grid output (about 10% faster masked publication with three outputs on an RTX 5090), declares the Cartpole configs directly, and shortened the guide. It changes no behaviour: the resolved Cartpole env and agent configs are identical, and the RTX lens textures, grid outputs and teacher labels match byte for byte.

Dependencies. A new optional extra, fovi, installs this package, fovi>=2.6 (PyPI) and OpenEXR>=3 (lens textures for RTX); through fovi it also brings beartype, fastcore, plum-dispatch, pygeodesic, scikit-image and tifffile. No core runtime code changes. Outside the new package and the contrib task, the PR touches the root pyproject.toml (the extra) and its extras test, uv.lock, docker/Dockerfile.base and .github/workflows/build.yaml (CI installs the extra and runs the package's tests), the docs, and isaaclab/programs.py with the standalone-script cases in isaaclab/test/app/standalone_script_cases.py (the demo).

Companion. A draft IsaacLabTutorial PR adds an SO-101 vial-placement task with active gaze that builds on this package.

Type of change

  • New feature (non-breaking change which adds functionality)

Validation

  • The tests cover what isaaclab_fovi builds on Isaac Lab, so that a later Isaac Lab change that breaks it fails here; fovi's own behaviour is left to fovi's tests. 77 simulator-free test cases under source/isaaclab_fovi/test (how the camera reads Isaac Lab's camera configs, its gaze and mount handling and sensor mask, its rays, lens textures and published layout, its pooling of the camera's buffers and segmentation, and the gaze manager terms) and 5 under source/isaaclab_tasks/test/contrib/test_active_gaze_ppo.py (gaze-supervised PPO and the gaze head). Isaac Lab's kitless contrib environment smoke test builds and steps IsaacContrib-Cartpole-Camera-Fovi, the standalone-script smoke test launches the foveated-camera demo, and the demo browser test checks its card (fixed Newton XPBD physics, the Newton viewer).
  • Render tests on Newton Warp, Newton OptiX and Isaac RTX (with Kit) and on OVRTX (Kit-less), three per renderer: the distance seen at the retina's centre matches the analytic distance along the gaze, from the first frame after a gaze change; the gaze stays relative to a turned or moved mount; rendered distances follow each foveated ray. OptiX is untested: the pinned pyoptix 9.1.1 fails with Unsupported ABI version on both machines available to me (driver 580.178 locally, and OSMO L40 nodes), where the stock Isaac-Cartpole-Camera task with renderer=newton_optix fails the same way; the 3 OptiX cases skip there.
  • The demo renders the same fixations on Newton + Warp, Isaac RTX and OVRTX.
  • fovi's software sampler, applied to a dense pinhole image of the same scene, uses the same sample positions as the rendered retina and agrees with it to 0.7/255 (Newton) and 2.8/255 (RTX) mean absolute RGB difference. This checks sampling parity, so the same retina can be computed from a real camera's images; it does not test sim-to-real transfer.

Testing burden

Measured on one RTX 6000 Ada with warm Warp caches, each test file in its own process as CI runs them:

Tests Cases Files Time
Simulator-free (isaaclab_fovi and the contrib PPO test) 82 8 55 s, mostly process start-up
Render, with Kit (Newton Warp, Newton OptiX, Isaac RTX) 9 (3 OptiX skip here) 1 26 s
Render, OVRTX without Kit 3 1 22 s

CI runs them in the existing job for isaaclab_assets, isaaclab_contrib, isaaclab_teleop, isaaclab_mimic and isaaclab_experimental. The Docker base image installs the fovi extra: fovi, OpenEXR, and through fovi beartype, fastcore, plum-dispatch, pygeodesic, scikit-image and tifffile.

Performance

Isaac Lab's runtime benchmark (random actions, no policy, so environments reset as in training), Cartpole with 1024 environments, 1000 timed steps after 100 warmup steps, one RTX 6000 Ada, NVIDIA driver 580.178.04, at 81f8fb7 with its locked environment (Isaac Sim 6.2.0.0, Newton 1.9.0.dev0, Warp 1.17.0, OVRTX 0.6.0, PyTorch 2.11.0+cu130, fovi 2.6.0). Step: mean env.step time ± its standard deviation. Memory: peak rise of the GPU's used memory (nvidia-smi, every 0.5 s), covering simulation and rendering but not training.

Renderer Stock (96×96) Fovi, 96×96 Fovi, 32×32 (the task's)
Newton + Warp 2.55 ± 0.64 ms, 1.10 GiB 3.03 ± 0.67 ms, 1.16 GiB 1.76 ± 0.67 ms, 0.66 GiB
Newton + OVRTX 18.11 ± 1.27 ms, 8.00 GiB 22.11 ± 1.42 ms, 8.06 GiB 11.88 ± 1.39 ms, 5.36 GiB

At the stock resolution the foveated camera costs about 0.5 ms (19%) on Newton and 4 ms (22%) on OVRTX, part of which is OVRTX's per-frame cost for a moving camera. Training 4096 foveated environments at 32×32 (8.0 GB) takes less memory than 512 stock environments at 96×96 (9.5 GB).

Training

Cartpole, 200 iterations, stock PPO settings, on L40 nodes at f17a5c2; final mean reward / success (fovi: gaze error to the pole tip). Runs are in the W&B project nvidia/isaaclab_fovi (NVIDIA-internal).

Physics + renderer Stock, 96×96, 512 envs Fovi, 32×32, 4096 envs
Newton + Warp 4.96 / 1.00 (run) 4.91 / 0.99, 0.58° (run)
Newton + OVRTX 4.92 / 0.98 (run) 4.66 / 1.00, 0.79° (run)
PhysX + Isaac RTX 4.94 / 1.00 (run) 4.93 / 1.00, 0.78° (run)
OVPhysX + OVRTX 4.87 / 1.00 (run) 4.93 / 1.00, 0.63° (run)

Fovi needs more iterations than stock because the policy also learns where to look. After the move to contrib, a local 150-iteration run (512 environments, Newton + Warp) reaches a mean reward of 4.70 with a 0.49° gaze error.

Checklist

Docker and GPU tests run on demand. Push the commits you want tested, then
comment run-ci on the pull request.

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with uv run isaaclab --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team infrastructure labels Nov 17, 2026
@nblauch nblauch changed the title Add isaaclab_fovi foveated camera sensors Add isaaclab_fovi foveated camera sensors and active-gaze camera tasks Nov 18, 2026
@nblauch
nblauch marked this pull request as ready for review November 18, 2026 05:08
@nblauch
nblauch requested a review from a team November 18, 2026 05:08

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Needs attention: Design and architecture (1), Scope and over-engineering (6), API (3), Implementation (5), Style consistency (9), Test quality (7). Other reviewed categories look good.

  • PR description: Title matches sensors/tasks; shorten repeated camera prose, correct fixed-prim claims, and link reproducible env.step/memory comparisons with revisions, commands, software versions, warmup, and measurement methods; hardware, backends, environment counts, and metrics are supplied.
  • Design and architecture: GazeActionCfg imports Warp runtime implementation during task configuration; separate the declarative configuration and reference GazeAction through a resolvable string.
  • Scope and over-engineering: Minimal implementation: Extend Camera and existing task/PPO owners; keep initializers, examples, test helpers, remaining sensor modules, mdp/gaze, PPO/config, and task registration/config/agent modules; merge virtual_gaze into foveated_camera and foveated_rays into foveated_newton_camera; delete unused teacher and OpenCV paths, share accessors, and defer unused grid buffers. Suggested PR breakdown: sensor/package integration with renderer validation; gaze/PPO with update validation; tasks with registration and training validation.
  • API: The documented camera_model with spawn=None path fails for OVRTX distance output; obtain the near clipping distance from the referenced camera prim.
  • Compatibility and deprecation: No breaking changes identified.
  • Implementation: Both backends convert integer segmentation IDs through float32, corrupting values above 2²⁴; preserve integer precision during discrete pooling.
  • Style consistency: Custom sensor exports and direct MDP/RSL-RL exports bypass the required stub-owned loader; use lazy_export() with explicit distributed .pyi stubs.
  • Test quality: The next-frame test waits four updates after changing gaze, allowing delayed updates to pass; assert the first frame after the command.

Significant concerns. Posted 31 inline findings.

Automated review; human maintainers own approval decisions.

Comment thread source/isaaclab_fovi/docs/README.md Resolved

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Needs attention: Scope and over-engineering (5), API (4), Implementation (6), Style consistency (5), Test quality (6). Other reviewed categories look good.

  • PR description: Title matches; correct fixed-prim and Kuka-resolution claims, distinguish parametrized cases from unit tests, trim pending-sweep prose, and attach revisions, commands, warmup, and software conditions to supplied timing, throughput, and memory measurements.
  • Design and architecture: Architecture looks good.
  • Scope and over-engineering: Minimal implementation: extend Camera and existing task configurations; merge virtual_gaze into FoveatedCamera and foveated_rays into Newton, replacing ray-packing kernels with native operations; delete the OpenCV fallback and duplicate gaze storage. Keep all remaining sensors/, mdp/, rsl_rl/, tasks/, examples/*, export modules, and test-support modules for their geometry, configuration, runtime, or validation contracts. Suggested PR breakdown: sensors, packaging, and backend-render validation; gaze terms and capture validation; PPO and task registration with training validation.
  • API: OVRTX distance output dereferences spawn despite documented spawn=None support; obtain clipping distance from the referenced camera.
  • Compatibility and deprecation: No breaking changes identified.
  • Implementation: Grid publication rewrites unrendered environments using current gaze masks, corrupting their cached frames; restrict publication to updated environments.
  • Style consistency: Public export modules bypass required stub-backed lazy_export, adding parallel export ownership; replace their maps and eager imports with lazy_export and distributed init.pyi stubs.
  • Test quality: The next-frame test waits four frames after changing gaze, allowing propagation delays to pass; assert the first post-command render after warming up.

Significant concerns. Posted 26 inline findings.

Automated review; human maintainers own approval decisions.

Comment thread source/isaaclab_fovi/docs/README.md Resolved
@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the test cleanup warning is non-blocking.

Findings

  1. P2 Test camera cleanup raises ▶

Summary

Adds isaaclab_fovi cameras and active-gaze tasks. This update adds tests for camera movement, segmentation pooling, and camera guards.

  • Foveated cameras put more samples around the gaze and publish them as images.
  • Policies can move the gaze and learn to aim it toward a task target.
  • Three camera tasks pair foveated images with gaze actions and teacher targets.
  • The foveated package installs as an optional extra and joins the package build job.

Reviews (3) · Last reviewed commit: "Test the review fixes at their boundarie..." · Reviewed by Greptile

Comment thread source/isaaclab_fovi/docs/README.md Resolved
@nblauch

nblauch commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews. 9c192b3 addresses them as follows.

Fixed

  • Sensor mask on a rolled mount (image_rotation_deg): the mask now checks the rays as rendered, through the mount roll. A new test derives the expected visibility through the camera pose instead of the mask kernel, at 0°, 30° and −75°.
  • Partial frame updates: the fused output kernel publishes only the environments the frame updated; the others keep their last frame. Covered by the publish kernel test.
  • Isaac RTX mount: a position-only set_world_poses keeps the saved mount instead of compounding the gaze, and set_world_poses_from_view refreshes it.
  • OVRTX distance_to_camera with spawn=None: the near clip comes from the camera prim.
  • OVRTX async_rendering with mask_to_sensor_fov is rejected, since the mask would use a newer gaze than the frame.
  • Lens textures live in a temporary directory the camera owns.
  • image_shape and data.image_shape report the published size.
  • oracle_loss_coef=0 trains plain PPO without reading teacher labels (tested).
  • Teacher mount: a new GazeActionCfg.mount. "fixed" (default) raises if the camera's mount moves; "tracked" reads each camera's current mount from the pose update, for software gaze on a moving head (Newton, OVRTX).
  • Pooled segmentation takes each sample's modal channel vector in its native dtype, so IDs above 2**24 and colorized labels stay exact (pool_int_mode="mode", the new default).
  • compare_outputs.py reads "frames"; render_foveated.py keeps frames only with --output; README uses base_camera; square_foveated.md states spherical only.

Restructured

  • VirtualGaze is merged into FoveatedCamera, together with the shared accessors and sensor mask; foveated_rays.py is folded into the Newton camera; tiles are validated before allocation; gaze and velocity are views of one state buffer; the unused single-point teacher is removed.
  • sensors, mdp and rsl_rl use lazy_export() with __init__.pyi stubs.
  • Comments and docstrings trimmed as suggested.

Tests

  • The gaze-change render case asserts the first frame after the command, on Newton, Isaac RTX and OVRTX.
  • Independent oracles for the grid layout (SamplingCoords.as_grid), the lens-texture inverse, and pooled camera output; parameter products trimmed; kitless markers on simulator-free files.

Kept, with reasons

  • GazeActionCfg stays beside GazeAction: the stock ActionManager does not resolve string class_types, and changing that is outside this package.
  • fovi stays out of source/isaaclab_fovi/pyproject.toml: package files list only Isaac Lab packages, and the root fovi extra installs it.
  • One PR: the tasks are the validation for the gaze terms and the camera, so reviewing them apart would leave each half untested.
  • The pooled and sample-set output paths still mask every environment per frame; only the fused native-grid path (the tasks' path) restricts publication to updated environments.

@nblauch

nblauch commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@isaaclab-review-bot review

@nblauch

nblauch commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Needs attention: Design and architecture (1), Scope and over-engineering (2), API (2), Implementation (6), Style consistency (5), Test quality (3). Other reviewed categories look good.

  • PR description: Title matches; replace “Nothing else changes” with “core runtime unchanged,” remove pending-sweep prose, and retain supplied hardware/backend/environment measurements while adding revisions, commands, warmup and timing/memory methods for reproducibility.
  • Design and architecture: FoveatedCameraCfg overrides explicit spawn settings across renderers; keep configuration declarative and apply required camera-lock changes during RTX construction.
  • Scope and over-engineering: Minimal implementation: Extend Camera, ActionTerm and stock task configurations; keep all sensor modules, exports/stubs, MDP, PPO/config, task modules, examples and test-support modules for their runtime or public contracts, deleting canonical_to_grid and collapsing ModalPooler into camera-owned tables. Essential flow remains sampling, rendering, publication, gaze and supervision. Suggested PR breakdown: sensors/install/docs with renderer validation; gaze/PPO with action/training validation; task configurations/registrations with task-resolution validation.
  • API: GazeAction ignores supplied action values and reads manager state instead; consume the argument through the stable launch buffer so direct calls honor the action-term interface.
  • Compatibility and deprecation: No breaking changes identified.
  • Implementation: Non-fused camera postprocessing modifies inactive rows using current gaze, corrupting cached frames; carry the render-update mask through extraction, masking, pooling and publication.
  • Style consistency: GazeActionCfg shares its runtime implementation module and directly references GazeAction; separate configuration with a resolvable class_type to preserve the prescribed import boundary.
  • Test quality: Renderer density cases assert sampling metadata rather than rendered spacing, allowing ignored foveated rays to pass; assert independent off-center rendered distances or retain one geometry case.

Significant concerns. Posted 19 inline findings.

Automated review; human maintainers own approval decisions.

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Needs attention: Design and architecture (3), API (2), Implementation (5), Style consistency (8), Test quality (5). Other reviewed categories look good.

  • PR description: Title and feature scope match; hardware, backends, counts, and measurements are supplied, but qualify fixed-mount behavior, shorten training prose while preserving template sections, and supply benchmark revisions, commands, environment, warmup, and metric methods.
  • Design and architecture: RTX cameras retain owned lens directories after closure; release them at camera teardown after renderer borrowers, rather than relying on garbage collection.
  • Scope and over-engineering: No over-engineering issues.
  • API: Short gaze batches resize state below the camera count while pose updates still traverse every camera; validate shape (num_envs, 2) before changing storage.
  • Compatibility and deprecation: No breaking changes identified.
  • Implementation: Newton and RTX non-fused postprocessing ignores update masks, applying current gaze to cached frames; restrict masking, pooling, extraction, and publication to rendered rows.
  • Style consistency: Gaze configuration resides with runtime kernels and directly references its implementation; separate declarative configuration and use a resolvable class string to preserve the required import boundary.
  • Test quality: Simulator-free pooling and Newton-buffer files lack kitless markers, excluding them from marker-based CI selection; mark both files while retaining distinct physical, backend, precision, frame, and packaging coverage.

Significant concerns. Posted 23 inline findings.

Automated review; human maintainers own approval decisions.

@nblauch

nblauch commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@isaaclab-review-bot review

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Needs attention: Design and architecture (3), Scope and over-engineering (2), API (1), Implementation (3), Style consistency (5), Test quality (4). Other reviewed categories look good.

  • PR description: The title matches; qualify “Nothing else changes”; retain supplied hardware/backend/environment measurements, but add baseline/candidate revisions, commands, warmup and metric methodology; replace unfinished-sweep promises with linked results.
  • Design and architecture: Isaac RTX authors lens settings during initialization rather than construction; prepare and author them before scene cloning to preserve the required camera-authoring boundary.
  • Scope and over-engineering: Minimal implementation: Extend Camera, ActionTerm, PPO and stock task configurations; keep all added sensors, mdp, rsl_rl, task, export, example and test-support modules for their runtime or contract responsibilities; delete ModalPooler, reuse the existing neighbour table, and inline _foveate. Essential flow remains sampling, rendering, publication, gaze and supervision. Suggested PR breakdown: (1) sensors, pooling and packaging—geometry/publication tests; (2) gaze and PPO—fixation/update tests; (3) tasks and documentation—configuration resolution and task validation.
  • API: GazeSupervisedPPO is missing from its package’s lazy exports, preventing the paired public import; add its named stub import and all entry.
  • Compatibility and deprecation: No breaking changes identified.
  • Implementation: Unpooled Newton sample-set publication aliases buffers redrawn for every environment, overwriting unselected frames; publish into stable storage and copy only updated rows.
  • Style consistency: Added camera classes violate the prescribed member ordering, making lifecycle and public interfaces harder to locate; place constructors first, then properties, public methods and private methods.
  • Test quality: The zero-coefficient PPO test checks logs and finiteness without proving optimization, so a no-op update passes; compare its update against stock PPO using the same rollout.

Significant concerns. Posted 18 inline findings.

Automated review; human maintainers own approval decisions.

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Needs attention: Design and architecture (2), Scope and over-engineering (2), API (2), Implementation (4), Style consistency (5), Test quality (5). Other reviewed categories look good.

  • PR description: Title matches; replace “Nothing else changes” with “Core runtime unchanged,” remove pending-sweep prose, and make supplied step-time/memory comparisons reproducible with revisions, commands, software, hardware, backends, environment counts, warmup, and measurement methods.
  • Design and architecture: GazeAction duplicates camera-owned gaze, so public camera commands leave observations and teacher labels stale; derive action gaze from the camera.
  • Scope and over-engineering: Minimal implementation: extend Camera, ActionTerm, and existing fovi poolers; delete ModalPooler and collapse Torch masking handoffs into Warp operations, preserving geometry→render→publication flow. Keep all listed sensor, MDP, RSL-RL, task, export/configuration, example, and test-support modules for their contracts or resource lifecycles. Suggested PR breakdown: sensors and packaging with backend render validation; optional pooling with dtype/publication validation; gaze, PPO, and tasks with action/training validation.
  • API: Direct installation omits fovi despite public configuration imports requiring it; declare fovi>=2.6 in the package dependencies.
  • Compatibility and deprecation: No breaking changes identified.
  • Implementation: Unpooled Newton sample-set output aliases a buffer redrawn for every environment, allowing partial updates to alter unselected frames; publish selected rows into stable camera-owned storage.
  • Style consistency: FoveatedRTXCamera places properties and static hooks before its dunder methods, violating the required member order; move construction and destruction methods first.
  • Test quality: The zero-coefficient PPO test accepts a no-op optimizer because zero oracle metrics and finite parameters still pass; compare its parameter update against stock PPO on the same rollout.

Significant concerns. Posted 20 inline findings.

Automated review; human maintainers own approval decisions.

@maxkra15 maxkra15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The aggregate wheel drops the Fovi task entry point. Can we preserve it and check isaaclab list-envs --keyword Fovi from an installed wheel?

@nblauch

nblauch commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@isaaclab-review-bot review

ooctipus and others added 20 commits November 19, 2026 20:00
- The sampling math from FOVI-Dex: the CMF, the warped Cartesian lattice, the rays and their sample spacing.

- Figures rendered from the demo scene: the retina on the viewing sphere and the published image at two fixations, the renderers, foveation strength, the sensor mask, pooling, and the Cartpole task's gaze following the pole.

- An interactive sampling viewer for an equidistant fisheye and a pinhole camera, each with its own field of view, as the style, CMF constant, resolution and gaze change.

- An interactive pooling viewer showing which photoreceptors each sample pools, with grid windows or, for isotropic retinas, fovi's own KNN neighbourhoods.

The viewers' geometry matches fovi's SamplingCoords, the camera's gaze rotation and GridPoolingLayer.
Keep the tests that guard isaaclab_fovi's own behaviour against changes in Isaac Lab: how it reads Isaac Lab's camera configs, its gaze and mount handling, its rays, lens textures and published layout, its pooling of the camera's buffers and segmentation, its gaze manager terms, and the renders on each backend. fovi's own behaviour, such as each pooling mode's arithmetic and its exclusion of invalid photoreceptors, is left to fovi's tests.

- Drop CPU copies of tests whose kernels and Python paths are the same on every device, and parameter values that take the same branch.

- Merge the square-shell grid tests into the native-grid tests, the gaze kernels into the camera's gaze tests (now test_camera_gaze.py), the renderer dispatch and import boundary into the config tests, the lens textures and schema into the RTX camera tests, and Newton's ray array into the Newton buffer tests. The import check's helper script moves inline.

- Render tests share renders: gaze following and gaze change in one, both kinds of mount move in one, and the density check renders once, comparing against an even grid's rays computed directly.

Simulator-free: 200 cases in 15 files (79.7 s with one process per file) become 83 in 8 (49.1 s). Render: 6 cases and 7 renders per backend become 3 cases and 3 renders, 61.6 s to 44.3 s here with OptiX skipped.
Every FoveatedCameraCfg and GazeActionCfg field now has a one-line docstring, and the class points to the Foveated Camera page. The page gains what the docstrings carried and it lacked: the retinal window settings, the mount roll, the isotropic cortical layout, the RTX exit texture size, fractional pool_res_mult for sample sets, and the neighbour-distance options.
The overview figure now shows a 180-degree fisheye camera among
sim-ready YCB and Isaac Lab objects, path-traced with OVRTX. One
fixation sits far to the side, where part of the retina falls outside
the lens and renders black; the other is near the centre. The viewing
sphere fades the scene outside the retina's field.
…ases

The demo browser test compares each demo card's physics and viewers with
the script cases. The foveated-camera card lists Newton XPBD and the
Newton viewer, while the cases assumed PhysX and every viewer, so the
test failed. The demo always runs Newton XPBD and shows its camera in
the Newton viewer, so the cases now say so.
Shares the camera setup across the Newton and RTX backends, reuses the
sensor mask across grid outputs, declares the Cartpole active-gaze
configs directly and shortens the Foveated Camera guide. On top, the
guide's overview figure now shows a 180-degree fisheye in a cluttered
scene of sim-ready objects.
@ooctipus
ooctipus force-pushed the feat/isaaclab-fovi-sensors branch from 5653042 to 2628fde Compare November 20, 2026 04:04
@ooctipus ooctipus added the ci:run-docker Trigger the on-demand Docker and GPU CI workflow label Nov 20, 2026
@ooctipus

Copy link
Copy Markdown
Collaborator

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Nov 20, 2026
The Sampling section derives the cortical distance from the CMF and
gives each style its equation: the warped grid's eccentricity at a
lattice radius, and the isotropic rings' samples per ring. Isotropic
samples lie on the viewing sphere and suit kNN convolution; they render
on Newton Warp and OptiX but are not yet recommended on RTX.

The page now opens with the demo, with its OVRTX options in brackets.
The sampling viewer shows one camera, switching between an equidistant
fisheye, the default, and a pinhole.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants