Repository navigation
Conversation
There was a problem hiding this comment.
Isaac Lab Review Bot
Needs attention: Design and architecture (1), Scope and over-engineering (6), API (3), Implementation (5), Style consistency (9), Test quality (7). Other reviewed categories look good.
- PR description: Title matches sensors/tasks; shorten repeated camera prose, correct fixed-prim claims, and link reproducible env.step/memory comparisons with revisions, commands, software versions, warmup, and measurement methods; hardware, backends, environment counts, and metrics are supplied.
- Design and architecture: GazeActionCfg imports Warp runtime implementation during task configuration; separate the declarative configuration and reference GazeAction through a resolvable string.
- Scope and over-engineering: Minimal implementation: Extend Camera and existing task/PPO owners; keep initializers, examples, test helpers, remaining sensor modules, mdp/gaze, PPO/config, and task registration/config/agent modules; merge virtual_gaze into foveated_camera and foveated_rays into foveated_newton_camera; delete unused teacher and OpenCV paths, share accessors, and defer unused grid buffers. Suggested PR breakdown: sensor/package integration with renderer validation; gaze/PPO with update validation; tasks with registration and training validation.
- API: The documented camera_model with spawn=None path fails for OVRTX distance output; obtain the near clipping distance from the referenced camera prim.
- Compatibility and deprecation: No breaking changes identified.
- Implementation: Both backends convert integer segmentation IDs through float32, corrupting values above 2²⁴; preserve integer precision during discrete pooling.
- Style consistency: Custom sensor exports and direct MDP/RSL-RL exports bypass the required stub-owned loader; use lazy_export() with explicit distributed .pyi stubs.
- Test quality: The next-frame test waits four updates after changing gaze, allowing delayed updates to pass; assert the first frame after the command.
Significant concerns. Posted 31 inline findings.
Automated review; human maintainers own approval decisions.
There was a problem hiding this comment.
Isaac Lab Review Bot
Needs attention: Scope and over-engineering (5), API (4), Implementation (6), Style consistency (5), Test quality (6). Other reviewed categories look good.
- PR description: Title matches; correct fixed-prim and Kuka-resolution claims, distinguish parametrized cases from unit tests, trim pending-sweep prose, and attach revisions, commands, warmup, and software conditions to supplied timing, throughput, and memory measurements.
- Design and architecture: Architecture looks good.
- Scope and over-engineering: Minimal implementation: extend Camera and existing task configurations; merge virtual_gaze into FoveatedCamera and foveated_rays into Newton, replacing ray-packing kernels with native operations; delete the OpenCV fallback and duplicate gaze storage. Keep all remaining sensors/, mdp/, rsl_rl/, tasks/, examples/*, export modules, and test-support modules for their geometry, configuration, runtime, or validation contracts. Suggested PR breakdown: sensors, packaging, and backend-render validation; gaze terms and capture validation; PPO and task registration with training validation.
- API: OVRTX distance output dereferences spawn despite documented spawn=None support; obtain clipping distance from the referenced camera.
- Compatibility and deprecation: No breaking changes identified.
- Implementation: Grid publication rewrites unrendered environments using current gaze masks, corrupting their cached frames; restrict publication to updated environments.
- Style consistency: Public export modules bypass required stub-backed lazy_export, adding parallel export ownership; replace their maps and eager imports with lazy_export and distributed init.pyi stubs.
- Test quality: The next-frame test waits four frames after changing gaze, allowing propagation delays to pass; assert the first post-command render after warming up.
Significant concerns. Posted 26 inline findings.
Automated review; human maintainers own approval decisions.
[Medium impact] The PR appears safe to merge; the test cleanup warning is non-blocking. Findings
SummaryAdds
Reviews (3) · Last reviewed commit: "Test the review fixes at their boundarie..." · Reviewed by Greptile |
|
Thanks for the reviews. 9c192b3 addresses them as follows. Fixed
Restructured
Tests
Kept, with reasons
|
|
@isaaclab-review-bot review |
|
@greptileai review |
There was a problem hiding this comment.
Isaac Lab Review Bot
Needs attention: Design and architecture (1), Scope and over-engineering (2), API (2), Implementation (6), Style consistency (5), Test quality (3). Other reviewed categories look good.
- PR description: Title matches; replace “Nothing else changes” with “core runtime unchanged,” remove pending-sweep prose, and retain supplied hardware/backend/environment measurements while adding revisions, commands, warmup and timing/memory methods for reproducibility.
- Design and architecture: FoveatedCameraCfg overrides explicit spawn settings across renderers; keep configuration declarative and apply required camera-lock changes during RTX construction.
- Scope and over-engineering: Minimal implementation: Extend Camera, ActionTerm and stock task configurations; keep all sensor modules, exports/stubs, MDP, PPO/config, task modules, examples and test-support modules for their runtime or public contracts, deleting canonical_to_grid and collapsing ModalPooler into camera-owned tables. Essential flow remains sampling, rendering, publication, gaze and supervision. Suggested PR breakdown: sensors/install/docs with renderer validation; gaze/PPO with action/training validation; task configurations/registrations with task-resolution validation.
- API: GazeAction ignores supplied action values and reads manager state instead; consume the argument through the stable launch buffer so direct calls honor the action-term interface.
- Compatibility and deprecation: No breaking changes identified.
- Implementation: Non-fused camera postprocessing modifies inactive rows using current gaze, corrupting cached frames; carry the render-update mask through extraction, masking, pooling and publication.
- Style consistency: GazeActionCfg shares its runtime implementation module and directly references GazeAction; separate configuration with a resolvable class_type to preserve the prescribed import boundary.
- Test quality: Renderer density cases assert sampling metadata rather than rendered spacing, allowing ignored foveated rays to pass; assert independent off-center rendered distances or retain one geometry case.
Significant concerns. Posted 19 inline findings.
Automated review; human maintainers own approval decisions.
There was a problem hiding this comment.
Isaac Lab Review Bot
Needs attention: Design and architecture (3), API (2), Implementation (5), Style consistency (8), Test quality (5). Other reviewed categories look good.
- PR description: Title and feature scope match; hardware, backends, counts, and measurements are supplied, but qualify fixed-mount behavior, shorten training prose while preserving template sections, and supply benchmark revisions, commands, environment, warmup, and metric methods.
- Design and architecture: RTX cameras retain owned lens directories after closure; release them at camera teardown after renderer borrowers, rather than relying on garbage collection.
- Scope and over-engineering: No over-engineering issues.
- API: Short gaze batches resize state below the camera count while pose updates still traverse every camera; validate shape (num_envs, 2) before changing storage.
- Compatibility and deprecation: No breaking changes identified.
- Implementation: Newton and RTX non-fused postprocessing ignores update masks, applying current gaze to cached frames; restrict masking, pooling, extraction, and publication to rendered rows.
- Style consistency: Gaze configuration resides with runtime kernels and directly references its implementation; separate declarative configuration and use a resolvable class string to preserve the required import boundary.
- Test quality: Simulator-free pooling and Newton-buffer files lack kitless markers, excluding them from marker-based CI selection; mark both files while retaining distinct physical, backend, precision, frame, and packaging coverage.
Significant concerns. Posted 23 inline findings.
Automated review; human maintainers own approval decisions.
|
@isaaclab-review-bot review |
There was a problem hiding this comment.
Isaac Lab Review Bot
Needs attention: Design and architecture (3), Scope and over-engineering (2), API (1), Implementation (3), Style consistency (5), Test quality (4). Other reviewed categories look good.
- PR description: The title matches; qualify “Nothing else changes”; retain supplied hardware/backend/environment measurements, but add baseline/candidate revisions, commands, warmup and metric methodology; replace unfinished-sweep promises with linked results.
- Design and architecture: Isaac RTX authors lens settings during initialization rather than construction; prepare and author them before scene cloning to preserve the required camera-authoring boundary.
- Scope and over-engineering: Minimal implementation: Extend Camera, ActionTerm, PPO and stock task configurations; keep all added sensors, mdp, rsl_rl, task, export, example and test-support modules for their runtime or contract responsibilities; delete ModalPooler, reuse the existing neighbour table, and inline _foveate. Essential flow remains sampling, rendering, publication, gaze and supervision. Suggested PR breakdown: (1) sensors, pooling and packaging—geometry/publication tests; (2) gaze and PPO—fixation/update tests; (3) tasks and documentation—configuration resolution and task validation.
- API: GazeSupervisedPPO is missing from its package’s lazy exports, preventing the paired public import; add its named stub import and all entry.
- Compatibility and deprecation: No breaking changes identified.
- Implementation: Unpooled Newton sample-set publication aliases buffers redrawn for every environment, overwriting unselected frames; publish into stable storage and copy only updated rows.
- Style consistency: Added camera classes violate the prescribed member ordering, making lifecycle and public interfaces harder to locate; place constructors first, then properties, public methods and private methods.
- Test quality: The zero-coefficient PPO test checks logs and finiteness without proving optimization, so a no-op update passes; compare its update against stock PPO using the same rollout.
Significant concerns. Posted 18 inline findings.
Automated review; human maintainers own approval decisions.
There was a problem hiding this comment.
Isaac Lab Review Bot
Needs attention: Design and architecture (2), Scope and over-engineering (2), API (2), Implementation (4), Style consistency (5), Test quality (5). Other reviewed categories look good.
- PR description: Title matches; replace “Nothing else changes” with “Core runtime unchanged,” remove pending-sweep prose, and make supplied step-time/memory comparisons reproducible with revisions, commands, software, hardware, backends, environment counts, warmup, and measurement methods.
- Design and architecture: GazeAction duplicates camera-owned gaze, so public camera commands leave observations and teacher labels stale; derive action gaze from the camera.
- Scope and over-engineering: Minimal implementation: extend Camera, ActionTerm, and existing fovi poolers; delete ModalPooler and collapse Torch masking handoffs into Warp operations, preserving geometry→render→publication flow. Keep all listed sensor, MDP, RSL-RL, task, export/configuration, example, and test-support modules for their contracts or resource lifecycles. Suggested PR breakdown: sensors and packaging with backend render validation; optional pooling with dtype/publication validation; gaze, PPO, and tasks with action/training validation.
- API: Direct installation omits fovi despite public configuration imports requiring it; declare fovi>=2.6 in the package dependencies.
- Compatibility and deprecation: No breaking changes identified.
- Implementation: Unpooled Newton sample-set output aliases a buffer redrawn for every environment, allowing partial updates to alter unselected frames; publish selected rows into stable camera-owned storage.
- Style consistency: FoveatedRTXCamera places properties and static hooks before its dunder methods, violating the required member order; move construction and destruction methods first.
- Test quality: The zero-coefficient PPO test accepts a no-op optimizer because zero oracle metrics and finite parameters still pass; compare its parameter update against stock PPO on the same rollout.
Significant concerns. Posted 20 inline findings.
Automated review; human maintainers own approval decisions.
maxkra15
left a comment
There was a problem hiding this comment.
The aggregate wheel drops the Fovi task entry point. Can we preserve it and check isaaclab list-envs --keyword Fovi from an installed wheel?
|
@isaaclab-review-bot review |
- The sampling math from FOVI-Dex: the CMF, the warped Cartesian lattice, the rays and their sample spacing. - Figures rendered from the demo scene: the retina on the viewing sphere and the published image at two fixations, the renderers, foveation strength, the sensor mask, pooling, and the Cartpole task's gaze following the pole. - An interactive sampling viewer for an equidistant fisheye and a pinhole camera, each with its own field of view, as the style, CMF constant, resolution and gaze change. - An interactive pooling viewer showing which photoreceptors each sample pools, with grid windows or, for isotropic retinas, fovi's own KNN neighbourhoods. The viewers' geometry matches fovi's SamplingCoords, the camera's gaze rotation and GridPoolingLayer.
Keep the tests that guard isaaclab_fovi's own behaviour against changes in Isaac Lab: how it reads Isaac Lab's camera configs, its gaze and mount handling, its rays, lens textures and published layout, its pooling of the camera's buffers and segmentation, its gaze manager terms, and the renders on each backend. fovi's own behaviour, such as each pooling mode's arithmetic and its exclusion of invalid photoreceptors, is left to fovi's tests. - Drop CPU copies of tests whose kernels and Python paths are the same on every device, and parameter values that take the same branch. - Merge the square-shell grid tests into the native-grid tests, the gaze kernels into the camera's gaze tests (now test_camera_gaze.py), the renderer dispatch and import boundary into the config tests, the lens textures and schema into the RTX camera tests, and Newton's ray array into the Newton buffer tests. The import check's helper script moves inline. - Render tests share renders: gaze following and gaze change in one, both kinds of mount move in one, and the density check renders once, comparing against an even grid's rays computed directly. Simulator-free: 200 cases in 15 files (79.7 s with one process per file) become 83 in 8 (49.1 s). Render: 6 cases and 7 renders per backend become 3 cases and 3 renders, 61.6 s to 44.3 s here with OptiX skipped.
Every FoveatedCameraCfg and GazeActionCfg field now has a one-line docstring, and the class points to the Foveated Camera page. The page gains what the docstrings carried and it lacked: the retinal window settings, the mount roll, the isotropic cortical layout, the RTX exit texture size, fractional pool_res_mult for sample sets, and the neighbour-distance options.
The overview figure now shows a 180-degree fisheye camera among sim-ready YCB and Isaac Lab objects, path-traced with OVRTX. One fixation sits far to the side, where part of the retina falls outside the lens and renders black; the other is near the centre. The viewing sphere fades the scene outside the retina's field.
…ases The demo browser test compares each demo card's physics and viewers with the script cases. The foveated-camera card lists Newton XPBD and the Newton viewer, while the cases assumed PhysX and every viewer, so the test failed. The demo always runs Newton XPBD and shows its camera in the Newton viewer, so the cases now say so.
Shares the camera setup across the Newton and RTX backends, reuses the sensor mask across grid outputs, declares the Cartpole active-gaze configs directly and shortens the Foveated Camera guide. On top, the guide's overview figure now shows a 180-degree fisheye in a cluttered scene of sim-ready objects.
5653042 to
2628fde
Compare
|
run-ci |
The Sampling section derives the cortical distance from the CMF and gives each style its equation: the warped grid's eccentricity at a lattice radius, and the isotropic rings' samples per ring. Isotropic samples lie on the viewing sphere and suit kNN convolution; they render on Newton Warp and OptiX but are not yet recommended on RTX. The page now opens with the demo, with its OVRTX options in brackets. The sampling viewer shows one camera, switching between an equidistant fisheye, the default, and a pinhole.
Description
Adds
isaaclab_fovi, a foveated camera for Isaac Lab built on the fovi library, active-gaze manager terms, and an active-gaze Cartpole task. A foveated camera samples the scene densely where it looks and sparsely in the periphery, on a retina-like grid, instead of on a uniform pixel grid. Try it withuv run --extra fovi isaaclab demo foveated-camera.Camera (
isaaclab_fovi.sensors, depends only onisaaclab)FoveatedCamera/FoveatedCameraCfgwork with thenewton_warp,newton_optix,isaac_rtxandovrtxrenderers. Newton casts one ray per retinal sample; RTX and OVRTX render through a lens lookup texture.spawndescribes (an ideal pinhole, or the OpenCV model inspawn.distortion), or any calibrated camera given ascamera_model.camera.set_gaze_angles(angles_deg). The gaze stays relative to the mount when the mount moves, andget_world_posesreports the mount on every renderer.mask_to_sensor_fovzeroes samples outside the physical camera's image.cmf_a_degsets the strength of foveation (default 0.591 deg, an estimate from human data); a large value gives a matched non-foveated baseline.(num_envs, H, W, C)image, asProxyArrayvalues like the coreCamera. Sample-set styles publish(num_envs, N, 1, C); they are experimental on RTX.pool_enabled,pool_mode); segmentation pools by modal label in its own dtype (pool_int_mode="modal"). Withmask_to_sensor_fov, pooling leaves out the photoreceptors the physical camera does not see, so samples at the edge of its image are not darkened.set_intrinsic_matricesleaves a foveated camera as configured and warns, as the core camera does for OpenCV lens models.Active gaze
isaaclab_fovi.mdp:GazeActionCfg(two bounded gaze actions, with a fixed or tracked mount),gaze_state, andgaze_teacher_labels, a teacher that labels the gaze action toward a point on an asset (never a policy input). They run as Warp kernels with recorded launches.IsaacContrib-Cartpole-Camera-Foviinisaaclab_tasks.contrib.active_gaze: the stock Cartpole camera task with a 32×32 foveated retina, two gaze actions and a teacher that looks at the pole's tip, with the stock image and critic observations, CNN and PPO settings. It is registered when thefoviextra is installed. ItsGazeSupervisedPPOadds a mean-only gaze supervision loss, and itsGazeGaussianDistributionhead holds the gaze actions' noise at a fixed 0.1: the gaze step clamps its actions, so a learned gaze noise meets no resistance and PPO's entropy bonus would grow it.Docs and demo. A concise Foveated Camera page under Sensors: an overview figure of a retina behind a 180° fisheye lens among sim-ready YCB and Isaac Lab objects, path-traced with OVRTX (one fixation far to the side shows how samples outside the lens render black), the sampling math, a configuration example, and two interactive viewers (
foveation-viewer.js): one draws the sampling grid for a pinhole and a fisheye camera as the style, CMF constant, resolution, field of view and gaze change, the other shows which photoreceptors each pooled sample averages. Their geometry matches fovi'sSamplingCoordsandGridPoolingLayerexactly. Also a section in Configure Rendering, thefoveated-camerademo (examples/demos/foveated_camera.py, on the Demos page), thefoviextra in the installation extras table and the troubleshooting import table, the task in the environment browser, and API pages forisaaclab_fovi.mdpandisaaclab_fovi.sensors. The docs build passes with warnings as errors.Simplification. nblauch#5 (by ooctipus, merged here) shares the camera setup across the Newton and RTX backends, records the sensor mask once per update and reuses it for every grid output (about 10% faster masked publication with three outputs on an RTX 5090), declares the Cartpole configs directly, and shortened the guide. It changes no behaviour: the resolved Cartpole env and agent configs are identical, and the RTX lens textures, grid outputs and teacher labels match byte for byte.
Dependencies. A new optional extra,
fovi, installs this package,fovi>=2.6(PyPI) andOpenEXR>=3(lens textures for RTX); throughfoviit also bringsbeartype,fastcore,plum-dispatch,pygeodesic,scikit-imageandtifffile. No core runtime code changes. Outside the new package and the contrib task, the PR touches the rootpyproject.toml(the extra) and its extras test,uv.lock,docker/Dockerfile.baseand.github/workflows/build.yaml(CI installs the extra and runs the package's tests), the docs, andisaaclab/programs.pywith the standalone-script cases inisaaclab/test/app/standalone_script_cases.py(the demo).Companion. A draft IsaacLabTutorial PR adds an SO-101 vial-placement task with active gaze that builds on this package.
Type of change
Validation
isaaclab_fovibuilds on Isaac Lab, so that a later Isaac Lab change that breaks it fails here; fovi's own behaviour is left to fovi's tests. 77 simulator-free test cases undersource/isaaclab_fovi/test(how the camera reads Isaac Lab's camera configs, its gaze and mount handling and sensor mask, its rays, lens textures and published layout, its pooling of the camera's buffers and segmentation, and the gaze manager terms) and 5 undersource/isaaclab_tasks/test/contrib/test_active_gaze_ppo.py(gaze-supervised PPO and the gaze head). Isaac Lab's kitless contrib environment smoke test builds and stepsIsaacContrib-Cartpole-Camera-Fovi, the standalone-script smoke test launches thefoveated-camerademo, and the demo browser test checks its card (fixed Newton XPBD physics, the Newton viewer).pyoptix9.1.1 fails withUnsupported ABI versionon both machines available to me (driver 580.178 locally, and OSMO L40 nodes), where the stockIsaac-Cartpole-Cameratask withrenderer=newton_optixfails the same way; the 3 OptiX cases skip there.Testing burden
Measured on one RTX 6000 Ada with warm Warp caches, each test file in its own process as CI runs them:
isaaclab_foviand the contrib PPO test)CI runs them in the existing job for
isaaclab_assets,isaaclab_contrib,isaaclab_teleop,isaaclab_mimicandisaaclab_experimental. The Docker base image installs thefoviextra:fovi,OpenEXR, and throughfovibeartype,fastcore,plum-dispatch,pygeodesic,scikit-imageandtifffile.Performance
Isaac Lab's runtime benchmark (random actions, no policy, so environments reset as in training), Cartpole with 1024 environments, 1000 timed steps after 100 warmup steps, one RTX 6000 Ada, NVIDIA driver 580.178.04, at 81f8fb7 with its locked environment (Isaac Sim 6.2.0.0, Newton 1.9.0.dev0, Warp 1.17.0, OVRTX 0.6.0, PyTorch 2.11.0+cu130, fovi 2.6.0). Step: mean
env.steptime ± its standard deviation. Memory: peak rise of the GPU's used memory (nvidia-smi, every 0.5 s), covering simulation and rendering but not training.At the stock resolution the foveated camera costs about 0.5 ms (19%) on Newton and 4 ms (22%) on OVRTX, part of which is OVRTX's per-frame cost for a moving camera. Training 4096 foveated environments at 32×32 (8.0 GB) takes less memory than 512 stock environments at 96×96 (9.5 GB).
Training
Cartpole, 200 iterations, stock PPO settings, on L40 nodes at f17a5c2; final mean reward / success (fovi: gaze error to the pole tip). Runs are in the W&B project
nvidia/isaaclab_fovi(NVIDIA-internal).Fovi needs more iterations than stock because the policy also learns where to look. After the move to contrib, a local 150-iteration run (512 environments, Newton + Warp) reaches a mean reward of 4.70 with a 0.49° gaze error.
Checklist
Docker and GPU tests run on demand. Push the commits you want tested, then
comment
run-cion the pull request.pre-commitchecks withuv run isaaclab --formatsource/<pkg>/changelog.d/for every touched package (do not editCHANGELOG.rstor bump package versions — CI handles that)CONTRIBUTORS.mdor my name already exists there