Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
00b2ed7
Add isaaclab_fovi foveated camera sensors
nblauch Nov 17, 2026
26beb7f
Remove calibrated dense camera and unused stereo geometry
nblauch Nov 17, 2026
97f0ba2
Apply repository lint rules to isaaclab_fovi
nblauch Nov 17, 2026
3345fff
Make FoveatedCamera a class and drop Tiled from names
nblauch Nov 17, 2026
6f434d0
Specify cmf_a in degrees with a 0.591 degree default
nblauch Nov 17, 2026
cbb7130
Default to spherical grid sampling and derive the RTX layout
nblauch Nov 17, 2026
a75465f
Remove the projection option and unused gaze paths
nblauch Nov 17, 2026
9646a5e
Document the foveated camera
nblauch Nov 17, 2026
60de526
Silence the sample-count search during camera setup
nblauch Nov 17, 2026
2d803f5
Lock the fovi extra
nblauch Nov 17, 2026
e366555
Publish foveated camera outputs as ProxyArray
nblauch Nov 17, 2026
c461799
Add render tests for the foveated camera
nblauch Nov 17, 2026
d0e81f1
Run the isaaclab_fovi tests in CI
nblauch Nov 17, 2026
6622795
Build the render example scene with InteractiveScene
nblauch Nov 17, 2026
d22276b
Sample in spherical coordinates and derive the camera from spawn optics
nblauch Nov 17, 2026
5655c2e
Reduce gaze to one angle setter and rename camera_calibration to came…
nblauch Nov 17, 2026
c5f9ee0
Take gaze angles in degrees
nblauch Nov 17, 2026
c34810a
Keep the gaze relative to the mount when an RTX foveated camera is moved
nblauch Nov 17, 2026
8406f37
Test that a gaze set after the first frame is rendered
nblauch Nov 18, 2026
a6f35c3
Add active-gaze camera tasks with gaze-supervised PPO
nblauch Nov 18, 2026
8da928f
Give the foveated Cartpole task a 32-sample retina
nblauch Nov 18, 2026
c30a05a
Compute the gaze, the sensor mask and the teacher in Warp kernels
nblauch Nov 18, 2026
6e546f2
Cut the per-step launch overhead of the gaze
nblauch Nov 18, 2026
d532be6
Let physics graph the gaze tasks' actions and record the gaze launches
nblauch Nov 18, 2026
7772365
Cut the gaze tasks' per-step launches
nblauch Nov 18, 2026
cdd12d7
Turn the OVRTX gaze in the pose update and publish its grid in one ke…
nblauch Nov 18, 2026
4ecc528
Address review findings on the foveated camera and active gaze
nblauch Nov 18, 2026
7ce9e17
Write lens textures with OpenEXR only and trim the gaze pose test matrix
nblauch Nov 18, 2026
fc306b4
Name the RTX lens module after OmniLensDistortionLutAPI
nblauch Nov 18, 2026
1c2ffdf
Drop the proprietary license header from the lens module
nblauch Nov 18, 2026
c70120d
Test the review fixes at their boundaries
nblauch Nov 18, 2026
ec021e0
Address the second review round on the foveated camera and active gaze
nblauch Nov 18, 2026
0604204
Remove the unused teacher_labels helper
nblauch Nov 18, 2026
ca9a1b0
Build foveated Newton rays in the renderer's (H, W, 1, 2) layout
nblauch Nov 18, 2026
2a27bed
Address the third review round and support Newton OptiX
nblauch Nov 18, 2026
75ea3ff
List Newton OptiX among the foveated camera's renderers
nblauch Nov 18, 2026
a8c5111
Author the RTX lens at camera construction for both renderers
nblauch Nov 18, 2026
6877418
Fix the gaze actions' noise with a dedicated Gaussian action head
nblauch Nov 18, 2026
ad9f212
Keep bundled packages' entry points in the aggregate wheel
nblauch Nov 18, 2026
1098fe5
Address the fourth review round on the foveated camera and active gaze
nblauch Nov 18, 2026
3398f57
Fix gaze poses, pooled outputs and the sensor mask on partial updates
nblauch Nov 18, 2026
7eb3d99
Add the fovi API docs and port the render example to launch_simulation
nblauch Nov 18, 2026
e25b152
Explain that set_intrinsic_matrices leaves a foveated camera as confi…
nblauch Nov 18, 2026
1a145a8
Address the fifth review round on the foveated camera
nblauch Nov 18, 2026
1924b4f
Address the sixth review round on the foveated camera
nblauch Nov 19, 2026
e029f28
Read the Isaac RTX mount after a gaze and reduce the motor penalty once
nblauch Nov 19, 2026
49df019
Move the foveated Cartpole task into isaaclab_tasks contrib and docum…
nblauch Nov 19, 2026
13e3daf
Reduce FOVI to a shared retinal grid and gaze control
ooctipus Nov 19, 2026
ef89322
Leave photoreceptors the camera does not see out of pooling
nblauch Nov 19, 2026
2259b16
Keep Cartpole with stock PPO and concise foveation docs
ooctipus Nov 19, 2026
ae484c9
Illustrate the foveated camera page with figures and interactive viewers
nblauch Nov 19, 2026
539652b
Preserve FOVI behavior while simplifying RTX lens setup
ooctipus Nov 20, 2026
7c6cd9e
Trim the tests to what isaaclab_fovi builds on Isaac Lab
nblauch Nov 19, 2026
59941d3
Consolidate foveated sampling and visibility
ooctipus Nov 20, 2026
f07833d
Keep config docstrings to one line and move their detail into the docs
nblauch Nov 19, 2026
82c0876
Reuse stock Cartpole camera observations for active gaze
ooctipus Nov 20, 2026
dc13eae
Simplify foveated camera and gaze config validation
ooctipus Nov 20, 2026
feca9bb
Inline physical camera calibration selection
ooctipus Nov 20, 2026
0a845ed
Simplify foveated camera setup and output allocation
ooctipus Nov 20, 2026
4762888
Declare Cartpole active-gaze presets directly
ooctipus Nov 20, 2026
4fb6b32
Declare Cartpole camera and PPO configs at their use sites
ooctipus Nov 20, 2026
bc842cc
Inline gaze setup and document direct camera construction
ooctipus Nov 20, 2026
2710975
Keep Newton render buffers separate from public camera data
ooctipus Nov 20, 2026
36e81ca
Use packed poses and labels in the gaze teacher
ooctipus Nov 20, 2026
a8ee54f
Consolidate teacher bindings and coverage
ooctipus Nov 20, 2026
d63c5e1
Show the foveated camera through a fisheye in a cluttered scene
nblauch Nov 20, 2026
104e041
Declare the foveated-camera demo's physics and viewer in the script c…
nblauch Nov 20, 2026
4495260
Merge the simplified foveated camera setup (nblauch/IsaacLab#5)
nblauch Nov 20, 2026
2628fde
Adapt FOVI to current Isaac Sim and RSL-RL APIs
ooctipus Nov 20, 2026
df89d89
remove fovi readme
nblauch Nov 20, 2026
8f154a6
Explain the retina's sampling styles and lead with the demo
nblauch Nov 20, 2026
2ec52ba
Make the foveated camera demo interactive
nblauch Nov 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -608,7 +608,7 @@ jobs:
image-tag: ${{ needs.config.outputs.ci_image_tag }}
isaacsim-base-image: ${{ needs.config.outputs.isaacsim_image_name }}
isaacsim-version: ${{ needs.config.outputs.isaacsim_image_tag }}
filter-pattern: "isaaclab_assets,isaaclab_contrib,isaaclab_teleop,isaaclab_mimic,isaaclab_experimental"
filter-pattern: "isaaclab_assets,isaaclab_contrib,isaaclab_teleop,isaaclab_mimic,isaaclab_experimental,isaaclab_fovi"
warp-cache: restore
ovrtx-shader-cache: restore
test-jobs: "4"
Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.base
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ COPY --parents source/*/pyproject.toml ${ISAACLAB_PATH}/

# The image's package set. Kit ships in the base image, so the teleop stack comes without the
# ``isaacsim`` wheel; ``teleop`` itself is left exactly as tuned.
ARG IMAGE_EXTRAS="--extra sb3 --extra skrl --extra rl-games --extra rsl-rl --extra viser --extra rerun --extra mimic --extra teleop --extra test --extra tetrahedralization --extra ov"
ARG IMAGE_EXTRAS="--extra sb3 --extra skrl --extra rl-games --extra rsl-rl --extra viser --extra rerun --extra mimic --extra teleop --extra test --extra tetrahedralization --extra ov --extra fovi"

RUN --mount=type=cache,target=/root/.cache/uv \
cd ${ISAACLAB_PATH} && \
Expand Down
5 changes: 5 additions & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@
sys.path.insert(0, os.path.abspath("../source/isaaclab_ov/isaaclab_ov"))
sys.path.insert(0, os.path.abspath("../source/isaaclab_visualizers"))
sys.path.insert(0, os.path.abspath("../source/isaaclab_visualizers/isaaclab_visualizers"))
# Only the package root: the package's ``rsl_rl`` subpackage would shadow RSL-RL as a top-level module.
sys.path.insert(0, os.path.abspath("../source/isaaclab_fovi"))

# -- Project information -----------------------------------------------------

Expand Down Expand Up @@ -299,6 +301,7 @@ def _read_pinned_versions() -> dict:
"hydra.core.config_store",
"omegaconf",
"newton",
"fovi",
]

# List of zero or more Sphinx-specific warning categories to be squelched (i.e.,
Expand Down Expand Up @@ -348,11 +351,13 @@ def _read_pinned_versions() -> dict:
"css/environment-browser.css",
"css/demo-browser.css",
"css/guide-browser.css",
"css/foveation-viewer.css",
]
html_js_files = [
"css/environment-browser.js",
"css/demo-browser.js",
"css/guide-browser.js",
"css/foveation-viewer.js",
]

html_theme_options = {
Expand Down
2 changes: 1 addition & 1 deletion docs/source/_static/css/demo-browser.js
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@
if (["rerun", "viser"].includes(fields.visualizer.value)) {
requiredExtras.add(fields.visualizer.value);
}
const extraOrder = ["isaacsim", "ovphysx", "ovrtx", "tetrahedralization", "teleop", "rerun", "viser"];
const extraOrder = ["isaacsim", "ovphysx", "ovrtx", "tetrahedralization", "teleop", "fovi", "rerun", "viser"];
const extras = extraOrder.filter((extra) => requiredExtras.has(extra));

const parts = ["uvx"];
Expand Down
1 change: 1 addition & 0 deletions docs/source/_static/css/environment-browser.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions docs/source/_static/css/foveation-pooling-isotropic.json

Large diffs are not rendered by default.

206 changes: 206 additions & 0 deletions docs/source/_static/css/foveation-viewer.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
/*
* Copyright (c) 2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
* All rights reserved.
*
* SPDX-License-Identifier: BSD-3-Clause
*/

/* Interactive sampling-grid and pooling viewers on the Foveated Camera page. */

.fovi-viewer {
--fovi-ink: #1f2a37;
--fovi-muted: #8a96a8;
--fovi-faint: #d7dee8;
--fovi-surface: #ffffff;
--fovi-panel: #f6f8fb;
--fovi-border: #dfe5ec;
--fovi-accent: #2458d3;
--fovi-camera: #167d72;
--fovi-highlight: #d47b00;
--fovi-body: #29364b;
margin: 1.25rem 0 1.75rem;
padding: 1rem;
border: 1px solid var(--fovi-border);
border-radius: 8px;
background: var(--fovi-panel);
color: var(--fovi-ink);
}

html[data-theme="dark"] .fovi-viewer {
--fovi-ink: #e3e8ef;
--fovi-muted: #7d8899;
--fovi-faint: #3a4556;
--fovi-surface: #11161d;
--fovi-panel: #1a2029;
--fovi-border: #343d4b;
--fovi-accent: #6d9cff;
--fovi-camera: #3cc2ad;
--fovi-highlight: #f0a33a;
--fovi-body: #5b6f8c;
}

.fovi-controls {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(12.5rem, 1fr));
gap: 0.6rem 1.2rem;
align-items: end;
}

.fovi-control {
display: grid;
grid-template-columns: 1fr auto;
gap: 0.15rem 0.5rem;
margin: 0;
font-size: 0.85rem;
}

.fovi-viewer [hidden] {
display: none;
}

.fovi-control-name {
grid-column: 1 / -1;
}

.fovi-control input[type="range"],
.fovi-control select {
width: 100%;
min-width: 0;
accent-color: var(--fovi-accent);
}

.fovi-control select {
padding: 0.15rem 0.3rem;
border: 1px solid var(--fovi-border);
border-radius: 4px;
background: var(--fovi-surface);
color: var(--fovi-ink);
}

.fovi-control-value {
min-width: 3.5rem;
font-variant-numeric: tabular-nums;
text-align: right;
}

.fovi-reset {
justify-self: start;
padding: 0.25rem 0.9rem;
border: 1px solid var(--fovi-border);
border-radius: 4px;
background: var(--fovi-surface);
color: var(--fovi-ink);
font-size: 0.85rem;
cursor: pointer;
}

.fovi-summary {
margin: 0.9rem 0 0.6rem;
font-size: 0.9rem;
}

.fovi-panels {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(19rem, 1fr));
gap: 1rem;
}

.fovi-panels-stacked {
grid-template-columns: 1fr;
}

.fovi-panels-three {
grid-template-columns: repeat(auto-fit, minmax(11rem, 1fr));
}

.fovi-panel {
margin: 0;
padding: 0.6rem;
border: 1px solid var(--fovi-border);
border-radius: 6px;
background: var(--fovi-surface);
}

.fovi-panel-control {
max-width: 22rem;
margin-bottom: 0.4rem;
}

.fovi-panel figcaption {
margin-bottom: 0.4rem;
font-weight: 600;
font-size: 0.9rem;
}

.fovi-views {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 0.4rem;
}

.fovi-canvas {
display: block;
width: 100%;
aspect-ratio: 1;
touch-action: none;
}

.fovi-sphere {
cursor: grab;
}

.fovi-panel-labels {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 0.4rem;
color: var(--fovi-muted);
font-size: 0.75rem;
text-align: center;
}

.fovi-panel-note {
margin: 0.4rem 0 0;
font-size: 0.8rem;
}

.fovi-legend {
display: flex;
flex-wrap: wrap;
align-items: center;
gap: 0.3rem 0.5rem;
margin: 0.7rem 0 0;
font-size: 0.8rem;
}

.fovi-key {
display: inline-block;
width: 0.7rem;
height: 0.7rem;
margin-left: 0.6rem;
border-radius: 50%;
}

.fovi-key:first-child {
margin-left: 0;
}

.fovi-key-accent {
background: var(--fovi-accent);
}

.fovi-key-muted {
background: var(--fovi-muted);
}

.fovi-key-camera {
background: var(--fovi-camera);
}

.fovi-key-highlight {
background: var(--fovi-highlight);
}

.fovi-legend-note {
margin-left: auto;
color: var(--fovi-muted);
}
Loading