Describe the Bug
Summary
On the Jetson Orin the bench_01_radar_SingleChanSimplePipeline fails in the simple_radar_pipeline_pulse_compression benchmark with:
Run: [1/5] simple_radar_pipeline_pulse_compression [Device=0 T=cuda::std::__4::complex<float>
Pulses=128 Channels=16 Samples=9000 Waveform Length=1000]
Fail: Unexpected error: matxException (matxOutOfMemory: Failed to allocate memory) - MatX/include/matx/core/allocator.h:233
The default allocation strategy in MatX is managed memory. On the Jetson Orin, the allocator falls back to host pinned memory (allocator.h:195, also validated via debug logs).
To Reproduce
Note: This problem only occurs on the Jetson family when cudaDevAttrConcurrentManagedAccess = 0. Therefore, you’ll need a Jetson with this hardware attribute (e.g. Jetson Orin AGX as in my case) to reproduce the bug.
1.) Build MatX with benchmarks enabled
2.) Execute the Radar Pipeline benchmark (./build/bench/bench_01_radar_SingleChanSimplePipeline by default)
3.) Inspect output; should match:
Run: [1/5] simple_radar_pipeline_pulse_compression [Device=0 T=cuda::std::__4::complex<float>
Pulses=128 Channels=16 Samples=9000 Waveform Length=1000]
Fail: Unexpected error: matxException (matxOutOfMemory: Failed to allocate memory) - MatX/include/matx/core/allocator.h:233
Expected Behavior
The benchmark run should not fail, especially not with an OOM error.
Code Snippets
I think a good fix would be to guard cudaMemPrefetchAsync by the memory space:
__MATX_INLINE__ void PrefetchDevice(cudaStream_t const stream) const noexcept
{
…
if (memory_space != MATX_MANAGED_MEMORY)
{
// Log warning in debug mode
return;
}
...
cudaMemPrefetchAsync(this->Data(), this->desc_.TotalSize() * sizeof(T), dev, stream);
...
}This makes PrefetchDevice/Host a no-op on allocations that are not using managed memory.
System Details
OS: Ubuntu 24.04.4 (JetPack 7)
CUDA version: 13.2
g++ version: 13.3.0
System: Jetson Orin AGX (sm_87)
Additional Context
One way to get memory space is to use GetPointerKind. I couldn't find any usage of PrefetchDevice/Host in the data path. It only ever appears during setup. In this case, using GetPointerKind seems to be fine. There's another way to do this: store the memory space when it's allocated. The allocator has this information, but it is dropped and never used somewhere.
The full trace can be found below:
Full investigation (click to expand)
In debug mode, MatX's own debug log shows the actual CUDA error:
[ERROR] cub.h:772 - rv: Error in ub::DeviceReduce::Sum (101 != 0)
Fail: Unexpected error: matxException (matxCudaError: ) - /home/ai-user/Workspace/MatX/include/matx/transforms/cub.h:772
Debug and Release configuration report different errors because the MATX_ASSERT_STR_EXP macro is a no-op in Release (guarded by NDEBUG). Thus, in Debug the returned error code from CUB is checked via the macro. In Release the check is skipped and only a later check then throws the matxOutOfMemory error.
Stepping through the compiled source in gdb revealed that the error is swallowed and misreported by CUB in CCCL (this is kind of tricky because CUB does that to prevent any sticky errors leaking into unrelated kernel launches).
After switching to cuda-gdb, I realised that the CUDA state is already corrupted when entering the PulseCompression() function.
I stepped through the constructor of the RadarPipeline class and pinned the error down to the PrefetchDevice call:
Thread 1 "bench_01_radar_" hit Breakpoint 1,
RadarPipeline<cuda::std::__4::complex<float> >::RadarPipeline(...)
cancelMask.PrefetchDevice(stream);
(cuda-gdb) step
matx::tensor_t<float, 1, matx::tensor_desc_t<cuda::std::__4::array<long long, 1ul>, cuda::std::__4::array<long long, 1ul>, 1> >::PrefetchDevice(...)
cudaMemPrefetchAsync(this->Data(),...)
(cuda-gdb) next
Cuda Driver error detected: Address range(0x1021e8200 and size: 12) specified for prefetch must belong to system-allocated memory or managed memory
Cuda Driver error detected: Returning 1 (CUDA_ERROR_INVALID_VALUE) from cuMemPrefetchAsync_v2
The program would run fine if the CUB error handling would not check the sticky error. In fact, when getting and discarding the last CUDA driver error in PrefetchDevice, the program runs without a crash. But since the Jetson falls back to host pinned memory, this is incorrect API usage.
Describe the Bug
Summary
On the Jetson Orin the bench_01_radar_SingleChanSimplePipeline fails in the simple_radar_pipeline_pulse_compression benchmark with:
The default allocation strategy in MatX is managed memory. On the Jetson Orin, the allocator falls back to host pinned memory (allocator.h:195, also validated via debug logs).
To Reproduce
Note: This problem only occurs on the Jetson family when cudaDevAttrConcurrentManagedAccess = 0. Therefore, you’ll need a Jetson with this hardware attribute (e.g. Jetson Orin AGX as in my case) to reproduce the bug.
1.) Build MatX with benchmarks enabled
2.) Execute the Radar Pipeline benchmark (./build/bench/bench_01_radar_SingleChanSimplePipeline by default)
3.) Inspect output; should match:
Expected Behavior
The benchmark run should not fail, especially not with an OOM error.
Code Snippets
I think a good fix would be to guard cudaMemPrefetchAsync by the memory space:
This makes PrefetchDevice/Host a no-op on allocations that are not using managed memory.
System Details
OS: Ubuntu 24.04.4 (JetPack 7)
CUDA version: 13.2
g++ version: 13.3.0
System: Jetson Orin AGX (sm_87)
Additional Context
One way to get memory space is to use GetPointerKind. I couldn't find any usage of PrefetchDevice/Host in the data path. It only ever appears during setup. In this case, using GetPointerKind seems to be fine. There's another way to do this: store the memory space when it's allocated. The allocator has this information, but it is dropped and never used somewhere.
The full trace can be found below:
Full investigation (click to expand)
In debug mode, MatX's own debug log shows the actual CUDA error:Debug and Release configuration report different errors because the MATX_ASSERT_STR_EXP macro is a no-op in Release (guarded by NDEBUG). Thus, in Debug the returned error code from CUB is checked via the macro. In Release the check is skipped and only a later check then throws the matxOutOfMemory error.
Stepping through the compiled source in gdb revealed that the error is swallowed and misreported by CUB in CCCL (this is kind of tricky because CUB does that to prevent any sticky errors leaking into unrelated kernel launches).
After switching to cuda-gdb, I realised that the CUDA state is already corrupted when entering the PulseCompression() function.
I stepped through the constructor of the RadarPipeline class and pinned the error down to the PrefetchDevice call:
The program would run fine if the CUB error handling would not check the sticky error. In fact, when getting and discarding the last CUDA driver error in PrefetchDevice, the program runs without a crash. But since the Jetson falls back to host pinned memory, this is incorrect API usage.