Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/skills/add-task-pipeline/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ When implementing task constructors like `create<Task>` (e.g. `createClassifier`
## 🚫 Avoid / Anti-Patterns

- **Do NOT access tensors by index:** Avoid using `tensors[0]` or `tensors[1]` throughout the function body. Always destructure and name them explicitly.
- **Do NOT name options parameters `opts`:** Always name options function parameters `options` (e.g. `options?: { threshold?: number }`). Suffixes like `Opts` for types or properties (e.g. `MyTaskOptions`, `ModelOpts`, `modelOpts`) are acceptable.
- **Do NOT define extra inner helper functions:** You must define **exactly two** inner functions inside the `create<Task>` constructor: the `dispose` function and the task `worklet` executor function. **Push back hard against implementing any other helper closures inside the constructor scope.** Placing other helper functions (especially those that are called from inside the worklet and use the `create<Task>` scope variables) inside `create<Task>` creates implicit dependencies and closures that capture variables, making the code extremely difficult to reason about and debug.
- **Do NOT leak raw Tensors to consumers:** The returned methods must never return raw `Tensor` objects to the API consumer. Always convert output data to standard JavaScript values/objects before returning.
- **Do NOT cross thread boundaries unnecessarily:** Minimize passing heavy objects between JS and the Worklet thread to avoid serialization overhead.
Expand Down
1 change: 1 addition & 0 deletions .agents/skills/core-guidelines/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ Use the following index to locate the specific procedural guides for your task:
## 💡 Key Coding Conventions

- **Worklets**: Ensure all TypeScript functions directly wrapping native JSI calls start with the `"worklet";` directive so they are compatible with worklet-based libraries (e.g., React Native Reanimated).
- **Options Parameter Naming**: Always name function and method options parameters `options` (not `opts`, `optsObj`, `taskOpts`, or `chunkOpts`). Using `Opts` as a type or property suffix (e.g. `ModelOpts`, `TaskOpts`, `modelOpts`) is acceptable.
- **Memory Management**: When writing native C++ code with JSI, pay close attention to JSI reference management and handle ExecuTorch lifecycle states safely.
- **Keep Core Clean**: Always build on top of core primitives. Do not modify files in `cpp/core/` or `src/core/` unless you are fixing a bug in the foundational runtime.

Expand Down
38 changes: 19 additions & 19 deletions packages/react-native-executorch/src/extensions/cv/ops/boxes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,23 +72,23 @@ export function decodeBox<F extends BoxFormat>(
* @category Utils
* @typeParam F Bounding box coordinate format.
* @param box The original BoundingBox.
* @param opts Options defining dimensions and resize modes.
* @param opts.from The source bounds (e.g. model input dimensions).
* @param opts.to The destination bounds (e.g. original image dimensions).
* @param opts.resizeMode The mode used to resize the image {@link ResizeMode}
* @param options Options defining dimensions and resize modes.
* @param options.from The source bounds (e.g. model input dimensions).
* @param options.to The destination bounds (e.g. original image dimensions).
* @param options.resizeMode The mode used to resize the image {@link ResizeMode}
* (excluding `'crop'`).
* @returns The scaled BoundingBox object.
*/
export function scaleBox<F extends BoxFormat>(
box: BoundingBox<F>,
opts: {
options: {
readonly from: { readonly width: number; readonly height: number };
readonly to: { readonly width: number; readonly height: number };
readonly resizeMode: Exclude<ResizeMode, 'crop'>;
}
): BoundingBox<F> {
'worklet';
const { from, to, resizeMode } = opts;
const { from, to, resizeMode } = options;

let scaleX: number;
let scaleY: number;
Expand All @@ -107,8 +107,8 @@ export function scaleBox<F extends BoxFormat>(

switch (box.format) {
case 'xyxy': {
const pMin = scalePoint({ x: box.xmin, y: box.ymin }, opts);
const pMax = scalePoint({ x: box.xmax, y: box.ymax }, opts);
const pMin = scalePoint({ x: box.xmin, y: box.ymin }, options);
const pMax = scalePoint({ x: box.xmax, y: box.ymax }, options);
return {
format: 'xyxy',
xmin: pMin.x,
Expand All @@ -118,7 +118,7 @@ export function scaleBox<F extends BoxFormat>(
} as BoundingBox<F>;
}
case 'xywh': {
const pMin = scalePoint({ x: box.xmin, y: box.ymin }, opts);
const pMin = scalePoint({ x: box.xmin, y: box.ymin }, options);
return {
format: 'xywh',
xmin: pMin.x,
Expand All @@ -128,7 +128,7 @@ export function scaleBox<F extends BoxFormat>(
} as BoundingBox<F>;
}
case 'cxcywh': {
const pCenter = scalePoint({ x: box.cx, y: box.cy }, opts);
const pCenter = scalePoint({ x: box.cx, y: box.cy }, options);
return {
format: 'cxcywh',
cx: pCenter.x,
Expand Down Expand Up @@ -164,13 +164,13 @@ export type NmsOptions = {
* @category Utils
* @param boxes Bounding boxes coordinate tensor.
* @param scores Bounding boxes confidence scores tensor.
* @param opts Options configuring NMS thresholds and execution mode.
* @param opts.boxFormat The bounding box format {@link BoxFormat}.
* @param opts.iouThreshold Intersection over Union (IoU) threshold for
* @param options Options configuring NMS thresholds and execution mode.
* @param options.boxFormat The bounding box format {@link BoxFormat}.
* @param options.iouThreshold Intersection over Union (IoU) threshold for
* suppression.
* @param opts.confidenceThreshold Minimum confidence score for candidate
* @param options.confidenceThreshold Minimum confidence score for candidate
* selection.
* @param opts.nmsType The NMS algorithm variant {@link NmsOptions.nmsType}.
* @param options.nmsType The NMS algorithm variant {@link NmsOptions.nmsType}.
* @returns The resulting indices of the non-suppressed boxes:
* - For `standard` NMS: A 1D array of indices (`number[]`) representing the
* selected boxes.
Expand All @@ -182,16 +182,16 @@ export type NmsOptions = {
export function nms(
boxes: Tensor,
scores: Tensor,
opts: NmsOptions & { readonly nmsType: 'standard' }
options: NmsOptions & { readonly nmsType: 'standard' }
): number[];
export function nms(
boxes: Tensor,
scores: Tensor,
opts: NmsOptions & { readonly nmsType: 'weighted' }
options: NmsOptions & { readonly nmsType: 'weighted' }
): number[][];
export function nms(boxes: Tensor, scores: Tensor, opts: NmsOptions): number[] | number[][] {
export function nms(boxes: Tensor, scores: Tensor, options: NmsOptions): number[] | number[][] {
'worklet';
return rnexecutorchJsi.cv.nms(boxes, scores, opts);
return rnexecutorchJsi.cv.nms(boxes, scores, options);
}

/**
Expand Down
26 changes: 13 additions & 13 deletions packages/react-native-executorch/src/extensions/cv/ops/image.ts
Original file line number Diff line number Diff line change
Expand Up @@ -105,20 +105,20 @@ export type NormalizeOptions = {
* @param dst The pre-allocated destination tensor to write the resized image
* to. `dst` must be in HWC layout and its number of channels must match `src`.
* Shape [H',W',C].
* @param opts Configuration options for resizing.
* @param opts.mode The resize algorithm mode {@link ResizeMode}. Defaults to
* @param options Configuration options for resizing.
* @param options.mode The resize algorithm mode {@link ResizeMode}. Defaults to
* `'stretch'`.
* @param opts.interpolation The pixel interpolation method
* @param options.interpolation The pixel interpolation method
* {@link InterpolationMethod}. Defaults to `'lanczos'`.
* @param opts.padValue Fill value for letterboxing. Defaults to `0`.
* @param options.padValue Fill value for letterboxing. Defaults to `0`.
* @returns The destination tensor containing the resized image.
*/
export function resize(src: Tensor, dst: Tensor, opts?: ResizeOptions): Tensor {
export function resize(src: Tensor, dst: Tensor, options?: ResizeOptions): Tensor {
'worklet';
return rnexecutorchJsi.cv.resize(src, dst, {
mode: opts?.mode ?? 'stretch',
interpolation: opts?.interpolation ?? 'lanczos',
padValue: opts?.padValue ?? 0,
mode: options?.mode ?? 'stretch',
interpolation: options?.interpolation ?? 'lanczos',
padValue: options?.padValue ?? 0,
});
}

Expand Down Expand Up @@ -184,19 +184,19 @@ export function toChannelsLast(src: Tensor, dst: Tensor): Tensor {
* @param src The source image tensor in CHW layout. Shape [C,H,W].
* @param dst The pre-allocated destination tensor to write the normalized
* values to. `dst` must have the same shape as `src`. Shape [C,H,W].
* @param opts Normalization scaling coefficients.
* @param opts.alpha Multiplicative scaling coefficient(s). Defaults to
* @param options Normalization scaling coefficients.
* @param options.alpha Multiplicative scaling coefficient(s). Defaults to
* `1 / 255.0`.
* @param opts.beta Additive offset coefficient(s). Defaults to `0.0`.
* @param options.beta Additive offset coefficient(s). Defaults to `0.0`.
* @returns The destination tensor containing the normalized image.
*/
export function normalize(src: Tensor, dst: Tensor, opts?: NormalizeOptions): Tensor {
export function normalize(src: Tensor, dst: Tensor, options?: NormalizeOptions): Tensor {
'worklet';
const defaultNormalizeOptions = {
alpha: 1 / 255.0,
beta: 0.0,
} as const;
return rnexecutorchJsi.cv.normalize(src, dst, { ...defaultNormalizeOptions, ...opts });
return rnexecutorchJsi.cv.normalize(src, dst, { ...defaultNormalizeOptions, ...options });
}

/**
Expand Down
12 changes: 6 additions & 6 deletions packages/react-native-executorch/src/extensions/cv/ops/points.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,23 @@ export type Point = {
* changes.
* @category Utils
* @param point The original coordinate point to scale.
* @param opts Options detailing the scaling factors and resize mode.
* @param opts.from The source bounds (e.g. model input dimensions).
* @param opts.to The destination bounds (e.g. original image dimensions).
* @param opts.resizeMode The mode used to resize the image {@link ResizeMode}
* @param options Options detailing the scaling factors and resize mode.
* @param options.from The source bounds (e.g. model input dimensions).
* @param options.to The destination bounds (e.g. original image dimensions).
* @param options.resizeMode The mode used to resize the image {@link ResizeMode}
* (excluding `'crop'`).
* @returns The scaled coordinate point.
*/
export function scalePoint(
point: Point,
opts: {
options: {
readonly from: { readonly width: number; readonly height: number };
readonly to: { readonly width: number; readonly height: number };
readonly resizeMode: Exclude<ResizeMode, 'crop'>;
}
): Point {
'worklet';
const { from, to, resizeMode } = opts;
const { from, to, resizeMode } = options;
switch (resizeMode) {
case 'letterbox': {
const scale = Math.min(from.width / to.width, from.height / to.height);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,14 +77,14 @@ export type KeypointDetection<F extends BoxFormat, L extends PropertyKey> = {
* @param tBoxes Bounding boxes tensor output from inference.
* @param tScores Scores tensor output from inference.
* @param tKeypoints Keypoints tensor output from inference.
* @param opts Post-processing configuration options.
* @param options Post-processing configuration options.
* @returns Structured keypoint detection results list.
*/
function postprocess<F extends BoxFormat, L extends PropertyKey>(
tBoxes: Tensor,
tScores: Tensor,
tKeypoints: Tensor,
opts: {
options: {
readonly from: { readonly width: number; readonly height: number };
readonly to: { readonly width: number; readonly height: number };
readonly boxFormat: F;
Expand All @@ -96,7 +96,7 @@ function postprocess<F extends BoxFormat, L extends PropertyKey>(
): KeypointDetection<F, L>[] {
'worklet';

const nmsGroups = nms(tBoxes, tScores, { ...opts, nmsType: 'weighted' });
const nmsGroups = nms(tBoxes, tScores, { ...options, nmsType: 'weighted' });

const boxes = tBoxes.getData(new Float32Array(tBoxes.numel));
const scores = tScores.getData(new Float32Array(tScores.numel));
Expand All @@ -107,15 +107,15 @@ function postprocess<F extends BoxFormat, L extends PropertyKey>(
for (const group of nmsGroups) {
const totalScore = group.reduce((total, idx) => total + (scores[idx] ?? 0), 0);
const weightedBox = new Float32Array(4);
const weightedKpt = new Float32Array(opts.landmarks.length * 3);
const weightedKpt = new Float32Array(options.landmarks.length * 3);

for (const idx of group) {
const score = totalScore === 0 ? 1 / group.length : scores[idx]!;
weightedBox.forEach((v, i) => {
weightedBox[i] = v + score * boxes[idx * 4 + i]!;
});
weightedKpt.forEach((v, i) => {
weightedKpt[i] = v + score * keypoints[idx * opts.landmarks.length * 3 + i]!;
weightedKpt[i] = v + score * keypoints[idx * options.landmarks.length * 3 + i]!;
});
}

Expand All @@ -129,11 +129,11 @@ function postprocess<F extends BoxFormat, L extends PropertyKey>(
}

const [a, b, c, d] = weightedBox;
const box = scaleBox(decodeBox([a!, b!, c!, d!], opts.boxFormat), opts);
const box = scaleBox(decodeBox([a!, b!, c!, d!], options.boxFormat), options);
const landmarks = {} as Landmarks<L>;

for (const [i, key] of opts.landmarks.entries()) {
const point = scalePoint({ x: weightedKpt[i * 3]!, y: weightedKpt[i * 3 + 1]! }, opts);
for (const [i, key] of options.landmarks.entries()) {
const point = scalePoint({ x: weightedKpt[i * 3]!, y: weightedKpt[i * 3 + 1]! }, options);
const confidence = weightedKpt[i * 3 + 2]!;
landmarks[key] = { ...point, confidence };
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,15 @@ export type ImagePreprocessorOptions = {
* input shapes. All intermediate scratch tensors are pre-allocated and safely
* disposed of when calling `dispose()`.
* @category Typescript API
* @param opts Normalization scaling coefficients, interpolation algorithms, and
* @param options Normalization scaling coefficients, interpolation algorithms, and
* crop/resize modes.
* @param outputShape Expected output shape of the model input tensor (must
* match `[1, 3, H, W]` or `[3, H, W]`).
* @returns An object containing the `process` runner function and a `dispose`
* method.
*/
export function createImagePreprocessor(
opts: ImagePreprocessorOptions,
options: ImagePreprocessorOptions,
outputShape: number[]
): {
/**
Expand Down Expand Up @@ -86,7 +86,7 @@ export function createImagePreprocessor(
] as const;

const [tColor, tChanFirst, tNorm, tOutput] = tensors;
const { resizeMode, interpolation, normalizeOpts, padValue } = opts;
const { resizeMode, interpolation, normalizeOpts, padValue } = options;

const dispose = () => tensors.forEach((t) => t.dispose());
const process = (input: ImageBuffer): Tensor => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -111,13 +111,13 @@ function hannWindow(size: number): Float32Array {
// threshold with hysteresis, pad both ends, then merge near-adjacent regions.
// `scores[i]` holds the non-speech probability of frame `i`, so the speech
// probability is `1 - scores[i]`.
function postprocess(scores: Float32Array, opts: Required<VadOptions>): Segment[] {
function postprocess(scores: Float32Array, options: Required<VadOptions>): Segment[] {
'worklet';
const threshold = opts.speechThreshold;
const minSpeechHops = Math.floor(opts.minSpeechDurationMs / HOP_LENGTH_MS);
const minSilenceHops = Math.floor(opts.minSilenceDurationMs / HOP_LENGTH_MS);
const speechPadHops = Math.floor(opts.speechPadMs / HOP_LENGTH_MS);
const maxMergeGapHops = opts.mergeGapMs / HOP_LENGTH_MS;
const threshold = options.speechThreshold;
const minSpeechHops = Math.floor(options.minSpeechDurationMs / HOP_LENGTH_MS);
const minSilenceHops = Math.floor(options.minSilenceDurationMs / HOP_LENGTH_MS);
const speechPadHops = Math.floor(options.speechPadMs / HOP_LENGTH_MS);
const maxMergeGapHops = options.mergeGapMs / HOP_LENGTH_MS;

// Threshold with hysteresis: a region must stay above the threshold for
// `minSpeechHops` to open a segment, and below it for `minSilenceHops` to
Expand Down Expand Up @@ -252,7 +252,7 @@ export async function createFsmnVoiceActivityDetector(

const detectVoiceWorklet = (waveform: Float32Array, options?: VadOptions): Segment[] => {
'worklet';
const opts: Required<VadOptions> = { ...defaultOptions, ...options };
const mergedOpts: Required<VadOptions> = { ...defaultOptions, ...options };
const numFrames = Math.floor((waveform.length - FRAME_LENGTH) / HOP_LENGTH);
if (numFrames <= 0) return [];

Expand Down Expand Up @@ -294,7 +294,7 @@ export async function createFsmnVoiceActivityDetector(
offset += realFrames;
}

return postprocess(scores, opts);
return postprocess(scores, mergedOpts);
};

const detectVoice = wrapAsync(detectVoiceWorklet, runtime);
Expand Down