newton.sensors.SensorCamera#

class newton.sensors.SensorCamera(model, *, default_clear_data=None, default_render_config=None, load_textures=True, deformable_surface_mode=DeformableSurfaceMode.BOTH)[source]#

Bases: object

Raytraced camera sensor that renders a Newton model.

A camera sensor owns an internal renderer built for a model plus the default render settings (default_clear_data, default_render_config) used when update() is not given per-call overrides. The caller supplies the camera-space rays, the world-space per-view camera transforms, and the output image buffers to update(); the number of views is inferred from the leading dimension of the camera transforms.

Camera frame convention: each camera looks along its local -Z axis, with +Y up and +X right (the USD/OpenGL convention). The per-view camera_transforms place that camera frame in world space, and the camera_rays bundle stores per-pixel origins in [..., 0] and directions in [..., 1], both expressed in camera space.

Deformable triangle surfaces use Model.tri_color as sRGB albedo, like shape colors, and fall back to white when triangle colors are unavailable. Deformable visual meshes are skinned from the state each frame and render white, or with their UV texture when RenderConfig.enable_textures is set. Set RenderConfig.enable_backface_culling to False to render both sides with camera-facing normals.

The scene is lit by the light from create_default_light(), if any, plus hemispheric ambient light (see set_ambient_light()) when RenderConfig.enable_ambient_lighting is set.

Experimental

Rendering of lights authored on the model, and how they combine with the sensor’s own lights, may change without a formal deprecation cycle. Future releases are expected to import lights authored in USD and MJCF into the model and render them, which changes the color output of scenes that author lights.

The render configuration types are exposed as nested attributes (e.g. SensorCamera.RenderConfig, SensorCamera.ClearData, SensorCamera.WorldRenderFlag); they are not part of the top-level newton namespace.

Example:

import warp as wp
import newton
from newton.sensors import SensorCamera

camera = SensorCamera(model)
camera.create_default_light()

width, height = 640, 480
camera_rays = SensorCamera.compute_camera_rays_pinhole(width, height, camera_fov=1.0, device=model.device)
camera_transforms = wp.array([wp.transform_identity()], dtype=wp.transformf, device=model.device)
color = camera.allocate_image_color(camera_transforms.shape[0], width, height)

camera.update(state, camera_transforms, camera_rays, color_image=color)

class AntiAliasing(*values)#

Bases: IntEnum

Anti-aliasing mode applied to multisampled camera-ray bundles.

MSAA = 2#

resolve every ray against its own nearest hit and reuse shading for subsamples covering the same surface.

Anti-aliases silhouettes against both the background and other objects at a fraction of the SSAA cost when shading is expensive. Each surface is shaded using the first ray in bundle order that hits it, which may not be the covered ray nearest the pixel center. The built-in ray helpers put a center ray first for three or more samples, but surfaces missed by that ray can still be shaded off-center. Interior shading is not supersampled. Standalone particles and deformable triangle meshes use the particle or face index alongside their shared hit IDs, so distinct surfaces are shaded separately.

Type:: Multisample

NONE = 0#: Resolve a single ray per pixel. Multisample ray bundles are rejected.

SSAA = 1#

trace and shade every ray in the bundle, then average the results.

Highest quality and highest cost; shading runs once per sample.

Type:: Supersample

class ClearData(clear_color=0, clear_depth=0.0, clear_shape_index=4294967295, clear_normal=(0.0, 0.0, 0.0), clear_albedo=0)#

Bases: object

Default values written to output images before rendering.

__init__(clear_color=0, clear_depth=0.0, clear_shape_index=4294967295, clear_normal=(0.0, 0.0, 0.0), clear_albedo=0)#

clear_albedo: int = 0#: Packed RGBA value written to the albedo output.

clear_color: int = 0#: Packed RGBA value written to the color output.

clear_depth: float = 0.0#: Depth value written to the depth and forward-depth outputs [m].

clear_normal: tuple[float, float, float] = (0.0, 0.0, 0.0)#: Normal vector written to the normal output.

clear_shape_index: int = 4294967295#: Shape-index sentinel written to the shape-index output.

class DeformableSurfaceMode(*values)#

Bases: IntEnum

Which deformable surfaces a SensorCamera uploads and renders.

Visual meshes can overlap the simulation surface or lie inside it. Drawing both may hide parts of the visual mesh. The choice is made when the sensor is constructed.

Experimental

This option may change without a formal deprecation cycle while deformable visual support is experimental.

BOTH = 0#: Render the simulation triangles (cloth and soft-body surfaces) and the visual meshes, as the viewers do.

SIMULATION_ONLY = 1#: Render the simulation triangles only; the model’s visual meshes are left out.

VISUAL_ONLY = 2#: Render the visual meshes only; every simulation triangle is left out, including surfaces without a visual mesh.

class GaussianRenderMode(*values)#

Bases: IntEnum

Gaussian Render Mode

FAST = 0#: Fast Render Mode

QUALITY = 1#: Quality Render Mode, collect hits until minimum transmittance is reached

class RenderConfig(enable_global_world=True, enable_textures=False, texture_projection_mode=TextureProjectionMode.CUBIC, enable_shadows=False, enable_ambient_lighting=True, enable_particles=True, enable_backface_culling=True, enable_fast_math=True, output_color_space=ColorSpace.SRGB, render_order=RenderOrder.PIXEL_PRIORITY, anti_aliasing=AntiAliasing.NONE, tile_width=16, tile_height=8, block_dim=64, max_distance=1000.0, gaussians_mode=GaussianRenderMode.FAST, gaussians_min_transmittance=0.49, gaussians_max_num_hits=20)#

Bases: object

Raytrace render settings shared across all worlds.

__init__(enable_global_world=True, enable_textures=False, texture_projection_mode=TextureProjectionMode.CUBIC, enable_shadows=False, enable_ambient_lighting=True, enable_particles=True, enable_backface_culling=True, enable_fast_math=True, output_color_space=ColorSpace.SRGB, render_order=RenderOrder.PIXEL_PRIORITY, anti_aliasing=AntiAliasing.NONE, tile_width=16, tile_height=8, block_dim=64, max_distance=1000.0, gaussians_mode=GaussianRenderMode.FAST, gaussians_min_transmittance=0.49, gaussians_max_num_hits=20)#

anti_aliasing: AntiAliasing = 0#

Anti-aliasing mode applied to multisampled ray bundles (see AntiAliasing).

Has no effect on single-sample ray bundles. Multisampled bundles require AntiAliasing.SSAA or AntiAliasing.MSAA.

block_dim: int = 64#

Thread block dimension forwarded to wp.launch for the render megakernel.

Launch-time tuning only: it does not affect kernel codegen, so the renderer excludes it from the kernel cache key and changing it never triggers a recompilation.

enable_ambient_lighting: bool = True#: Enable ambient lighting for the scene.

enable_backface_culling: bool = True#

Cull back-facing triangles of rigid meshes and deformable surfaces.

When disabled, deformable triangle surfaces render from both sides with normals oriented toward the camera.

enable_fast_math: bool = True#: Compile render kernels with CUDA fast math.

enable_global_world: bool = True#: Include shapes that belong to no specific world.

enable_particles: bool = True#

Enable standalone particle rendering.

When the model has simulation triangles, particles referenced by them or by tetrahedra are not emitted as spheres, even when that surface is hidden.

enable_shadows: bool = False#

Enable shadow rays for directional lights.

Shadows are only rendered when a light also casts them (e.g. a light created via create_default_light() with enable_shadows=True); with no shadow-casting light this setting has no effect.

enable_textures: bool = False#: Enable texture-mapped rendering for shapes and skinned meshes.

gaussians_max_num_hits: int = 20#: Maximum Gaussian hits accumulated per ray.

gaussians_min_transmittance: float = 0.49#: Minimum transmittance before early-out during Gaussian rendering.

gaussians_mode: GaussianRenderMode = 0#: Gaussian splatting render mode (see GaussianRenderMode).

max_distance: float = 1000.0#: Maximum ray distance [m].

output_color_space: ColorSpace = 1#

Color space for packed color and albedo outputs.

Use ColorSpace.SRGB for display-encoded bytes or ColorSpace.LINEAR for linear RGB bytes.

render_order: RenderOrder = 0#: Render traversal order (see RenderOrder).

texture_projection_mode: TextureProjectionMode = 0#: Projection mode for texture-mapped shapes without UVs.

tile_height: int = 8#: Tile height [px] for RenderOrder.TILED traversal.

tile_width: int = 16#: Tile width [px] for RenderOrder.TILED traversal.

class RenderOrder(*values)#

Bases: IntEnum

Render Order

PIXEL_PRIORITY = 0#: Render the same pixel of every view before continuing to the next one

TILED = 2#: Render pixels in tiles, defined by tile_width x tile_height

VIEW_PRIORITY = 1#: Render all pixels of a whole view before continuing to the next one

class TextureProjectionMode(*values)#

Bases: IntEnum

Projection mode for texture-mapped shapes without authored UVs.

CUBIC = 0#: Project from the dominant local axis and sample once.

TRIPLANAR = 1#: Blend samples from all three local axes using normal-based weights.

class Utils#

Bases: object

Post-processing helpers for SensorCamera render outputs.

Converts raw render buffers into display-ready RGBA (to_rgba_*), tiles per-view images into a single grid buffer (flatten_*_to_rgba), and projects ray-distance depth onto the camera forward axis (convert_ray_depth_to_forward_depth). All helpers are staticmethod()s reached as SensorCamera.Utils.<helper>(...); the view count and device are read from the passed image (its leading dimension and device).

static convert_ray_depth_to_forward_depth(depth_image, camera_transforms, camera_rays, *, out_depth=None)#

Convert ray-distance depth to forward (planar) depth.

Projects each pixel’s camera-space hit position, including its ray origin offset, onto the camera’s forward axis, producing depth measured from the camera origin perpendicular to the image plane. The forward axis is derived from each camera transform by transforming camera-space (0, 0, -1) into world space.

Parameters:

depth_image (wp.array3d[wp.float32]) – Ray-distance depth [m] from update(), shape (view_count, height, width).
camera_transforms (wp.array[wp.transformf]) – World-space camera transforms, shape (view_count,).
camera_rays (wp.array4d[wp.vec3f]) – Camera-space rays from SensorCamera, shape (height, width, 1, 2). Ray direction vectors must be unit length; non-unit directions scale the converted depth. This conversion does not support multisampled depth; request forward_depth_image from update() instead.
out_depth (wp.array3d[wp.float32] | None) – Output forward-depth array [m] with the same shape as depth_image. If None, allocates a new one.

Returns:

Forward (planar) depth array, same shape as depth_image [m].

Return type:

wp.array3d[wp.float32]

static flatten_color_image_to_rgba(image, *, out_buffer=None, views_per_row=None)#

Flatten rendered color image to a tiled RGBA buffer.

Arranges one tile per view in a grid. Useful for writing a single pre-tiled image to disk; use to_rgba_from_color() with log_image() for in-viewer display.

Parameters:

image (wp.array3d[wp.uint32]) – Color output from update(), shape (view_count, height, width).
out_buffer (wp.array3d[wp.uint8] | None) – Pre-allocated RGBA buffer. If None, allocates a new one.
views_per_row (int | None) – Views per row in the grid. If None, picks a square-ish layout.

static flatten_depth_image_to_rgba(image, *, out_buffer=None, views_per_row=None, depth_range=None)#

Flatten rendered depth image to a tiled RGBA buffer.

Encodes depth as grayscale: inverts values (closer = brighter) and normalizes to the [50, 255] range. Background pixels (no hit) remain black. Useful for writing a single pre-tiled image to disk; use to_rgba_from_depth() with log_image() for in-viewer display.

Parameters:

image (wp.array3d[wp.float32]) – Depth output from update(), shape (view_count, height, width).
out_buffer (wp.array3d[wp.uint8] | None) – Pre-allocated RGBA buffer. If None, allocates a new one.
views_per_row (int | None) – Views per row in the grid. If None, picks a square-ish layout.
depth_range (wp.array[wp.float32] | tuple[float, float] | None) – Optional (near, far) [m] for normalization. Accepts a 2-element wp.array[wp.float32] on the image device or a Python (near, far) tuple. Values must be finite with near < far; array values are caller-validated to avoid a device-to-host transfer. If None, the per-frame range is computed on device from the image’s positive depth values (matches to_rgba_from_depth()).

static flatten_normal_image_to_rgba(image, *, out_buffer=None, views_per_row=None)#

Flatten rendered normal image to a tiled RGBA buffer.

Arranges one tile per view in a grid. Useful for writing a single pre-tiled image to disk; use to_rgba_from_normal() with log_image() for in-viewer display.

Parameters:

image (wp.array3d[wp.vec3f]) – Normal output from update(), shape (view_count, height, width).
out_buffer (wp.array3d[wp.uint8] | None) – Pre-allocated RGBA buffer. If None, allocates a new one.
views_per_row (int | None) – Views per row in the grid. If None, picks a square-ish layout.

static to_rgba_from_color(image)#

Reinterpret packed uint32 RGBA color sensor output as uint8 RGBA.

Returns a zero-copy view: each uint32 (R | G<<8 | B<<16 | A<<24) aliases 4 contiguous uint8 channels. The returned array shares memory with image; do not write into it.

The returned array plugs directly into log_image().

Parameters:: image (wp.array3d[wp.uint32]) – Color sensor output, shape (view_count, H, W), dtype uint32 (packed RGBA: R | G<<8 | B<<16 | A<<24). Must be contiguous.
Returns:: Array of shape (view_count, H, W, 4), dtype uint8, aliasing image.
Return type:: wp.array4d[wp.uint8]

static to_rgba_from_depth(image, *, depth_range=None, out_buffer=None)#

Convert float32 depth sensor output to uint8 grayscale RGBA.

Closer pixels render brighter; miss pixels (depth <= 0; matches the default ClearData.clear_depth = 0.0 sentinel) render black. Alpha = 255.

Parameters:

image (wp.array3d[wp.float32]) – Depth output, shape (view_count, H, W), dtype float32. Non-positive values denote ray misses.
depth_range (wp.array[wp.float32] | tuple[float, float] | None) – Optional (near, far) [m] for normalization. Accepts a 2-element wp.array[wp.float32] on the image device or a Python (near, far) tuple. Values must be finite with near < far; array values are caller-validated to avoid a device-to-host transfer. If None, the per-frame range is computed on device from the image’s positive depth values (matches flatten_depth_image_to_rgba()).
out_buffer (wp.array4d[wp.uint8] | None) – Optional pre-allocated output of shape (view_count, H, W, 4), dtype uint8.

Returns:

Array of shape (view_count, H, W, 4), dtype uint8. Suitable for log_image().

Return type:

wp.array4d[wp.uint8]

static to_rgba_from_normal(image, *, out_buffer=None)#

Convert vec3 normal sensor output to uint8 RGBA.

Parameters:

image (wp.array3d[wp.vec3f]) – Normal output, shape (view_count, H, W), dtype vec3f.
out_buffer (wp.array4d[wp.uint8] | None) – Optional pre-allocated output of shape (view_count, H, W, 4), dtype uint8.

Returns:

Array of shape (view_count, H, W, 4), dtype uint8. Suitable for log_image().

Return type:

wp.array4d[wp.uint8]

static to_rgba_from_shape_index(image, *, colors=None, out_buffer=None)#

Convert uint32 shape-index sensor output to uint8 RGBA.

Parameters:

image (wp.array3d[wp.uint32]) – Shape-index output, shape (view_count, H, W), dtype uint32.
colors (wp.array2d[wp.uint8] | None) – Optional RGB palette of shape (num_entries, 3), dtype uint8. If provided, each pixel is colored by looking up its shape index in this palette (indices past the palette length render black). If None, a deterministic hash palette is used (good for debugging which shape hit which pixel without a predefined class map).
out_buffer (wp.array4d[wp.uint8] | None) – Optional pre-allocated output of shape (view_count, H, W, 4), dtype uint8.

Returns:

Array of shape (view_count, H, W, 4), dtype uint8. Suitable for log_image().

Return type:

wp.array4d[wp.uint8]

class WorldRenderFlag(*values)#

Bases: IntEnum

Negative disable sentinels for the per-view world_indices array.

Each entry of world_indices is either a non-negative world index to render for that view, or one of these negative sentinels to skip the view.

DISABLE_CLEAR = -102#: Skip rendering and write clear values to output pixels.

DISABLE_PRESERVE = -101#: Skip rendering and leave output pixels unchanged.

static compute_camera_rays_fisheye_ftheta(width, height, optical_center_x, optical_center_y, *, image_width=None, image_height=None, k0=0.0, k1=1.0, k2=0.0, k3=0.0, k4=0.0, max_fov=2.0 * math.pi, sample_count=1, out_rays=None, device=None)#

Compute camera-space rays for one F-theta fisheye camera.

Inverts the F-theta radius polynomial r = k0 + k1 theta + k2 theta^2 + k3 theta^3 + k4 theta^4, which must be monotonic over [0, min(max_fov / 2, pi)].

Parameters:

width (int) – Output image width [px].
height (int) – Output image height [px].
optical_center_x (float) – Optical center x-coordinate [px].
optical_center_y (float) – Optical center y-coordinate [px].
image_width (float | None) – Calibration image width [px] (the F-theta nominal width). If None, uses width.
image_height (float | None) – Calibration image height [px] (the F-theta nominal height). If None, uses height.
k0 (float) – Constant F-theta polynomial coefficient [px].
k1 (float) – Linear F-theta polynomial coefficient [px/rad].
k2 (float) – Quadratic F-theta polynomial coefficient [px/rad^2].
k3 (float) – Cubic F-theta polynomial coefficient [px/rad^3].
k4 (float) – Quartic F-theta polynomial coefficient [px/rad^4].
max_fov (float) – Maximum field of view [rad]. Pixels whose undistorted angle exceeds max_fov / 2 receive a zero ray.
sample_count (int) – Number of rays per pixel. Use one for a centered ray.
out_rays (wp.array4d[wp.vec3f] | None) – Optional output buffer, shape (height, width, sample_count, 2) of vec3f. If None, a new one is allocated.
device (Device | str | None) – Device for the ray bundle. Defaults to the current Warp device.

Returns:

Ray origins ([..., 0]) and directions ([..., 1]), shape (height, width, sample_count, 2) of vec3f.

Return type:

wp.array4d[wp.vec3f]

static compute_camera_rays_fisheye_kannala_brandt(width, height, optical_center_x, optical_center_y, *, image_width=None, image_height=None, k0=1.0, k1=0.0, k2=0.0, k3=0.0, max_fov=2.0 * math.pi, sample_count=1, out_rays=None, device=None)#

Compute camera-space rays for one Kannala-Brandt fisheye camera.

Inverts the Kannala-Brandt radius polynomial r = k0 theta + k1 theta^3 + k2 theta^5 + k3 theta^7, which must be monotonic over [0, min(max_fov / 2, pi)].

Parameters:

width (int) – Output image width [px].
height (int) – Output image height [px].
optical_center_x (float) – Optical center x-coordinate [px].
optical_center_y (float) – Optical center y-coordinate [px].
image_width (float | None) – Calibration image width [px]. If None, uses width.
image_height (float | None) – Calibration image height [px]. If None, uses height.
k0 (float) – First Kannala-Brandt polynomial coefficient [px/rad].
k1 (float) – Second Kannala-Brandt polynomial coefficient [px/rad^3].
k2 (float) – Third Kannala-Brandt polynomial coefficient [px/rad^5].
k3 (float) – Fourth Kannala-Brandt polynomial coefficient [px/rad^7].
max_fov (float) – Maximum field of view [rad]. Pixels whose undistorted angle exceeds max_fov / 2 receive a zero ray.
sample_count (int) – Number of rays per pixel. Use one for a centered ray.
out_rays (wp.array4d[wp.vec3f] | None) – Optional output buffer, shape (height, width, sample_count, 2) of vec3f. If None, a new one is allocated.
device (Device | str | None) – Device for the ray bundle. Defaults to the current Warp device.

Returns:

Ray origins ([..., 0]) and directions ([..., 1]), shape (height, width, sample_count, 2) of vec3f.

Return type:

wp.array4d[wp.vec3f]

static compute_camera_rays_fisheye_opencv(width, height, fx, fy, cx, cy, *, image_width=None, image_height=None, k1=0.0, k2=0.0, k3=0.0, k4=0.0, max_fov=2.0 * math.pi, sample_count=1, out_rays=None, device=None)#

Compute camera-space rays for one OpenCV fisheye camera.

Inverts the OpenCV fisheye radius polynomial r = theta (1 + k1 theta^2 + k2 theta^4 + k3 theta^6 + k4 theta^8), which must be monotonic over [0, min(max_fov / 2, pi)].

Parameters:

width (int) – Output image width [px].
height (int) – Output image height [px].
fx (float) – Horizontal focal length [px].
fy (float) – Vertical focal length [px].
cx (float) – Principal point x-coordinate [px].
cy (float) – Principal point y-coordinate [px].
image_width (float | None) – Calibration image width [px]. If None, uses width.
image_height (float | None) – Calibration image height [px]. If None, uses height.
k1 (float) – First OpenCV fisheye distortion coefficient.
k2 (float) – Second OpenCV fisheye distortion coefficient.
k3 (float) – Third OpenCV fisheye distortion coefficient.
k4 (float) – Fourth OpenCV fisheye distortion coefficient.
max_fov (float) – Maximum field of view [rad]. Pixels whose undistorted angle exceeds max_fov / 2 receive a zero ray.
sample_count (int) – Number of rays per pixel. Use one for a centered ray.
out_rays (wp.array4d[wp.vec3f] | None) – Optional output buffer, shape (height, width, sample_count, 2) of vec3f. If None, a new one is allocated.
device (Device | str | None) – Device for the ray bundle. Defaults to the current Warp device.

Returns:

Ray origins ([..., 0]) and directions ([..., 1]), shape (height, width, sample_count, 2) of vec3f.

Return type:

wp.array4d[wp.vec3f]

static compute_camera_rays_pinhole(width, height, *, camera_fov=None, focal_length=None, horizontal_aperture=None, vertical_aperture=None, horizontal_aperture_offset=0.0, vertical_aperture_offset=0.0, sample_count=1, out_rays=None, device=None)#

Compute camera-space rays for one pinhole camera.

Provide either camera_fov or the aperture triple (focal_length, horizontal_aperture, vertical_aperture), not both. The focal length and apertures share consistent units; only their ratios affect the ray directions.

Parameters:

width (int) – Image width [px].
height (int) – Image height [px].
camera_fov (float | None) – Vertical field of view [rad], in (0, pi). Mutually exclusive with the aperture parameters.
focal_length (float | None) – Lens focal length; must be positive.
horizontal_aperture (float | None) – Horizontal sensor aperture; must be positive.
vertical_aperture (float | None) – Vertical sensor aperture; must be positive.
horizontal_aperture_offset (float) – Horizontal principal-point offset.
vertical_aperture_offset (float) – Vertical principal-point offset.
sample_count (int) – Number of rays to generate per pixel. Use one for a centered ray per pixel.
out_rays (wp.array4d[wp.vec3f] | None) – Optional output buffer, shape (height, width, sample_count, 2) of vec3f.
device (Device | str | None) – Device for the ray bundle. Defaults to the current Warp device.

Returns:

Ray origins and directions, shape (height, width, sample_count, 2) of vec3f.

Return type:

wp.array4d[wp.vec3f]

static compute_camera_rays_pinhole_opencv(width, height, fx, fy, cx, cy, *, image_width=None, image_height=None, k1=0.0, k2=0.0, k3=0.0, k4=0.0, k5=0.0, k6=0.0, p1=0.0, p2=0.0, s1=0.0, s2=0.0, s3=0.0, s4=0.0, sample_count=1, out_rays=None, device=None)#

Compute camera-space rays for one OpenCV pinhole camera.

Inverts OpenCV’s rational radial, tangential, and thin-prism distortion model with damped Newton iteration. The four- and five-coefficient variants are represented by leaving unused coefficients at zero. Pixels whose inverse cannot be verified within the solver tolerance receive a zero direction.

Parameters:

width (int) – Output image width [px].
height (int) – Output image height [px].
fx (float) – Horizontal focal length [px].
fy (float) – Vertical focal length [px].
cx (float) – Principal point x-coordinate [px].
cy (float) – Principal point y-coordinate [px].
image_width (float | None) – Calibration image width [px]. If None, uses width.
image_height (float | None) – Calibration image height [px]. If None, uses height.
k1 (float) – First numerator radial distortion coefficient.
k2 (float) – Second numerator radial distortion coefficient.
k3 (float) – Third numerator radial distortion coefficient.
k4 (float) – First denominator radial distortion coefficient.
k5 (float) – Second denominator radial distortion coefficient.
k6 (float) – Third denominator radial distortion coefficient.
p1 (float) – First tangential distortion coefficient.
p2 (float) – Second tangential distortion coefficient.
s1 (float) – First thin-prism distortion coefficient.
s2 (float) – Second thin-prism distortion coefficient.
s3 (float) – Third thin-prism distortion coefficient.
s4 (float) – Fourth thin-prism distortion coefficient.
sample_count (int) – Number of rays per pixel. Use one for a centered ray.
out_rays (wp.array4d[wp.vec3f] | None) – Optional output buffer, shape (height, width, sample_count, 2) of vec3f. If None, a new one is allocated.
device (Device | str | None) – Device for the ray bundle. Defaults to the current Warp device.

Returns:

Ray origins ([..., 0]) and directions ([..., 1]), shape (height, width, sample_count, 2) of vec3f.

Raises:

ValueError – If any focal length or calibration image dimension is non-positive, or if any calibration value is non-finite.

Return type:

wp.array4d[wp.vec3f]

static compute_camera_rays_pinhole_usd(width, height, camera, *, time=None, sample_count=1, out_rays=None, device=None)#

Compute camera-space rays for one USD pinhole camera.

Reads the perspective intrinsics (focal length and aperture) from a USD camera and builds the matching pinhole ray bundle.

Parameters:

width (int) – Image width [px].
height (int) – Image height [px].
camera (Any) – A UsdGeom.Camera or a Usd.Prim that is a camera. Its projection must be perspective.
time (Any | None) – USD time to sample the camera attributes at (a Usd.TimeCode or a frame number). If None, the default time code is used.
sample_count (int) – Number of rays per pixel. Use one for a centered ray.
out_rays (wp.array4d[wp.vec3f] | None) – Optional output buffer, shape (height, width, sample_count, 2) of vec3f. If None, a new one is allocated.
device (Device | str | None) – Device for the ray bundle. Defaults to the current Warp device.

Returns:

Ray origins ([..., 0]) and directions ([..., 1]), shape (height, width, sample_count, 2) of vec3f.

Return type:

wp.array4d[wp.vec3f]

__init__(model, *, default_clear_data=None, default_render_config=None, load_textures=True, deformable_surface_mode=DeformableSurfaceMode.BOTH)#

Construct a camera sensor for a model.

Parameters:

model (Model) – Newton simulation model to render. The sensor builds and owns its internal renderer for this model.
default_clear_data (ClearData | None) – Clear values used by update() when its clear_data argument is None. Defaults to ClearData().
default_render_config (RenderConfig | None) – Render settings used by update() when its render_config argument is None. Defaults to RenderConfig().
load_textures (bool) – Load the textures of shape meshes and deformable visual meshes from file paths or in-memory images. A texture that cannot be loaded raises ValueError here, so pass False for depth-only use, for example when replaying a recording whose texture files are elsewhere; it leaves all of them untextured.
deformable_surface_mode (DeformableSurfaceMode) –
Which deformable surfaces the sensor uploads and renders: DeformableSurfaceMode.BOTH renders the simulation triangles and the visual meshes, as the viewers do; DeformableSurfaceMode.SIMULATION_ONLY leaves the visual meshes out; DeformableSurfaceMode.VISUAL_ONLY leaves every simulation triangle out, including surfaces without a visual mesh, and their particles stay hidden. Like load_textures, this is decided at construction and cannot change later; per-render switches live on RenderConfig. See Camera sensor output for how visual meshes relate to the simulation surface.

Experimental

This deformable-visual option may change without a formal deprecation cycle.

allocate_image(view_count, width, height, dtype)#

Allocate a zeroed output image array on the model device.

Parameters:

view_count (int) – Number of views (the array’s leading dimension).
width (int) – Image width [px].
height (int) – Image height [px].
dtype (Any) – Warp element type for the array (e.g. wp.uint32, wp.float32, wp.vec3f).

Returns:

Zeroed array of shape (view_count, height, width) and dtype dtype, on the model device.

Return type:

wp.array[Any]

allocate_image_albedo(view_count, width, height)#

Allocate an RGBA albedo output array (packed uint32).

Parameters:

view_count (int) – Number of views (the array’s leading dimension).
width (int) – Image width [px].
height (int) – Image height [px].

Returns:

Zeroed array of shape (view_count, height, width), dtype uint32.

Return type:

wp.array3d[wp.uint32]

allocate_image_color(view_count, width, height)#

Allocate an RGBA color output array (packed uint32).

Parameters:

view_count (int) – Number of views (the array’s leading dimension).
width (int) – Image width [px].
height (int) – Image height [px].

Returns:

Zeroed array of shape (view_count, height, width), dtype uint32.

Return type:

wp.array3d[wp.uint32]

allocate_image_color_hdr(view_count, width, height)#

Allocate a linear HDR color output array (vec3f).

Parameters:

view_count (int) – Number of views (the array’s leading dimension).
width (int) – Image width [px].
height (int) – Image height [px].

Returns:

Zeroed array of shape (view_count, height, width), dtype vec3f.

Return type:

wp.array3d[wp.vec3f]

allocate_image_depth(view_count, width, height)#

Allocate a ray-distance depth output array [m].

Parameters:

view_count (int) – Number of views (the array’s leading dimension).
width (int) – Image width [px].
height (int) – Image height [px].

Returns:

Zeroed array of shape (view_count, height, width), dtype float32.

Return type:

wp.array3d[wp.float32]

allocate_image_depth_forward(view_count, width, height)#

Allocate a forward (planar) depth output array [m].

Parameters:

view_count (int) – Number of views (the array’s leading dimension).
width (int) – Image width [px].
height (int) – Image height [px].

Returns:

Zeroed array of shape (view_count, height, width), dtype float32.

Return type:

wp.array3d[wp.float32]

allocate_image_normal(view_count, width, height)#

Allocate a world-space surface-normal output array (vec3f).

Parameters:

view_count (int) – Number of views (the array’s leading dimension).
width (int) – Image width [px].
height (int) – Image height [px].

Returns:

Zeroed array of shape (view_count, height, width), dtype vec3f.

Return type:

wp.array3d[wp.vec3f]

allocate_image_shape_index(view_count, width, height)#

Allocate a shape-index output array.

Parameters:

view_count (int) – Number of views (the array’s leading dimension).
width (int) – Image width [px].
height (int) – Image height [px].

Returns:

Zeroed array of shape (view_count, height, width), dtype uint32.

Return type:

wp.array3d[wp.uint32]

assign_checkerboard_material(*, shape_indices, resolution=64, checker_size=32)#

Assign a gray checkerboard texture material to selected shapes.

Each call sets the whole selection: the selected shapes show the checkerboard and every other shape is untextured. Skinned visual meshes keep their textures.

Parameters:

shape_indices (Sequence[int] | ndarray) – Shape indices that should use the checkerboard texture.
resolution (int) – Texture resolution [px] (square texture).
checker_size (int) – Size of each checkerboard square [px].

compute_camera_transforms_usd(cameras, *, time=None, xform=None)#

Read world-space camera transforms from one USD camera per view.

Converts each camera pose from its USD stage up axis to the rendered model’s up axis, then composes an optional scene xform (e.g. the transform passed to ModelBuilder.add_usd). Use the result as the camera_transforms argument to update().

Parameters:

cameras (Any) – A single UsdGeom.Camera/Usd.Prim (one view) or a sequence of them (one per view, in view order).
time (Any | None) – USD time to sample the camera poses at (a Usd.TimeCode or a frame number). If None, the default time code is used.
xform (Any | None) – Optional scene transform (pos, quat) applied on top of the up-axis conversion, matching the pose used when importing the stage.

Returns:

World-space camera transforms, shape (view_count,) of transformf, on the model device.

Return type:

wp.array[wp.transformf]

create_default_light(*, enable_shadows=True, direction=None, color=None)#

Create a default directional light for the rendered scene.

This light replaces any light created before.

Parameters:

enable_shadows (bool) – Enable shadow casting for this light. Shadows are only rendered when the render config also enables them, i.e. enable_shadows=True here and render_config.enable_shadows=True (the latter defaults to False); both switches must be set.
direction (vec3f | None) – Normalized light direction. If None, defaults to normalized (-1, 1, -1) for a Z-up model, rotated to the model’s up axis otherwise, so the light always points down.
color (Sequence[float] | vec3f | None) –
Linear RGB radiance of the light. If None, defaults to white (1, 1, 1). Changing it does not recompile the render kernel.

Experimental

The color argument may change, for example to separate color from intensity, without a formal deprecation cycle.

Experimental

How this light combines with lights authored on the model may change without a formal deprecation cycle.

set_ambient_light(sky_color, ground_color=None)#

Set the hemispheric ambient light used when ambient lighting is enabled.

Surfaces facing the model’s up axis receive sky_color, surfaces facing away receive ground_color, and other orientations blend between the two. The defaults are (0.2, 0.2, 0.225) and (0.05, 0.05, 0.06). Changing the colors does not recompile the render kernel.

Parameters:

sky_color (Sequence[float] | vec3f) – Linear RGB ambient radiance from above.
ground_color (Sequence[float] | vec3f | None) – Linear RGB ambient radiance from below. If None, sky_color is used, which gives uniform ambient light.

Experimental

This method, its defaults, and how it combines with ambient light authored on the model may change without a formal deprecation cycle.

sync_deformable_meshes(state, *, deformable_visuals=None)#

Synchronize render-only deformable triangle-mesh points from state.

update() calls this by default (pass sync_deformables=False to skip it when you already synced). Call it explicitly only when you opt out of the automatic sync. Scenes without simulation triangles or visual meshes are a no-op. Shape and particle BVHs are refit separately via bvh_refit_shapes() and bvh_refit_particles(). Deformable visual meshes are skinned from state in at most four launches, however many worlds were replicated. Supplied visual buffers are consumed as-is; update or replay them before this call. Their source state and freshness are not checked.

Parameters:

state (newton.State) – Current particle positions and body transforms.
deformable_visuals (DeformableVisuals | None) – Experimental precomputed visual buffers for this model. Share these with a viewer to avoid skinning twice. If omitted, skin directly into the camera’s reusable point buffer.

Raises:

TypeError – If deformable_visuals is not DeformableVisuals.
ValueError – If the buffers belong to another model, or if this camera was constructed with DeformableSurfaceMode.SIMULATION_ONLY and cannot render them.
RuntimeError – If supplied visual buffers have never been updated.

Experimental

The deformable_visuals argument is experimental.

update(state, camera_transforms, camera_rays, *, color_image=None, depth_image=None, forward_depth_image=None, shape_index_image=None, normal_image=None, albedo_image=None, hdr_color_image=None, world_indices=None, clear_data=None, render_config=None, sync_deformables=True, deformable_visuals=None)#

Render this camera sensor.

The number of views is inferred from camera_transforms.shape[0]; all non-None output arrays must have shape (view_count, height, width) matching the camera_rays image dimensions.

On any frame whose geometry moved, refit the model’s shape and particle BVHs with bvh_refit_shapes() and bvh_refit_particles() (both are built initially by finalize()) before calling this; otherwise the render reads stale bounds. Deformable triangle-mesh points are synced from state automatically (see sync_deformables).

The camera looks along its local -Z axis with +Y up and +X right (the USD/OpenGL convention); camera_transforms place that frame in world space. camera_rays[..., 0] are per-sample ray origins and camera_rays[..., 1] are ray directions, both in camera space.

Parameters:

state (newton.State) – Simulation state with body and particle transforms.
camera_transforms (wp.array[wp.transformf]) – World-space camera transform per view [m, rad], shape (view_count,) of transformf, on the model device.
camera_rays (wp.array4d[wp.vec3f]) – Camera-space ray origins and directions, shape (height, width, sample_count, 2) of vec3f, on the model device. Bundles with sample_count > 1 require render_config.anti_aliasing to be AntiAliasing.SSAA or AntiAliasing.MSAA.
color_image (wp.array3d[wp.uint32] | None) – Output RGBA color buffer (packed uint32).
depth_image (wp.array3d[wp.float32] | None) – Output ray-distance depth from each ray origin [m].
forward_depth_image (wp.array3d[wp.float32] | None) – Output depth measured from the camera origin along its forward axis [m], including per-ray origin offsets.
shape_index_image (wp.array3d[wp.uint32] | None) – Output shape-index buffer.
normal_image (wp.array3d[wp.vec3f] | None) – Output world-space surface normals.
albedo_image (wp.array3d[wp.uint32] | None) – Output albedo buffer (packed uint32).
hdr_color_image (wp.array3d[wp.vec3f] | None) – Output linear HDR color buffer.
world_indices (wp.array[wp.int32] | None) – Optional per-view world selector, shape (view_count,). Defaults to the identity mapping (view i renders world i). A valid entry is a world index in [0, model.world_count). A WorldRenderFlag sentinel disables the view (DISABLE_CLEAR clears the outputs, DISABLE_PRESERVE leaves them unchanged). Any other value, including -1 (reserved for future global-world rendering) and indices >= model.world_count, is treated as DISABLE_CLEAR.
clear_data (ClearData | None) – Clear values for this call. Defaults to default_clear_data.
render_config (RenderConfig | None) – Render settings for this call. Defaults to default_render_config.
sync_deformables (bool) – Sync simulation surfaces and visual mesh points from state before rendering. Set False if you already called sync_deformable_meshes() this frame.
deformable_visuals (DeformableVisuals | None) – Experimental precomputed visual buffers for this model. If omitted, skin directly into the camera’s point buffer. Requires sync_deformables=True; otherwise pass the buffers to sync_deformable_meshes() explicitly. Update or replay supplied buffers before rendering; their freshness is not checked.

Raises:

ValueError – If deformable_visuals is given with sync_deformables=False, plus the sync_deformable_meshes() errors when it syncs.
TypeError – If supplied visual buffers have the wrong type.
RuntimeError – If supplied visual buffers have never been updated.

Experimental

The deformable_visuals argument is experimental. Other camera arguments retain their existing API status.

default_clear_data: ClearData#: Clear values used by update() when its clear_data argument is None.

default_render_config: RenderConfig#: Render settings used by update() when its render_config argument is None.

property device: Device#: Device of the model this sensor renders.