newton.sensors.SensorTiledCamera#
- class newton.sensors.SensorTiledCamera(model, *, config=None, load_textures=True)[source]#
Bases:
objectWarp-based tiled camera sensor for raytraced rendering across multiple worlds.
Renders up to six image channels per (world, camera) pair:
color – RGBA shaded image (
uint32).hdr_color – linear shaded RGB image (
vec3f).depth – ray-hit distance [m] (
float32); negative means no hit.normal – surface normal at hit point (
vec3f).albedo – unshaded surface color (
uint32).shape_index – shape id per pixel (
uint32).
All output arrays have shape
(world_count, camera_count, height, width). Use theflatten_*helpers to rearrange them into tiled RGBA buffers for display, with one tile per (world, camera) pair laid out in a grid.Shapes without the
VISIBLEflag are excluded.Example
sensor = SensorTiledCamera(model) rays = sensor.utils.compute_pinhole_camera_rays(width, height, fov) color = sensor.utils.create_color_image_output(width, height) # After setup, build BVHs once for the initial state. state = model.state() newton.geometry.build_bvh_shape(model, state) newton.geometry.build_bvh_particle(model, state) # Before each later frame that changes geometry, refit BVHs. newton.geometry.refit_bvh_shape(model, state) newton.geometry.refit_bvh_particle(model, state) sensor.update(state, camera_transforms, rays, color_image=color)
See
RenderConfigfor optional rendering settings andClearData/DEFAULT_CLEAR_DATA/GRAY_CLEAR_DATAfor image-clear presets.- class ClearData(clear_color=0, clear_depth=0.0, clear_shape_index=4294967295, clear_normal=(0.0, 0.0, 0.0), clear_albedo=0)#
Bases:
objectDefault values written to output images before rendering.
- __init__(clear_color=0, clear_depth=0.0, clear_shape_index=4294967295, clear_normal=(0.0, 0.0, 0.0), clear_albedo=0)#
- class Config(checkerboard_texture=False, default_light=False, default_light_shadows=False, enable_ambient_lighting=True, colors_per_world=False, colors_per_shape=False, backface_culling=True, enable_textures=False, enable_particles=True)#
Bases:
objectSensor configuration.
Deprecated since version 1.1: Use
RenderConfigandSensorTiledCamera.utils.*instead.- __init__(checkerboard_texture=False, default_light=False, default_light_shadows=False, enable_ambient_lighting=True, colors_per_world=False, colors_per_shape=False, backface_culling=True, enable_textures=False, enable_particles=True)#
- backface_culling: bool = True#
Deprecated since version 1.1: Use
render_config.enable_backface_cullinginstead.
- checkerboard_texture: bool = False#
Deprecated since version 1.1: Use
SensorTiledCamera.utils.assign_checkerboard_material_to_all_shapes()instead.
- colors_per_shape: bool = False#
Deprecated since version 1.1: Use shape colors instead (e.g.
builder.add_shape_cylinder(..., color=(r, g, b))).
- colors_per_world: bool = False#
Deprecated since version 1.1: Use shape colors instead (e.g.
builder.add_shape_cylinder(..., color=(r, g, b))).
- default_light: bool = False#
Deprecated since version 1.1: Use
SensorTiledCamera.utils.create_default_light()instead.
- default_light_shadows: bool = False#
Deprecated since version 1.1: Use
SensorTiledCamera.utils.create_default_light(enable_shadows=True)instead.
- enable_ambient_lighting: bool = True#
Deprecated since version 1.1: Use
render_config.enable_ambient_lightinginstead.
- class GaussianRenderMode(*values)#
Bases:
IntEnumGaussian Render Mode
- FAST = 0#
Fast Render Mode
- QUALITY = 1#
Quality Render Mode, collect hits until minimum transmittance is reached
- class RenderConfig(enable_global_world=True, enable_textures=False, enable_shadows=False, enable_ambient_lighting=True, enable_particles=True, enable_backface_culling=True, render_order=RenderOrder.PIXEL_PRIORITY, tile_width=16, tile_height=8, max_distance=1000.0, gaussians_mode=GaussianRenderMode.FAST, gaussians_min_transmittance=0.49, gaussians_max_num_hits=20)#
Bases:
objectRaytrace render settings shared across all worlds.
- __init__(enable_global_world=True, enable_textures=False, enable_shadows=False, enable_ambient_lighting=True, enable_particles=True, enable_backface_culling=True, render_order=RenderOrder.PIXEL_PRIORITY, tile_width=16, tile_height=8, max_distance=1000.0, gaussians_mode=GaussianRenderMode.FAST, gaussians_min_transmittance=0.49, gaussians_max_num_hits=20)#
- class RenderLightType(*values)#
Bases:
IntEnumLight types supported by the Warp raytracer.
- DIRECTIONAL = 1#
Directional Light.
- SPOTLIGHT = 0#
Spotlight.
- class RenderOrder(*values)#
Bases:
IntEnumRender Order
- PIXEL_PRIORITY = 0#
Render the same pixel of every view before continuing to the next one
- TILED = 2#
Render pixels in tiles, defined by tile_width x tile_height
- VIEW_PRIORITY = 1#
Render all pixels of a whole view before continuing to the next one
- class Utils(render_context)#
Bases:
objectUtility functions for the RenderContext.
- __init__(render_context)#
- assign_checkerboard_material_to_all_shapes(resolution=64, checker_size=32)#
Assign a gray checkerboard texture material to all shapes. Creates a gray checkerboard pattern texture and applies it to all shapes in the scene.
- assign_random_colors_per_shape(seed=100)#
Assign a random color to each shape.
Deprecated since version 1.1: Use shape colors instead (e.g.
builder.add_shape_cylinder(..., color=(r, g, b))).- Parameters:
seed (int) – Random seed.
- assign_random_colors_per_world(seed=100)#
Assign each world a random color, applied to all its shapes.
Deprecated since version 1.1: Use shape colors instead (e.g.
builder.add_shape_cylinder(..., color=(r, g, b))).- Parameters:
seed (int) – Random seed.
- compute_pinhole_camera_rays(width, height, camera_fovs)#
Compute camera-space ray directions for pinhole cameras.
Generates rays in camera space (origin at the camera center, direction normalized) for each pixel based on the vertical field of view.
- convert_ray_depth_to_forward_depth(depth_image, camera_transforms, camera_rays, out_depth=None)#
Convert ray-distance depth to forward (planar) depth.
Projects each pixel’s hit distance along its ray onto the camera’s forward axis, producing depth measured perpendicular to the image plane. The forward axis is derived from each camera transform by transforming camera-space
(0, 0, -1)into world space.- Parameters:
depth_image (wp.array4d[wp.float32]) – Ray-distance depth [m] from
update(), shape(world_count, camera_count, height, width).camera_transforms (wp.array2d[wp.transformf]) – World-space camera transforms, shape
(camera_count, world_count).camera_rays (wp.array4d[wp.vec3f]) – Camera-space rays from
compute_pinhole_camera_rays(), shape(camera_count, height, width, 2).out_depth (wp.array4d[wp.float32] | None) – Output forward-depth array [m] with the same shape as depth_image. If
None, allocates a new one.
- Returns:
Forward (planar) depth array, same shape as depth_image [m].
- Return type:
wp.array4d[wp.float32]
- create_albedo_image_output(width, height, camera_count=1)#
Create an albedo output array for
update().
- create_default_light(enable_shadows=True, direction=None)#
Create a default directional light oriented at
(-1, 1, -1).
- create_hdr_color_image_output(width, height, camera_count=1)#
Create a linear HDR color output array for
update().
- create_normal_image_output(width, height, camera_count=1)#
Create a normal output array for
update().
- create_shape_index_image_output(width, height, camera_count=1)#
Create a shape-index output array for
update().
- flatten_color_image_to_rgba(image, out_buffer=None, worlds_per_row=None)#
Flatten rendered color image to a tiled RGBA buffer.
Arranges
(world_count * camera_count)tiles in a grid. Each tile shows one camera’s view of one world. Useful for writing a single pre-tiled image to disk; useto_rgba_from_color()withlog_image()for in-viewer display.- Parameters:
image (wp.array4d[wp.uint32]) – Color output from
update(), shape(world_count, camera_count, height, width).out_buffer (wp.array3d[wp.uint8] | None) – Pre-allocated RGBA buffer. If None, allocates a new one.
worlds_per_row (int | None) – Tiles per row in the grid. If None, picks a square-ish layout.
- flatten_depth_image_to_rgba(image, out_buffer=None, worlds_per_row=None, depth_range=None)#
Flatten rendered depth image to a tiled RGBA buffer.
Encodes depth as grayscale: inverts values (closer = brighter) and normalizes to the
[50, 255]range. Background pixels (no hit) remain black. Useful for writing a single pre-tiled image to disk; useto_rgba_from_depth()withlog_image()for in-viewer display.- Parameters:
image (wp.array4d[wp.float32]) – Depth output from
update(), shape(world_count, camera_count, height, width).out_buffer (wp.array3d[wp.uint8] | None) – Pre-allocated RGBA buffer. If None, allocates a new one.
worlds_per_row (int | None) – Tiles per row in the grid. If None, picks a square-ish layout.
depth_range (wp.array[wp.float32] | None) – Depth range to normalize to, shape
(2,)[near, far]. If None, computes from image.
- flatten_normal_image_to_rgba(image, out_buffer=None, worlds_per_row=None)#
Flatten rendered normal image to a tiled RGBA buffer.
Arranges
(world_count * camera_count)tiles in a grid. Each tile shows one camera’s view of one world. Useful for writing a single pre-tiled image to disk; useto_rgba_from_normal()withlog_image()for in-viewer display.- Parameters:
image (wp.array4d[wp.vec3f]) – Normal output from
update(), shape(world_count, camera_count, height, width).out_buffer (wp.array3d[wp.uint8] | None) – Pre-allocated RGBA buffer. If None, allocates a new one.
worlds_per_row (int | None) – Tiles per row in the grid. If None, picks a square-ish layout.
- to_rgba_from_color(image)#
Reinterpret packed
uint32RGBA color sensor output asuint8RGBA.Returns a zero-copy view: each
uint32(R | G<<8 | B<<16 | A<<24) aliases 4 contiguousuint8channels and the(world_count, camera_count)axes are flattened. The returned array shares memory with image; do not write into it.The returned array plugs directly into
log_image(). World is the slower-changing axis: tileihasworld = i // camera_countandcamera = i % camera_count.- Parameters:
image (wp.array(dtype=wp.uint32, ndim=4)) – Color sensor output, shape
(world_count, camera_count, H, W), dtypeuint32(packed RGBA:R | G<<8 | B<<16 | A<<24). Must be contiguous; arrays returned byupdate()always satisfy this.- Returns:
Array of shape
(world_count * camera_count, H, W, 4), dtypeuint8, aliasing image.- Return type:
wp.array(dtype=wp.uint8, ndim=4)
- to_rgba_from_depth(image, depth_range=None, out_buffer=None)#
Convert float32 depth sensor output to
uint8grayscale RGBA.Closer pixels render brighter; miss pixels (depth <= 0; matches the default
ClearData.clear_depth = 0.0sentinel) render black. Alpha = 255.- Parameters:
image (wp.array4d[wp.float32]) – Depth output, shape
(world_count, camera_count, H, W), dtypefloat32. Non-positive values denote ray misses.depth_range (wp.array[wp.float32] | tuple[float, float] | None) – Optional
(near, far)[m] for normalization. Accepts a 2-elementwp.array[wp.float32]or a Python(near, far)tuple. IfNone, the per-frame range is computed on device byfind_depth_range()(matchesflatten_depth_image_to_rgba()).out_buffer (wp.array4d[wp.uint8] | None) – Optional pre-allocated output of shape
(world_count * camera_count, H, W, 4), dtypeuint8.
- Returns:
Array of shape
(world_count * camera_count, H, W, 4), dtypeuint8. Suitable forlog_image().- Return type:
wp.array4d[wp.uint8]
- to_rgba_from_normal(image, out_buffer=None)#
Convert vec3 normal sensor output to
uint8RGBA.- Parameters:
image (wp.array4d[wp.vec3f]) – Normal output, shape
(world_count, camera_count, H, W), dtypevec3f.out_buffer (wp.array4d[wp.uint8] | None) – Optional pre-allocated output of shape
(world_count * camera_count, H, W, 4), dtypeuint8.
- Returns:
Array of shape
(world_count * camera_count, H, W, 4), dtypeuint8. Suitable forlog_image().- Return type:
wp.array4d[wp.uint8]
- to_rgba_from_shape_index(image, colors=None, out_buffer=None)#
Convert uint32 shape-index sensor output to
uint8RGBA.- Parameters:
image (wp.array4d[wp.uint32]) – Shape-index output, shape
(world_count, camera_count, H, W), dtypeuint32.colors (wp.array2d[wp.uint8] | None) – Optional RGB palette of shape
(num_entries, 3), dtypeuint8. If provided, each pixel is colored by looking up its shape index in this palette (indices past the palette length render black). IfNone, a deterministic hash palette is used (good for debugging which shape hit which pixel without a predefined class map).out_buffer (wp.array4d[wp.uint8] | None) – Optional pre-allocated output of shape
(world_count * camera_count, H, W, 4), dtypeuint8.
- Returns:
Array of shape
(world_count * camera_count, H, W, 4), dtypeuint8. Suitable forlog_image().- Return type:
wp.array4d[wp.uint8]
- __init__(model, *, config=None, load_textures=True)#
Initialize the tiled camera sensor from a simulation model.
Builds the internal
RenderContext, loads shape geometry (and optionally textures) from model, and exposesutilsfor creating output buffers, computing rays, and assigning materials.- Parameters:
model (Model) – Simulation model whose shapes will be rendered.
config (Config | RenderConfig | None) – Rendering configuration. Pass a
RenderConfigto control raytrace settings directly, orNoneto use defaults. The legacyConfigdataclass is still accepted but deprecated.load_textures (bool) – Load texture data from the model. Set to
Falseto skip texture loading when textures are not needed.
- assign_checkerboard_material_to_all_shapes(resolution=64, checker_size=32)#
Assign a gray checkerboard texture material to all shapes.
Creates a gray checkerboard pattern texture and applies it to all shapes in the scene.
Deprecated since version 1.1: Use
SensorTiledCamera.utils.assign_checkerboard_material_to_all_shapesinstead.
- assign_random_colors_per_shape(seed=100)#
Assign a random color to each shape.
Deprecated since version 1.1: Use shape colors instead (e.g.
builder.add_shape_cylinder(..., color=(r, g, b))).- Parameters:
seed (int) – Random seed.
- assign_random_colors_per_world(seed=100)#
Assign each world a random color, applied to all its shapes.
Deprecated since version 1.1: Use shape colors instead (e.g.
builder.add_shape_cylinder(..., color=(r, g, b))).- Parameters:
seed (int) – Random seed.
- compute_pinhole_camera_rays(width, height, camera_fovs)#
Compute camera-space ray directions for pinhole cameras.
Generates rays in camera space (origin at the camera center, direction normalized) for each pixel based on the vertical field of view.
Deprecated since version 1.1: Use
SensorTiledCamera.utils.compute_pinhole_camera_raysinstead.
- create_albedo_image_output(width, height, camera_count=1)#
Create an albedo output array for
update().Deprecated since version 1.1: Use
SensorTiledCamera.utils.create_albedo_image_outputinstead.
- create_color_image_output(width, height, camera_count=1)#
Create a color output array for
update().Deprecated since version 1.1: Use
SensorTiledCamera.utils.create_color_image_outputinstead.
- create_default_light(enable_shadows=True)#
Create a default directional light oriented at
(-1, 1, -1).Deprecated since version 1.1: Use
SensorTiledCamera.utils.create_default_lightinstead.- Parameters:
enable_shadows (bool) – Enable shadow casting for this light.
- create_depth_image_output(width, height, camera_count=1)#
Create a depth output array for
update().Deprecated since version 1.1: Use
SensorTiledCamera.utils.create_depth_image_outputinstead.
- create_normal_image_output(width, height, camera_count=1)#
Create a normal output array for
update().Deprecated since version 1.1: Use
SensorTiledCamera.utils.create_normal_image_outputinstead.
- create_shape_index_image_output(width, height, camera_count=1)#
Create a shape-index output array for
update().Deprecated since version 1.1: Use
SensorTiledCamera.utils.create_shape_index_image_outputinstead.
- flatten_color_image_to_rgba(image, out_buffer=None, worlds_per_row=None)#
Flatten rendered color image to a tiled RGBA buffer.
Arranges
(world_count * camera_count)tiles in a grid. Each tile shows one camera’s view of one world.Deprecated since version 1.1: Use
SensorTiledCamera.utils.flatten_color_image_to_rgbainstead.- Parameters:
image (wp.array4d[wp.uint32]) – Color output from
update(), shape(world_count, camera_count, height, width).out_buffer (wp.array3d[wp.uint8] | None) – Pre-allocated RGBA buffer. If None, allocates a new one.
worlds_per_row (int | None) – Tiles per row in the grid. If None, picks a square-ish layout.
- flatten_depth_image_to_rgba(image, out_buffer=None, worlds_per_row=None, depth_range=None)#
Flatten rendered depth image to a tiled RGBA buffer.
Encodes depth as grayscale: inverts values (closer = brighter) and normalizes to the
[50, 255]range. Background pixels (no hit) remain black.Deprecated since version 1.1: Use
SensorTiledCamera.utils.flatten_depth_image_to_rgbainstead.- Parameters:
image (wp.array4d[wp.float32]) – Depth output from
update(), shape(world_count, camera_count, height, width).out_buffer (wp.array3d[wp.uint8] | None) – Pre-allocated RGBA buffer. If None, allocates a new one.
worlds_per_row (int | None) – Tiles per row in the grid. If None, picks a square-ish layout.
depth_range (wp.array[wp.float32] | None) – Depth range to normalize to, shape
(2,)[near, far]. If None, computes from image.
- flatten_normal_image_to_rgba(image, out_buffer=None, worlds_per_row=None)#
Flatten rendered normal image to a tiled RGBA buffer.
Arranges
(world_count * camera_count)tiles in a grid. Each tile shows one camera’s view of one world.Deprecated since version 1.1: Use
SensorTiledCamera.utils.flatten_normal_image_to_rgbainstead.- Parameters:
image (wp.array4d[wp.vec3f]) – Normal output from
update(), shape(world_count, camera_count, height, width).out_buffer (wp.array3d[wp.uint8] | None) – Pre-allocated RGBA buffer. If None, allocates a new one.
worlds_per_row (int | None) – Tiles per row in the grid. If None, picks a square-ish layout.
- sync_transforms(state)#
Synchronize triangle-mesh points from the simulation state.
update()calls this automatically when state is not None.Shape and particle BVHs on
modelmust be built once viabuild_bvh_shape()andbuild_bvh_particle()before first use. Before later frames that change geometry, refit them viarefit_bvh_shape()andrefit_bvh_particle()prior to callingupdate().- Parameters:
state (newton.State) – The current simulation state containing particle positions.
- update(state=None, camera_transforms=None, camera_rays=None, *, color_image=None, depth_image=None, shape_index_image=None, normal_image=None, albedo_image=None, clear_data=DEFAULT_CLEAR_DATA, refit_bvh=None, hdr_color_image=None)#
Render output images for all worlds and cameras.
Each output array has shape
(world_count, camera_count, height, width)where element[world_id, camera_id, y, x]corresponds to the ray incamera_rays[camera_id, y, x]. Each output channel is optional – pass None to skip that channel’s rendering entirely.Shape and particle BVHs on
modelmust be built once for the initial state viabuild_bvh_shape()andbuild_bvh_particle()before first use. Before later frames that change geometry, refit them for state viarefit_bvh_shape()andrefit_bvh_particle()before calling this method.- Parameters:
state (State | None) – Simulation state with body and particle transforms. Passing
Noneis deprecated and will be removed in a future release.camera_transforms (wp.array2d[wp.transformf] | None) – Camera-to-world transforms, shape
(camera_count, world_count).camera_rays (wp.array4d[wp.vec3f] | None) – Camera-space rays from
compute_pinhole_camera_rays(), shape(camera_count, height, width, 2).color_image (wp.array4d[wp.uint32] | None) – Output for RGBA color. None to skip.
depth_image (wp.array4d[wp.float32] | None) – Output for ray-hit distance [m]. None to skip.
shape_index_image (wp.array4d[wp.uint32] | None) – Output for per-pixel shape id. None to skip.
normal_image (wp.array4d[wp.vec3f] | None) – Output for surface normals. None to skip.
albedo_image (wp.array4d[wp.uint32] | None) – Output for unshaded surface color. None to skip.
clear_data (ClearData | None) – Values to clear output buffers with. See
DEFAULT_CLEAR_DATA,GRAY_CLEAR_DATA.refit_bvh (bool | None) – Refit the BVH before rendering. This is deprecated, use
build_bvh_shape(),refit_bvh_shape(),build_bvh_particle(), andrefit_bvh_particle()explicitly before calling this method instead.hdr_color_image (wp.array4d[wp.vec3f] | None) – Output for linear HDR color. None to skip.
- DEFAULT_CLEAR_DATA = ClearData(clear_color=0, clear_depth=0.0, clear_shape_index=4294967295, clear_normal=(0.0, 0.0, 0.0), clear_albedo=0)#
- GRAY_CLEAR_DATA = ClearData(clear_color=4284900966, clear_depth=0.0, clear_shape_index=4294967295, clear_normal=(0.0, 0.0, 0.0), clear_albedo=4278190080)#
- property render_config: RenderConfig#
Low-level raytrace settings on the internal
RenderContext.Populated at construction from
Configand from fixed defaults (for example global world and shadow flags on the context). Attributes may be modified to change behavior for subsequentupdate()calls.- Returns:
The live
RenderConfiginstance (same object asrender_context.configwithout triggering deprecation warnings).
- property render_context: RenderContext#
Internal Warp raytracing context used by
update()and buffer helpers.Deprecated since version 1.1: Direct access is deprecated and will be removed. Prefer this class’s public methods, or
render_configforRenderConfigaccess.- Returns:
The shared
RenderContextinstance.