newton.sensors.RaycastSensor#

class newton.sensors.RaycastSensor(model, camera_position, camera_direction, camera_up, fov_radians, width, height, max_distance=1000.0)[source]#

Bases: object

Raycast-based depth sensor for generating depth images.

The RaycastSensor simulates a depth camera by casting rays from a virtual camera through each pixel in an image. For each pixel, it finds the closest intersection with the scene geometry and records the distance as a depth value.

The sensor supports perspective cameras with configurable field of view, aspect ratio, and resolution. The resulting depth image has the same resolution as specified, with depth values representing the distance from the camera to the closest surface along each ray.

Camera Coordinate System

The camera uses a right-handed coordinate system where: - The forward direction (camera_direction) is the direction the camera is looking - The up direction (camera_up) defines the camera’s vertical orientation - The right direction (camera_right) is computed as the cross product of forward and up

Depth Values

  • Positive depth values: Distance to the closest surface

  • Negative depth values (-1.0): No intersection found (ray missed all geometry)

device#

The device (CPU/GPU) where computations are performed

camera_position#

3D position of the camera in world space

camera_direction#

Forward direction vector (normalized)

camera_up#

Up direction vector (normalized)

camera_right#

Right direction vector (normalized)

fov_radians#

Vertical field of view in radians

aspect_ratio#

Width/height aspect ratio

width#

Image width in pixels

height#

Image height in pixels

depth_image#

2D depth image array (height, width)

__init__(model, camera_position, camera_direction, camera_up, fov_radians, width, height, max_distance=1000.0)#

Initialize a RaycastSensor.

Parameters:

  • model (Model) – The Newton model containing the geometry to raycast against

  • camera_position (tuple[float, float, float] | ndarray) – 3D position of the camera in world space

  • camera_direction (tuple[float, float, float] | ndarray) – Forward direction of the camera (will be normalized)

  • camera_up (tuple[float, float, float] | ndarray) – Up direction of the camera (will be normalized)

  • fov_radians (float) – Vertical field of view in radians

  • width (int) – Image width in pixels

  • height (int) – Image height in pixels

  • max_distance (float) – Maximum ray distance; rays beyond this return no hit

eval(state)#

Evaluate the raycast sensor to generate a depth image.

Casts rays from the camera through each pixel and records the distance to the closest intersection with the scene geometry.

Parameters:

state (State) – The current state of the simulation containing body poses

get_depth_image()#

Get the depth image as a 2D array.

Returns:

  • Positive: Distance to closest surface

  • -1.0: No intersection found

Return type:

2D depth image array with shape (height, width). Values are

get_depth_image_numpy()#

Get the depth image as a numpy array.

Returns:

Numpy array with shape (height, width) containing depth values. Values are the same as get_depth_image() but as a numpy array.

Return type:

ndarray

point_camera_at(target, position=None, up=None)#

Point the camera at a specific target location.

Parameters:
update_camera_parameters(fov_radians=None, width=None, height=None, max_distance=None)#

Update camera intrinsic parameters.

Parameters:

  • fov_radians (float | None) – New vertical field of view in radians

  • width (int | None) – New image width in pixels

  • height (int | None) – New image height in pixels

  • max_distance (float | None) – New maximum ray distance

update_camera_pose(position=None, direction=None, up=None)#

Update the camera pose parameters.

Parameters: