Create, Design, and Deploy Robotics Applications Using New NVIDIA Isaac Foundation Models and Workflows

The application of robotics is rapidly expanding in diverse environments such as smart manufacturing facilities, commercial kitchens, hospitals, warehouse logistics, and agricultural fields. The industry is shifting towards intelligent automation, which requires enhanced robot capabilities to perform functions including perception, mapping, navigation, load handling, object grasping, and intricate assembly tasks.

AI plays a pivotal role in this evolution, enhancing robotic performance. By integrating NVIDIA AI acceleration, robots can tackle complex tasks with greater precision and efficiency, unlocking their full potential across diverse applications.

At COMPUTEX, we announced several new features to help roboticists and engineers build smart robots. These include:

NVIDIA Isaac Perceptor, a new reference workflow for autonomous mobile robots (AMRs) and automated guided vehicles (AGVs).
NVIDIA Isaac Manipulator offers new foundation models and a reference workflow for industrial robotic arms.
NVIDIA Jetson for Robotics, with new updates in NVIDIA JetPack 6.0.
NVIDIA Isaac Sim 4.0 brings NVIDIA Isaac Lab, a lightweight app for robot learning.

Video 1. The world’s leaders in robot development are adopting NVIDIA Isaac for research, development and production of next generation AI-enabled robots.

NVIDIA Isaac Perceptor

AMRs and AGVs are critical to assembly line efficiency, material handling, and healthcare logistics. As these robots navigate complex and unstructured environments, the ability to perceive and react to their surroundings becomes essential.

Isaac Perceptor, built on top of the NVIDIA Isaac Robot Operating System (ROS), empowers original equipment manufacturers (OEMs), freight service providers, software vendors, and the AMR ecosystem to accelerate development for robotics. Teams can equip mobile robots with perception capabilities for successful navigation and obstacle avoidance in unstructured environments.

Isaac Perceptor early collaborators include industry leaders in warehousing/ intralogistics to automakers, industrial robotics manufacturing companies, and robotic solution providers, such as ArcBest, BYD Electronics, Gideon, KION, Kudan, idealworks, RGo, and Teradyne Robotics.

Key features of Isaac Perceptor

Isaac Perceptor provides features to deliver multi-camera, 3D surround-vision capabilities for AI-based autonomous mobile robots.

Multi-camera AI-based depth perception

Isaac Perceptor processes 16.5M depth points per second per camera at 30 Hz. The stereo disparity is calculated from a time-synchronized image pair sourced from a stereo camera and is used to produce a depth image or a point cloud for a scene. An efficient semi-supervised deep neural network (ESS DNN) provides a GPU-accelerated package for DNN-based stereo disparity.

Side-by-side images showing a forklift captured from a camera and AI-learned predictions of obstacles such as forklift tines. — *Figure 1. ESS DNN detecting obstacles at 5m*

Multi-camera visual inertial odometry

Isaac ROS Visual SLAM provides ROS 2 packages for visual simultaneous localization and mapping (VSLAM) and visual odometry (VO). This is based on the NVIDIA CUDA Visual SLAM (cuVSLAM) library and provides robust navigation with less than 1% translation error while navigating in featureless environments.

Navigating environments with sparse visual features or repetitive patterns presents a well-known challenge for VSLAM solutions. This can be mitigated by fusing input from multiple viewpoints. In the latest update, cuVSLAM incorporates concurrent visual odometry estimation from multiple stereo cameras.

Our testing indicated a marked improvement. Robots consistently achieved their navigation goals using multiple cameras, compared to less than 25% with a single camera.

VO method	Runtime
cuVSLAM	5 ms
FRVO, S-PTAM	30 ms
ORB-SLAM2	60 ms

Table 1. Performance comparison of cuVSLAM with FRVO, S-PTAM, and ORB-SLAM2. cuVSLAM shows accelerated performance in robot navigation using multiple cameras

Learn more on the GitHub page.

A GIF that shows how a robot is navigating using multiple camera angles. — *Figure 2. Isaac ROS Visual SLAM with one camera compared to two cameras and then four cameras*

Real-time, multi-camera voxel grid mapping

At the core of the Isaac Perceptor is nvblox, the CUDA-accelerated 3D reconstruction library that can identify obstacles up to five meters away to provide a 2D costmap and update them in under 300 ms.

Isaac ROS nvblox provides ROS 2 packages for 3D scene reconstruction and local obstacle costmap generation for navigation. This package can be used for stationary environments and scenes with people and mobile objects.

What’s new in this release is multiple-camera support for expanded coverage using up to three HAWK cameras, providing about a 270° field of view.

For more information, visit the Isaac ROS nvblox documentation.

A GIF showing voxel 3D reconstruction of a warehouse using Nvblox. The visualization highlights Isaac ROS Nvblox also reconstructing overhanging obstacles during navigation. — *Figure 3. Voxel 3D reconstruction using Isaac ROS Nvblox, including reconstruction of overhanging obstacles*

NVIDIA Nova Orin Developer Kit

This developer kit, featuring the NVIDIA Jetson AGX Orin, supports up to six cameras, including up to three stereo and three fisheye cameras, with intra-camera latency of under 100 microseconds.

The stereo cameras boast a resolution of 2MP per camera, with a field of view of 110X70, suitable for 3D occupancy grid mapping, depth perception, visual odometry, and people detection. Purchase a Nova Orin developer kit from Segway or Leopard Imaging to use Isaac Perceptor.

Isaac Perceptor has a reference graph supporting up to three stereo cameras on this Developer Kit. With enhanced modularity with ROS 2 packages, this release also features a reference integration with Nav2 on the Nova Carter reference robot.

Enhanced compatibility with cameras and sensors

Isaac Perceptor brings enhanced support for integration with camera and sensor partners. Orbbec successfully integrated its Gemini 335L camera with NVIDIA Isaac Perceptor components. This integration is demonstrated on the NVIDIA Jetson AGX Orin using Isaac ROS Visual SLAM and Nvblox.

LIPS also successfully integrated its AE450 camera with the Isaac Perceptor component, Nvblox.

NVIDIA Isaac Manipulator

Isaac Manipulator is a workflow of NVIDIA-accelerated libraries and AI models. It enables developers to bring AI acceleration to robotic arms, or manipulators, that can seamlessly perceive, understand, and interact with their environments.

Its foundation models and accelerated libraries can be integrated as independent modules or as an entire workflow in solutions development. Along with independent, modular components, developers are also provided sample workflows (ROS 2 launch scripts) that combine Isaac Manipulator components for a full end-to-end reference integration.

A block diagram showing an Isaac Manipulator optimized workflow using NVIDIA components including SyntheticaDETR for object detection, FoundationPose for pose estimation, ESS depth estimation, NVBlox voxel mapping, and cuMotion for trajectory planning. — *Figure 4. An example of an Isaac Manipulator workflow leveraging NVIDIA components (in green)*

Isaac Manipulator early collaborators include robotic developer platform companies, OEMs, and ISVs/SIs, including Intrinsic (an Alphabet company), Siemens, Solomon, Techman Robot, Teradyne Robotics, Vention, and Yaskawa.

Key features of Isaac Manipulator

Isaac Manipulator brings AI features to accelerate the development of robotic arms.

cuMotion for faster path planning

This GPU-accelerated motion planner helps reduce cycle times. cuMotion is available as a plugin for the MoveIt 2 motion planning framework, an open-source project developed by an international community and led by PickNik Robotics.

cuMotion runs trajectory optimization across several seeds in parallel and returns the best solution.

A GIF showing a UR10 robot arm in motion using the NVIDIA cuMotion plugin with MoveIt2. — *Figure 5. NVIDIA cuMotion plugin to PickNik’s MoveIt 2*

Solomon, a leader in advanced vision and robotics solutions, is an Isaac Manipulator early collaborator. Their bin-picking system enhanced by Isaac Manipulator cuMotion delivered eight times faster path planning and reduced path singularity occurrences by 50% compared to conventional algorithms.

Metric	Improvement Ratio (%)
Success Rate Improvement	346.43
Move Time Reduction	55.50
Trajectory Length Reduction	42.27
Trajectory Planning Time Reduction	816.66

Table 2. Performance enhancements in Solomon’s Bin Picking System with Isaac Manipulator. Solomon experienced significant improvements in success rate, move time, trajectory length, and planning time, with reductions in path singularity occurrences. Data provided by Solomon

FoundationPose

FoundationPose is a new unified foundation model for single-shot 6D pose estimation and tracking of novel objects. This model is designed to work with high accuracy in applications that encounter previously unseen objects, without fine-tuning.

FoundationPose is currently at the top of the 2023 BOP Leaderboard for 6D localization of unseen objects. It is robust to occlusions, rapid motion, and diverse object properties like texture and scale, ensuring reliable performance across scenarios. Developers can generate realistic views of the object from any angle. Get the Foundation Pose model from GitHub.

A GIF showing pose estimation and tracking using FoundationPose. The visualization shows five objects on a table with accurate 3D poses and bounding boxes around them. — *Figure 6. Pose estimation and tracking using NVIDIA FoundationPose*

SyntheticaDETR

SyntheticaDETR is a set of Real-Time DEtection TRansformer (DETRs)-based models for single-shot, image space object detection trained on synthetic data generated using NVIDIA Omniverse. It implements a more efficient approach to traditional object detectors by predicting all objects at once using a transformer encoder-decoder architecture.

Trained on synthetic and real-world data, SyntheticaDETR is at the top of the BOP leaderboard for 2D detection for seen objects on the YCB-Video dataset (with a mean average precision of 0.885 and a mean average recall of 0.903).

These models can also detect objects as a 2D bounding box region-of-interest for pose estimators like NVIDIA FoundationPose. Download the SyntheticaDETR model and download Isaac Manipulator.

NVIDIA JetPack 6.0

NVIDIA Isaac ROS 3.0 is compatible with JetPack 6.0 and is supported on all NVIDIA Jetson Orin modules and developer kits.

Modular, API-driven services to build generative AI and robotics applications faster and easier are coming soon with NVIDIA Jetson Platform Services. These pre-built and customizable services are designed to accelerate AI application development on NVIDIA Jetson Orin system-on-modules.

NVIDIA Isaac Sim 4.0

Using Isaac Sim, developers can generate synthetic data and diverse, virtual complex test environments with industry-leading sensor and robot-type testing. This enables highly realistic simulations for testing thousands of robots simultaneously in real time.

NVIDIA Isaac Lab

Isaac Lab is a lightweight reference application built on the Isaac Sim platform and plays a pivotal role in robot foundation model training. It supports reinforcement learning, imitation learning, and transfer learning. It can train a wide range of robot embodiments, for developers to explore designs and functionalities.

The new release also provides ease of use with VSCode integration with a compatibility checker, multi-GPU support for reinforcement learning, performance improvements with RTX sensor tiled rendering, optimized cache, and shader management.

Additional new features in Isaac Sim include:

Ease of use with PIP installation and a wizard for importing robots and more.
Improved performance with up to 80% faster synthetic data generation (SDG)
New SDG formats that support the COCO format and custom writer for pose estimation.
ROS 2 launch support with an end-to-end workflow and better performance for image-based publishers.
More built-in robots support: including Universal Robots UR20 and UR30 and Boston Dynamics Spot. There’s also a host of humanoids including the 1X Neo, Unitree H1, Agility Digit, Fourier Intelligence GR1, Sanctuary A1 Phoenix, and XiaoPeng PX5.