The demand for real-time insights and autonomous decision-making is growing across industries, and healthcare and medical devices are no exception. Relying on real-time edge AI, the next generation of healthcare promises to deliver more precise treatments, improve patient outcomes, and increase operational efficiencies.
Operating rooms of the future, for example, will increasingly incorporate devices that are AI-enabled and interconnected, giving real-time access to and exchange of holistic patient data, operative insights, decisions, and actions.
In such a future, software as a medical device (SaMD) must operate under strict requirements, handling large-scale data with stringent performance and latency constraints while deployed in distributed healthcare systems. This requires interoperability to ensure efficient, reliable, and secure data connectivity and exchange between various sensors, displays, controls, and applications without compromising performance or latency.
This post demonstrates how to integrate NVIDIA Holoscan and RTI Connext to create AI-powered medical device applications with high interoperability, low latency, and distributed connectivity. This integration achieves these benefits with minimal overhead and reduced implementation effort and complexity.
NVIDIA Holoscan for real-time AI sensor processing
NVIDIA Holoscan provides developers with a singular production-ready framework for building an end-to-end real-time AI sensor processing pipeline, from sensor data ingress to accelerated computing and AI inferencing, real-time visualization, actuation, and data stream egress.
This comprehensive solution effectively addresses the multitude of edge AI development challenges. Holoscan ensures optimal application performance while abstracting away development complexities, reducing time to market, and offering the convenience of coding in Python and C++, all in a low-code, high-performance infrastructure. In brief, Holoscan is a framework for connecting high-speed I/O to GPU in a modular, scalable, software-defined, and GPU-centric fashion.
RTI Connext for real-time data-centric connectivity
RTI Connext, based on the Data Distribution Service (DDS) standard, streamlines connectivity across complex and scalable systems with a distributed and real-time software communication framework. With Connext, Holoscan applications can integrate with distributed data sources and applications with little overhead while also minimizing the implementation effort and complexity of achieving the performance, reliability, and security that such healthcare systems require.
Connext provides real-time information exchange between complex system components, while enabling stringent reliability, cybersecurity, and performance requirements. Connext provides the framework to process, analyze, and act on high-volume, real-time data with low latency in a redundant, fault-tolerant architecture. Medical systems built on Connext are resilient, self-forming, and self-healing with no single point of failure.
The wide range of quality of service tuning options helps meet the need for real-time video and correlated data in distributed, intelligent surgical systems. Connext also includes capabilities like automatic discovery and security without adding large bandwidth overhead. Built-in security based on the proven DDS Security standard provides the foundation for authentication and encryption, but also security logging and granular access control. This keeps critical systems safe from security breaches, and meets cybersecurity requirements enforced by regulatory agencies such as the FDA.
Integrating NVIDIA Holoscan and RTI Connext
Today’s healthcare system is built on numerous installed legacy systems that weren’t originally designed with AI capabilities in mind and where NVIDIA Holoscan is not currently natively supported. In addition, today’s healthcare systems and medical devices require complex connectivity between sensors, actuators, control systems, and interfaces.
The integration of Connext with Holoscan enables Holoscan developers to transform an existing legacy install base to AI-enabled and SW-defined devices. This works through integration of Holoscan as a sidecar (companion compute module) to those devices where Holoscan is not natively supported.
For example, a significant portion of existing medical devices are currently Windows-based (particularly in the domain of medical imaging), where Holoscan is not natively supported. As a second example, Holoscan as a sidecar could bring advanced AI capabilities into robotic surgery systems running real-time operating systems (RTOS) on non-NVIDIA systems. A third example is low-end sensory medical devices such as patient monitoring, where devices could be augmented with powerful AI algorithms while the hardware or software making up the legacy system would otherwise limit the addition of such new capabilities.
Holoscan DDS interoperability through RTI Connext DDS offers a solution to all three of these examples, providing a scalable, AI-enabled Holoscan sidecar that seamlessly communicates with the legacy systems in real time.
Holoscan offers exceptional infrastructure for GPU-accelerated software as a medical device (SaMD), enabling the innovation and deployment of AI-powered workflows within next-generation healthcare systems. These workflows generally need to operate on large-scale data, with very strict latency restrictions. As such, it’s imperative that data can be passed between various sensors, displays, controls, and Holoscan applications in an efficient, reliable, and secure manner.
By using RTI Connext, Holoscan applications can integrate with distributed healthcare systems with little overhead while also minimizing the implementation effort and complexity of achieving the performance, reliability, and security that such systems require. In cases where Connext is already being used, introducing new Holoscan-powered AI workflows may even be possible without modifications to the existing system.
Example Holoscan application with RTI Connext integration
This section provides an example use case; specifically, a Holoscan application running on a dedicated system that acts as a sidecar. The application reads frames from a DDS databus using RTI Connext, processes the frame data within a Holoscan workflow, and then publishes the results back to the databus with Connext such that the processed frame data can be read by another device for display.
This example enhances a common scenario in healthcare systems in which multiple sensors capture data that is then aggregated for display on a separate monitoring system. Adding AI-powered Holoscan workflows into the middle of this data flow can often be done with little modification to the existing components. The sensor acquisition and display output is also often done using systems that would not natively support Holoscan. Connext helps to bridge these gaps.
The core components of this example are the Holoscan DDS video streaming operators that are available through nvidia-holoscan/holohub on GitHub. These operators enable Holoscan applications to read and write video frames from a DDS databus in real time. With these operators, Holoscan applications can read video frames from the databus (to be used as the source for workflow processing), and write processed results back to the databus (for consumption by another component).
In addition, the following new application support shows the sidecar use case as a stand-alone example:
- The dds_video application can be used to either write video frames to a DDS databus (captured from a V4L2 video device, such as a USB camera), or to read frames from DDS and render them to the display through Holoviz, the Holoscan visualization module.
- The body_pose_estimation application has been modified such that the input video frames can originate from a DDS databus and the output video frames, which include the body pose estimation overlay, can be published back to a DDS databus.
Combining these two applications demonstrates the sidecar dataflow using three processes:
- One dds_video process captures frames from the camera sensor and publishes them to DDS.
- A body_pose_estimation process receives the input sensor frames from DDS, processes the frames through the body pose estimation model, then outputs the frames with the inference results overlaid on top of the images to DDS.
- Another dds_video process receives the processed frames and renders them to the display.
Figure 3 shows the setup for this example. Note that each of the three processes can be run on any system, as long as they are discoverable to each other on the same DDS domain. For example, through a multicast-capable network.
To run this example locally, start by reading the HoloHub DDS Operators documentation regarding the dependency requirements for setting up RTI Connext. To learn how to build and run the applications, see the DDS Support section of the Body Pose Estimation documentation.
Summary
Integrating NVIDIA Holoscan with RTI Connext offers Holoscan developers in the medical device industry numerous advantages in the transition to AI-enhanced systems and devices. These include seamless integration with distributed healthcare systems with minimal overhead, the enhancement of legacy systems with advanced AI algorithms, and more.
To get started, download Holoscan 2.0 and check out the Holoscan and DDS integration reference applications at nvidia-holoscan/holohub, including:
- DDS Base Operator
- DDS Video Operators
- DDS Video Application
- Body Pose Estimation Application (featuring DDS Support)
Visit the NVIDIA Developer Holoscan forum to ask questions and share information.
