NVIDIA Boosts Robotics with New Omniverse AI Tools & Models

NVIDIA has unveiled a suite of new tools and AI models under its Omniverse and Cosmos platforms, designed to significantly accelerate the development and deployment of advanced robotics solutions. These innovations, powered by the latest NVIDIA RTX PRO Servers and DGX Cloud, aim to empower developers to create physically accurate digital twins, reconstruct real-world environments within simulations, generate synthetic data for training physical AI models, and build intelligent agents capable of understanding the physical world.

According to Rev Lebaredian, vice president of Omniverse and simulation technologies at NVIDIA, the convergence of computer graphics and artificial intelligence is poised to fundamentally transform the robotics industry. He emphasized that by combining AI reasoning with scalable, physically accurate simulation, NVIDIA is enabling the creation of future robots and autonomous vehicles that will revolutionize industries valued in trillions of dollars.

Among the key announcements are new NVIDIA Omniverse software development kits (SDKs) and libraries, now available for building and deploying industrial AI and robotics simulation applications. These SDKs introduce crucial data interoperability between MuJoCo (MJCF) and Universal Scene Description (OpenUSD), opening up seamless robot simulation across platforms for over 250,000 MuJoCo robot learning developers. Further enhancing realism, new Omniverse NuRec libraries and AI models introduce Omniverse RTX ray-traced 3D Gaussian splatting, a sophisticated rendering technique that allows developers to capture, reconstruct, and simulate the real world in 3D using sensor data.

Complementing these advancements, the open-source robot simulation and learning frameworks, NVIDIA Isaac Sim 5.0 and NVIDIA Isaac Lab 2.2, are now accessible on GitHub. Isaac Sim now integrates NuRec neural rendering and new OpenUSD-based robot and sensor schemas, helping developers bridge the challenging gap between simulation and real-world performance. The impact of NuRec is already being seen, with its integration into CARLA, a prominent open-source simulator utilized by over 150,000 developers, and its adoption by autonomous vehicle toolchain leader Foretellix, which is leveraging NuRec, NVIDIA Omniverse Sensor RTX, and Cosmos Transfer to enhance its physically accurate synthetic data generation. Data engine specialist Voxel51’s FiftyOne, used by companies like Ford and Porsche, also supports NuRec for streamlined data preparation. Major players such as Amazon Devices & Services, Boston Dynamics, Figure AI, Hexagon, RAI Institute, Lightwheel, and Skild AI are already adopting Omniverse libraries, Isaac Sim, and Isaac Lab to accelerate their AI robotics development.

Beyond simulation, NVIDIA’s Cosmos World Foundation Models (WFMs), which have seen over 2 million downloads, enable developers to generate diverse training data for robots at scale using various prompts. New models unveiled at SIGGRAPH promise significant improvements in synthetic data generation speed, accuracy, language support, and control. Notably, Cosmos Transfer-2, soon to be released, will simplify prompting and accelerate photorealistic synthetic data creation from 3D simulation scenes or spatial control inputs. A distilled version of Cosmos Transfer further enhances speed, reducing a 70-step distillation process to a single step, allowing the model to run at unprecedented speeds on NVIDIA RTX PRO Servers. Companies like Lightwheel, Moon Surgical, and Skild AI are already utilizing Cosmos Transfer to expedite physical AI training by simulating a wide range of conditions at scale.

A significant breakthrough in world understanding comes with NVIDIA Cosmos Reason, a new open and customizable 7-billion-parameter reasoning Vision Language Model (VLM) designed for physical AI and robotics. Unlike previous VLMs that excelled at object recognition but struggled with multi-step tasks or ambiguity, Cosmos Reason allows robots and vision AI agents to reason more like humans, leveraging prior knowledge, physics understanding, and common sense to interpret and act in the real world. Its applications span data curation and annotation, enabling automated, high-quality preparation of massive datasets; robot planning and reasoning, acting as the intelligent core for deliberate decision-making in robot vision language action (VLA) models; and video analytics AI agents for extracting insights and performing root-cause analysis on large volumes of video data. NVIDIA’s own robotics and DRIVE teams are employing Cosmos Reason for data curation, filtering, annotation, and VLA post-training, while Uber uses it for annotating and captioning autonomous vehicle training data. Magna is integrating Cosmos Reason into its City Delivery platform to help autonomous vehicles adapt more quickly to new urban environments. Additionally, VAST Data, Milestone Systems, and Linker Vision are adopting Cosmos Reason to automate traffic monitoring, enhance safety, and improve visual inspection in both urban and industrial settings.

To support these demanding workloads, NVIDIA has also announced new AI infrastructure. The NVIDIA RTX PRO Blackwell Servers offer a unified architecture for every robot development task, from training and synthetic data generation to robot learning and simulation. Furthermore, NVIDIA DGX Cloud, now available on Microsoft Azure Marketplace, provides Omniverse developers with a fully managed platform, simplifying the streaming of OpenUSD- and NVIDIA RTX-based applications at scale from the cloud, thereby minimizing infrastructure orchestration and management burdens. Accenture and Hexagon are among the first industry leaders to adopt this platform.

To further cultivate the developer ecosystem, NVIDIA is launching an OpenUSD Curriculum and Certification program, addressing the growing demand for USD expertise with support from AOUSD members and industry leaders. They are also engaging in an open-source collaboration with Lightwheel, integrating robot policy training and evaluation frameworks into NVIDIA Isaac Lab, complete with parallel reinforcement learning capabilities, benchmarks, and simulation-ready assets for robot manipulation and locomotion.