How Nvidia's Research Lab Fueled Its $4T AI & Robotics Growth
When Bill Dally joined Nvidia’s research lab in 2009, it was a modest operation of about a dozen people, primarily focused on ray tracing, a sophisticated rendering technique used in computer graphics. Today, that once-small research arm has grown to over 400 individuals, playing a pivotal role in transforming Nvidia from a nineties startup known for video game GPUs into a $4 trillion powerhouse at the forefront of the artificial intelligence revolution. Now, the lab is setting its sights on developing the foundational technologies for robotics and advanced AI, with some of its pioneering work already appearing in commercial products, including a new suite of AI models, libraries, and infrastructure for robotics developers unveiled recently.
Dally, now Nvidia’s chief scientist, began consulting for the company in 2003 while still at Stanford University. A few years later, as he prepared to step down as department chair of Stanford’s computer science department, he planned a sabbatical. Nvidia, however, had a different vision. David Kirk, then heading the research lab, and Nvidia CEO Jensen Huang, believed a permanent position was a superior path. Dally recounted their “full-court press” to persuade him, an effort that ultimately succeeded. He reflected that the role became a “perfect fit” for his interests and talents, a place where he could make his most significant contribution to the world.
Upon taking the helm of the lab in 2009, Dally prioritized expansion. Researchers immediately branched out beyond ray tracing, delving into areas like circuit design and Very Large-Scale Integration (VLSI), a complex process that packs millions of transistors onto a single microchip. The lab’s growth has been continuous ever since. Dally emphasizes a strategic approach to research, constantly evaluating new and exciting areas to identify those with the highest potential to positively impact the company.
For a significant period, this strategic focus translated into building superior GPUs for artificial intelligence. Nvidia was remarkably early to the burgeoning AI field, exploring the concept of AI-optimized GPUs as far back as 2010—more than a decade before the current AI frenzy took hold. Dally recalls recognizing the transformative potential of AI early on and convincing Huang to “double down” on it. This foresight led to the specialization of their GPUs, the development of extensive supporting software, and proactive engagement with AI researchers globally, long before the mainstream recognized its relevance.
As Nvidia now commands a dominant position in the AI GPU market, the company’s research efforts have shifted to identify new frontiers beyond AI data centers. This exploration has led them squarely into physical AI and robotics. Dally envisions a future where robots become a massive global force, and Nvidia aims to be the “brains of all the robots,” necessitating the development of critical underlying technologies.
This is where Sanja Fidler, Nvidia’s vice president of AI research, comes in. Fidler joined the research lab in 2018, having already been involved in developing simulation models for robots with her team at MIT. Her work immediately captured Jensen Huang’s interest at a researchers’ reception. Fidler described the decision to join Nvidia as irresistible, citing both the compelling subject matter and a strong cultural alignment. Huang’s invitation, “come work with me, not with us, not for us,” resonated deeply.
Fidler subsequently established a research lab in Toronto, focusing on building simulations for physical AI within Omniverse, Nvidia’s platform for virtual world creation. A primary challenge in constructing these simulated environments was acquiring the necessary 3D data. This involved not only sourcing a vast volume of potential images but also developing the technology to convert these images into 3D renditions usable by simulators. Nvidia invested in “differentiable rendering,” a technology that makes rendering amenable to AI, enabling the conversion of 2D images or videos into 3D models.
Omniverse released the first iteration of its image-to-3D model, GANverse3D, in 2021, and subsequently tackled the same process for video. By utilizing videos from robots and self-driving cars, they developed the Neuric Neural Reconstruction Engine, first announced in 2022, to create these advanced 3D models and simulations. These innovations form the technological backbone of the company’s Cosmos family of world AI models, which were unveiled at CES in January.
The lab’s current focus is on significantly accelerating these models. Just as video games require real-time responsiveness, robots demand even faster reaction times. Fidler explains that a robot could process the world 100 times faster than it unfolds, meaning that making these models substantially quicker would be immensely valuable for robotic and physical AI applications. Nvidia continues to make strides in this area, recently announcing a new fleet of world AI models specifically designed for generating synthetic data to train robots, alongside new libraries and infrastructure software for robotics developers.
Despite the rapid progress and the current enthusiasm surrounding robots, particularly humanoids, the Nvidia research team maintains a realistic perspective. Both Dally and Fidler caution that the widespread presence of humanoids in homes is still several years away, drawing parallels to the hype and timeline associated with autonomous vehicles. Dally emphasizes that AI, from visual AI enabling robot perception to generative AI assisting with task and motion planning and manipulation, has been the critical enabler. As individual challenges are overcome and the volume of training data for neural networks expands, the capabilities of these robots will continue to grow exponentially.