Advancing to AI's next frontier: Notes from a talk

This discussion between Jeff Dean and Bill Dally at GTC 2026 covers recent advances, challenges, and future directions in machine learning, with a focus on large language models, hardware-software co-design, agentic systems, and AI’s societal impacts.

In summary, the conversation underscores the synergistic evolution of ML models, hardware architectures, and AI-driven design processes that are collectively pushing the boundaries of AI capabilities. Key drivers include lowering inference latency, enabling autonomous agentic workflows, and applying AI to its own development lifecycle. Societal benefits in education and healthcare are anticipated as near-term impactful domains. However, challenges in energy efficiency, hardware flexibility, toolchain speed, and organizational scale remain active areas of focus.

Summary of the Conversation: Jeff Dean (Google) and Bill Dally (NVIDIA)

Key Highlights:

• Rapid Progress in ML Capabilities
  • One of the biggest advances in the last year is that ML models have significantly improved at tasks with verifiable rewards such as mathematics and coding.
  • For example, Google's Gemini model achieved gold medals in competitive math (International Math Olympiad) and coding (ICPC contests).
  • Emergence of agent-based workflows enables models to autonomously execute long-running tasks over hours or days, correcting themselves without constant human supervision.
• Low-Latency Inference and Hardware Innovations
  • NVIDIA is innovating to drastically reduce inference latency, aiming to increase token processing speeds from hundreds to tens of thousands per second.
  • Even inside a single GPU or AI accelerator, data constantly moves between compute units, memory blocks, and tensor cores. Much of the delay comes not from computation itself, but from routing and scheduling overhead.
  • Key latency sources are on-chip and off-chip communication delays.
    • On-chip improvements include static scheduling (where communication paths are predetermined rather than dynamically negotiated during execution) in order to to remove routing overhead and reduce latency to ~30 nanoseconds.
    • Off-chip latency, which becomes important during the communication between chips and memory systems, is dominated by physical layer (PHY) signal processing, responsible for transmitting high-speed electrical signals across interconnects. A surprising insight there was that lowering bandwidth slightly can simplify signal detection and reduce latency.
  • Future hardware could enable running large models at 10,000–20,000 tokens per second per user, critical for real-time agentic systems. Currently consumer chat systems have it in the 100 tokens/sec output.
• Agentic Systems and Self-Improvement
  • Although fully autonomous model self-improvement (e.g., retraining itself over a month) is not yet realized, initial signs exist where models run many experiments, discard poor-performing ones, and refine promising approaches.
  • This approach resembles natural language neural architecture search (NAS), allowing researchers to specify research spaces in natural language for the AI to explore.
  • This partnership between researchers and AI agents could massively accelerate research productivity.
• Predicting ML Trends and Future-Proofing Hardware
  • Hardware design faces the challenge of predicting ML model evolution over 2–5 years.
  • To mitigate these risks, hardware is designed with flexibility to support various model architectures and resource requirements (math, memory bandwidth/capacity, communication).
  • Different models (e.g., with different attention mechanisms) shift resource usage patterns, sometimes leaving parts of hardware underutilized.
  • Possible future strategy: produce hardware variants optimized for different model types.
• Scaling Models Beyond Data Limits
  • Contrary to some opinions, there is still abundant untapped data (e.g., video, audio, autonomous vehicle data).
  • Modern frontier models are now capable of generating synthetic data and although there are concerns that repeatedly training on AI-generated outputs could lead to repetition or “model collapse". The speakers argued that carefully curated synthetic data can significantly improve training efficiency when combined with high-quality real-world data.
• Many techniques developed in earlier eras of machine learning are becoming relevant again at large scale. Techniques like data augmentation, regularization (dropout, distillation), and revisiting methods from image models can enhance training efficiency and reduce overfitting in extremely large models.
• Training vs. Inference Hardware.
  • AI training and AI inference have very different computational requirements, which means future AI systems may require increasingly specialized hardware architectures. Training requires large memory capacity to store activations for backpropagation; inference discards activations immediately and is much lighter from a memory perspective.
• In addition, different stages of Inference stress hardware in different ways:
  • Prefill stage, the first stage of inference, processes the initial prompt or context window. It resembles training with parallel token processing.
  • Decode stage, where, the model generates tokens one at a time, with each new token depending on the previous output. This creates a highly sequential process, often running matrix-vector or skinny matrix multiplications, where latency becomes the dominant bottleneck.
• Future hardware could specialize into at least three types:
  • training/prefill (optimized for large-scale parallel computation)
  • decode-attention(optimized for low-latency attention operations, KV cache access)
  • decode-feedforward hardware(optimized for sequential feedforward inference and energy efficient decoding).
• Model Architecture Trends
  • Growing use of sparse models, especially Mixture of Experts (MoE): Modern frontier models are increasingly adopting sparse architectures such as Mixture of Experts (MoE), where only a subset of the model’s parameters are activated for any given token or task. Instead of every parameter participating in every computation, the model dynamically routes inputs to specialized “expert” subnetworks. This allows models to scale to trillions of parameters while keeping inference costs manageable and improving specialization across different domains or reasoning styles.
  • Sparse activation complicates batching and computation regularity: While sparse models improve efficiency, they also introduce significant systems and hardware challenges. Since different tokens may activate different experts, computation becomes irregular and difficult to batch efficiently across GPUs or accelerators. This creates communication overhead, load balancing problems, and reduced hardware utilization compared to dense models where every operation is uniform and predictable.
• Attention mechanisms are evolving to scale beyond quadratic complexity for very large token counts. Traditional transformer attention scales quadratically with sequence length, making extremely long contexts computationally expensive and memory intensive. As models move toward million-token and eventually billion-token contexts, new attention architectures are being developed to reduce this scaling burden while preserving enough contextual fidelity for reasoning and retrieval.
• To manage extremely large contexts, researchers are exploring architectures that process information in stages rather than attending uniformly to all tokens simultaneously. Techniques include:
  • chunk-based or windowed attention, where models focus locally before aggregating globally,
  • hierarchical clustering of information into summaries or semantic groups,
  • and lightweight retrieval or filtering stages that identify the most relevant portions of context before applying expensive full attention.
  • These approaches aim to make reasoning over trillions of tokens computationally feasible.
• Hierarchical or multi-stage attention mechanisms may provide “illusion” of attending to massive contexts efficiently. Future systems may not literally attend equally to every token in a gigantic context. Instead, they may simulate effective long-context understanding through layered retrieval, summarization, memory compression, and selective attention mechanisms. In practice, this could give models the functional capability of reasoning over enormous knowledge spaces while only applying intensive computation to the most relevant information at each stage.
• AI in Chip Design
  • AI is extensively used to accelerate chip design phases. AI systems are being applied to circuit layout, architecture exploration, timing optimization, verification, debugging, and even documentation assistance. This is especially important as modern chips contain tens to hundreds of billions of transistors and must optimize simultaneously for power, performance, area, thermals, and manufacturability:
    • Standard cell library porting using reinforcement learning: The porting and optimization of standard cell libraries—the foundational building blocks used in chip design is underway. Traditionally, adapting these libraries to new manufacturing nodes required months of painstaking manual engineering. AI systems such as Nvidia’s NVCell use reinforcement learning to automate large portions of this process, reducing timelines from months to overnight GPU-driven optimization runs while often producing layouts that outperform human-designed baselines in power, timing, or area efficiency.
• Optimization of carry lookahead chains with RL, producing novel, more efficient designs: Reinforcement learning systems have also been applied to optimize highly specialized logic structures such as carry lookahead chains used in arithmetic circuits. Instead of relying solely on established engineering heuristics, AI agents explore vast combinatorial design spaces and occasionally discover unconventional architectures that improve latency, routing efficiency, or transistor utilization in ways human designers may not have considered.
• Use of specialized LLMs fine-tuned on proprietary Nvidia design documents: Companies are developing domain-specific language models such as Nvidia’s ChipNeMo and BugNeMo, which are fine-tuned on internal design documentation, bug databases, architecture specifications, and engineering discussions. These systems help engineers quickly retrieve institutional knowledge, understand legacy designs, generate documentation, explain hardware behavior, and accelerate onboarding for junior engineers. They can also assist in identifying likely root causes of hardware bugs by correlating historical bug patterns and design metadata.
• Agentic systems explore architectural design spaces, running simulations to narrow down promising options.
• Design verification remains a bottleneck. While AI is accelerating many design stages, verification—the process of ensuring a chip behaves correctly under all possible conditions—remains extremely expensive and time-consuming. Verification often consumes more engineering effort than the original design itself. The long-term goal is to reduce verification cycles while maintaining reliability, especially as chips become increasingly heterogeneous and AI-accelerator-heavy. AI is now being explored to assist with:
• automated test generation,
• bug localization,
• simulation prioritization,
• formal verification support,
• and code refactoring.
• Challenges in Orchestrating Agentic Systems
  • Sparse, delayed, and sometimes unverifiable reward signals complicate training of agentic systems.
  • Latency bottlenecks arise from slow startup times of traditional tools (e.g., compilers) designed for human speed, which must be re-engineered for AI speed.
  • New tool designs are emerging to better support these fast, agent-driven workflows.
• Energy Efficiency Focus
  • Multiply-add operations consume femtojoules, whereas moving data from memory consumes orders of magnitude more energy.
  • Key to efficiency is minimizing data movement:
    • Performing many operations in SRAM close to compute units reduces energy.
    • Stacking DRAM on top of compute chips can reduce energy per bit by an order of magnitude and increase bandwidth.
  • Exploiting sparsity is promising but challenging due to loss of regularity and control overhead.
  • Numeric formats like NVFP4 with block scaling factors enable very low precision computation with effective accuracy.
• Interconnection Network Trade-offs
  • Different network topologies (2D/3D torus vs. switch-based fat trees or dragonfly) suit different workloads: In torus-style topologies, each node connects directly to a small number of neighboring nodes in a structured grid. These designs are highly efficient for workloads where communication is mostly local and predictable, since nearby accelerators can exchange data with very low latency and without involving centralized switching infrastructure. Architectures such as fat-tree or dragonfly networks use layers of high-bandwidth switches to efficiently route traffic between distant nodes across massive clusters. These designs reduce bottlenecks for globally distributed communication patterns and can provide much higher aggregate bandwidth across the entire system, albeit often at greater infrastructure complexity and cost.
  • Local communication benefits from direct connections; global communication benefits from high-radix switch fabrics. Many AI workloads exhibit strong locality characteristics, where some communication occurs frequently within small groups of accelerators while only occasional synchronization occurs globally. For example:
    • tensor-parallel workloads may require intense low-latency communication among nearby GPUs,
    • while model synchronization or parameter updates may require broader cluster-wide communication. Optimizing for these differing traffic patterns is becoming increasingly important as models and clusters scale.
  • Hybrid approaches may combine both for optimal performance depending on traffic patterns. Future large-scale AI systems may combine multiple networking approaches within the same cluster. For instance:
    • direct local interconnects could provide extremely fast communication within pods or racks,
    • while high-radix switched fabrics handle longer-range cluster-wide traffic.
• Societal Impact of AI
  • Most promising applications include:
    • Education: Personalized AI tutors can dramatically improve learning outcomes by adapting to individual needs and preferred modalities.
    • Healthcare: AI can integrate diverse data (wearables, genetics) to deliver personalized care and health coaching, potentially improving health behaviors and outcomes.
    • Scientific Discovery: Accelerating research through AI assistance.
  • Challenges include privacy, regulation, and misinformation risks.
  • Education systems should embrace AI tools rather than restrict them, analogous to calculators enabling higher-level math learning.
• Organizational and Cultural Reflections
  • Growth of companies like Google and NVIDIA introduces bureaucracy and diluted community feeling.
  • Leadership efforts can maintain startup-like culture even at large scales.
  • Larger scale enables more ambitious projects, leveraging abundant resources.

Timeline Table of Notable Points (Approximate)