State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Channel #490

In this podcast episode, Lex Fridman hosts a deep and technical conversation with machine learning researchers Nathan Lambert and Sebastian Raschka, exploring the current landscape and future of artificial intelligence in 2026. They discuss state-of-the-art developments in large language models (LLMs), coding automation, scaling laws, the rapidly evolving AI ecosystems in China and the United States, agent-based AI systems, GPU hardware dominance, and the perennial debate surrounding artificial general intelligence (AGI) and superintelligence (ASI). The discussion weaves through both cutting-edge technical details and broader societal implications, blending expert insights with accessible explanations.

The Global AI Landscape: China Vs. United States

The podcast opens by framing the so-called "DeepSeek moment" in early 2025, when the Chinese company DeepSeek released their DeepSeek R1 model, an open-weight LLM that surprised many with near state-of-the-art performance while being cheaper and less compute-intensive. This event has ignited intense competition and innovation in both China and the U.S., accelerating developments across research and product domains.

Sebastian and Nathan emphasize that "winning" in AI is a multi-dimensional concept. Technologies are not proprietary in the traditional sense, as knowledge and ideas flow freely through frequent movement of researchers between institutions. Instead, budget, hardware access, and infrastructure become crucial differentiators. Chinese companies are aggressively pursuing open-weight models, fueling an ecosystem of startups like MiniMax and z.AI, who are producing compelling open models, rivaling DeepSeek's early dominance.

The U.S. ecosystem remains strong with players like OpenAI, Anthropic, and Google pushing proprietary models and platforms optimized for broad consumer use. While many Chinese models emphasize openness to gain international influence and ease security concerns among users wary of Chinese APIs, American companies often leverage closed models with deep integration and large, paying user bases, particularly in enterprise. The lack of clear-cut "take-all" winners reflects the diverse market needs and business models in play.

Open Weight Models and Architectural Innovations

The guests delve into the proliferation of open-weight LLMs globally, particularly highlighting prominent Chinese models such as DeepSeek, MiniMax, z.AI's GLM, and Qwen, alongside Western open models like OLMo from the Allen Institute, Mistral AI, and NVIDIA's NeMo. The open model ecosystem has grown rapidly over 2024-2025, fostering access and learning despite huge costs involved in training and scaling.

Architectural adaptations define many of these models' strengths: mixture of experts (MoE) layers enable very large parameter models with efficient compute by sparsely activating experts relevant to a task. Attention mechanism variants—group query attention, sliding window, multi-head latent attention—improve memory and compute trade-offs for long contexts. Some models pursue linear-scaling attention or substitutes inspired by state space models to reduce inference costs on ultra-long inputs.

Despite these innovations, Sebastian stresses that fundamentally these architectures are still transformers evolved from the GPT-2 architecture, with mostly incremental tweaks rather than radical breakthroughs. The large gains in capabilities over the past years owe significantly to data quality improvements, synthetic data generation, and innovative training techniques rather than radical architectural departures.

Scaling Laws: Pre-Training, Post-Training, and Inference

A substantial part of the discussion centers on scaling laws, the predictable power-law relationships connecting compute, data, model size, and resulting model performance. While classic scaling laws describe pre-training, newer variants highlight intriguing avenues in post-training (i.e., reinforcement learning) and inference-time compute scaling.

Sebastian and Nathan emphasize that while scaling pre-training still yields improvements, there are diminishing returns and extraordinary costs — often multi-million or billion-dollar levels just to train the largest models. The cost of inference at scale (serving hundreds of millions of users) increasingly dominates financial outlays. As such, labs now balance investments across pre-training, mid-training on specialized datasets (e.g., long context or domain-specific data), and post-training methods like Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR).

RLVR, exemplified by DeepSeek's milestone, enables models to improve by generating answers, assessing correctness, and iteratively reinforcing accurate problem-solving rather than blindly memorizing answers. This has unlocked advanced reasoning, code generation, and tool use capabilities at scale beyond what pre-training alone enabled.

The burst of progress from inference-time scaling, where models invest more compute generating detailed chains of thought before answering, adds another dimension to these scaling frontiers. While tensor computation strategies (FP8, FP4) and improved distributed training frameworks accelerate training speed, algorithmic improvements and better curated datasets remain crucial low-hanging fruit to squeeze out performance gains.

Programming and Code Generation with LLMs

Both guests disclose their personal preferences and workflows for programming assistance with LLMs, using a variety of tools such as OpenAI Codex in VSCode, Anthropic's Claude Code, and startups like Cursor. The contrast between micromanagement of token-by-token generation outputs and higher-level agentic design support reflects evolving programming paradigms accelerated by AI.

They highlight the importance of user interface design, model style, tool integration, and memory features in shaping the programmer experience. Debugging remains a key pain point and benefit zone for AI assistance, with Grok-4 Heavy gaining recognition as especially adept in complex debugging tasks.

Programmers often balance AI-assisted code writing and reviewing to maintain control and retain valuable learning through struggle, emphasizing that too much reliance risks atrophy of skills and enthusiasm for problem-solving.

Reinforcement Learning Post-Training: RLHF and RLVR

The podcast thoroughly explains RLHF, where human feedback shapes model behavior, and RLVR, which scales reward-based learning with instructive grading in verifiable domains like math and code.

RLVR dynamically scores model-generated answers against verifiable correct answers, allowing models to discover reasoning behaviors and step-by-step problem-solving. This approach unlocks emergent capabilities unprecedented in earlier transformer-only training approaches.

Despite some skepticism due to data contamination concerns in benchmarks like QuEN, RLVR demonstrates significant accuracy improvements with relatively limited post-training compute investment, highlighting its potential as a foundational technique.

RLHF remains vital for tuning models toward user-aligned preferences, style, transparency, and ethical guardrails but lacks the same linear scalability of RLVR.

Continual Learning and Memory in LLMs

Continual learning—the ability for models to quickly assimilate ongoing feedback by weight updates—is recognized as a critical milestone toward adaptable AI capable of on-the-job learning resembling human employees.

However, its high computational cost and engineering challenges constrain widespread implementation. In contrast, in-context learning (feeding increasingly large, curated text contexts to models at inference) offers a more immediate, albeit limited, mechanism for personalization and adaptation.

Techniques like low-rank adaptation (LoRA) enable lightweight fine-tuning, balancing efficiency and retention without catastrophic forgetting.

Long context windows continue to improve steadily, but still face trade-offs between memory size, compute demands, and quality. Hybrid attention mechanisms and recursive model designs are promising research directions to extend effective context length without prohibitive computational cost.

Alternative Architectures: Text Diffusion Models and Beyond

The podcast touches on growing interest in alternatives to autoregressive transformers, notably text diffusion models inspired by successes in image generation (e.g., Stable Diffusion).

Text diffusion models generate all tokens in parallel by iteratively denoising corrupted text, potentially offering faster inference and better computational efficiency for some tasks like large code diffs. However, challenges remain in quality parity with autoregressive models and applying diffusion to tool use or reasoning problems, currently limiting their general applicability.

Hybrid approaches, recursive language models that decompose complex tasks into solvable sub-tasks, and enhanced value function modeling promise further breakthroughs in inference efficiency and problem-solving abilities.

Robotics, Agents, and Embodied AI

The guests briefly discuss the synergies between language models and robotics. While LLM progress indirectly benefits robotics research through improved software tooling and data efficiencies, robotics faces unique challenges tied to physical embodiment, real-world uncertainty, and safety that make fully autonomous, learned robots for homes a longer-term prospect.

Robotic foundation models and world models — simulators that internally model physical environments — are areas of exploration aimed at accelerating automation. The real applied opportunities may lie more in industrial and commercial robotics initially, such as logistics and warehousing automation.

Safety in robotics and responsible adoption loom large as critical concerns, with the complexity and consequences of embodied failures far exceeding those in abstract language tasks.

Business, Ecosystem, and AI Industry Evolution

The podcast explores ongoing startup consolidation, valuation concerns, IPO hesitations, and the dynamics between open and closed AI ecosystems.

Open-weight model development in the U.S. is gaining renewed government and institutional support to compete with China's open model ecosystem. The "Adam Project" at the Allen Institute for AI exemplifies coordinated efforts to build genuinely open LLMs to sustain U.S. AI leadership and innovation.

Public listings of AI companies are scarce, and there is speculation on whether giants like OpenAI, Anthropic, or xAI will IPO or remain venture-backed amid massive capital inflows and high expenditures.

GPU hardware dominance by NVIDIA is credited largely to Jensen Huang's visionary leadership, product ecosystem, and the CUDA developer platform that entrenches their position despite increasing competition and innovation attempts on chip design.

The Evolution of AI Education and Research Careers

Sebastian and Nathan share perspectives on education, recommending that beginners build LLMs from scratch at small scale to deeply understand transformers and inference. The complexity rapidly escalates with model size and distributed training, making open-weight projects and libraries like Hugging Face's Transformers more suitable once foundational skills are mastered.

Pathways to contributing research vary: open model evaluation is accessible with modest compute; novel algorithmic or architectural research often requires vast resources and collaboration with elite labs. Career decisions depend on individual preferences regarding intellectual freedom, publishing, compensation, and cultural environment in academia, industry, or startups.

The "struggle" inherent in learning AI architectures and concepts is underscored as essential to genuine mastery, with LLMs seen as complementary learning aids rather than shortcuts.

AGI, ASI, and the Future of Intelligence

The conversation reflects diverse opinions on defining AGI and ASI, with many agreeing on milestones like AI systems capable of replacing remote human workers or automating research and programming at superhuman levels.

Predictions for timelines vary, with estimates ranging into the 2030s or beyond. The "singularity" is envisioned not as a single event but a jagged, uneven progression of systems excelling in specialized tasks rather than monolithic mastery.

The promise of AI vastly increasing human productivity and democratizing knowledge access is tempered by caution toward risks, uneven adoption, and challenges in replicating general intelligence, creativity, and consciousness.

Societal Impacts, Ethics, and Safety

The guests discuss how AI, while improving access to knowledge and automating routine tasks, also poses potential harms, including misinformation, mental health impacts, inequality, and concentration of economic power.

They emphasize the importance of human agency in steering AI development and usage, highlighting ongoing challenges in defining preferences, managing biases, and ensuring responsible deployment.

The evolving human-AI relationship will place new premiums on physical, face-to-face interactions and authentic experiences amidst an increasing flood of AI-generated "slop" or low-value content.

The necessity for transparent business models, clear distinctions between AI-generated content and advertising, and cautious integration of AI into workflows is discussed as early experiments with monetization via ads loom.

The Path Ahead

While no definitive conclusions on when or how AGI or ASI might emerge are reached, the discourse expresses optimism that incremental, sustained progress in scaling laws, reinforcement learning, tool use, model architectures, and open ecosystems will continue shaping AI's trajectory.

The importance of collaborative ecosystems, diverse funding, and ethical governance accompanies recognition of the key role singular figures like Jensen Huang and Sam Altman have played in accelerating AI's revolution.

In totality, the episode offers a sweeping and nuanced survey of AI's technical advances, industry shifts, cultural debates, and the philosophical complexities framing humanity's unfolding partnership with machine intelligence.