Script
Here is The Daily FM summary of the Latent Space that aired on Thursday October 1st. MIT PhD student Alex Zhang joined Swyx and Vibhu for a wide-ranging conversation about a central idea in modern AI: today’s frontier models may be far more capable than their interfaces, harnesses, and agent systems allow us to see. [1]
Zhang began with GPU Mode and KernelBench, communities and benchmarks built around writing high-performance GPU kernels. While AI now generates many leaderboard-winning kernels, he argued that human expertise still matters enormously. The hard part is not only proposing fast code, but verifying that it is stable and reliable in real systems rather than gaming a benchmark. A knowledgeable engineer can steer a model toward the right insight, potentially replacing billions of tokens’ worth of blind search. The same lesson, Zhang said, applies to math, coding, and agents: experts are not obsolete; they become powerful verifiers and directors. [2]
That theme carried into his advice for researchers. Academia’s advantage, he argued, is not competing directly with frontier labs on compute. It is taking weird, ambitious bets that industry may initially dismiss as trivial or useless. He cited ideas such as SWE-bench, ReAct, Quiet-STaR, and his own Recursive Language Models, or RLMs, which were initially met with skepticism before shaping how the field thinks about agents. [3]
RLMs are not a new base-model architecture. They are a harness design: the model stores context outside its prompt, in a filesystem or code environment, and uses code to call tools and recursively spawn subagents. Instead of cramming an entire long task into one ever-growing conversation, the system breaks work into smaller, locally manageable problems. Zhang’s claim is that this produces compositional generalization: a strategy learned on short tasks can extend to much longer tasks, or even transfer across apparently unrelated domains, because the high-level program stays similar. [4]
He thinks many popular coding-agent harnesses—Claude Code, Codex, and Pi—are structurally more alike than they appear. Their differences may matter less than finding genuinely new harness abstractions. Prime Agent, a project built with Prime Intellect, pushes the RLM approach by making IPython its central tool, externalizing memory to disk, allowing agent-to-agent communication, and supporting persistent subagents that survive beyond a single interaction. [5]
The conversation then turned to swarms. Zhang believes the future “language model” users query may secretly be a large scaffold or swarm of agents, while the user sees only a clean interface. OpenAI’s reported experiment—10,000 agents running for 88 hours, producing 130 billion output tokens and solving difficult mathematical problems—was impressive precisely because it showed that enough search plus shared context can uncover solutions. But Zhang cautioned that most swarm activity may be wasted exploration. The unsolved challenge is getting agents to converge efficiently rather than simply burning tokens. [6]
A striking related point involved newer fast models such as GEV, which challenge the assumption that every language model must be an autoregressive text generator. If a task only needs a quick binary classification or calibrated decision, generating hundreds of tokens is wasteful. Zhang expects more models optimized for distinct output spaces, latency requirements, and roles inside larger agent systems. [7]
His final thesis was “capability overhang.” Frontier models may already be smart enough to automate many long-running but ordinary knowledge-work tasks; the missing ingredient is a reliable general-purpose harness that lets them learn, use tools, maintain context, and persist over time. For Zhang, that is both a research opportunity and a call for more unusual, high-conviction experiments. [8]
Thank you for listening to Latent Space in 3 minutes from The Daily FM. See you next time!
- Latent Space: Academia is for Ambition — Alex Zhang, MIT
...exploring how much capability we’re leaving on the table by wrapping increasingly powerful models in primitive systems. RLMs took over the timeline early this year: and an RLM based harness was the first to ~solve ARC-AGI-3 before OpenAI’s Astra: and is even today, influencing new research that has more extreme implications than RLMs: We go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open-ende...
- Latent Space: Academia is for Ambition — Alex Zhang, MIT
...language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open-ended research at Sakana AI, speculative programmatic tool calling, capability overhang, Neuralese, and where Alex thinks the next big research opportunities may lie. We discuss: * Why AI-generated GPU kernels still leave substantial room for human expertise * How one expert insight can potentially replace enormous amounts of brute-force token search * Why PhD students
- Latent Space: Academia is for Ambition — Alex Zhang, MIT
...creasingly powerful models in primitive systems. RLMs took over the timeline early this year: and an RLM based harness was the first to ~solve ARC-AGI-3 before OpenAI’s Astra: and is even today, influencing new research that has more extreme implications than RLMs: We go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open-ended research at Sakana AI, speculative programmatic tool calling, capabili...
- Latent Space: Academia is for Ambition — Alex Zhang, MIT
...ng is exploring how much capability we’re leaving on the table by wrapping increasingly powerful models in primitive systems. RLMs took over the timeline early this year: and an RLM based harness was the first to ~solve ARC-AGI-3 before OpenAI’s Astra: and is even today, influencing new research that has more extreme implications than RLMs: We go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open...
- Latent Space: Academia is for Ambition — Alex Zhang, MIT
...ew research that has more extreme implications than RLMs: We go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open-ended research at Sakana AI, speculative programmatic tool calling, capability overhang, Neuralese, and where Alex thinks the next big research opportunities may lie. We discuss: * Why AI-generated GPU kernels still leave substantial room for human expertise * How one expert insight...
- Latent Space: Academia is for Ambition — Alex Zhang, MIT
...and is even today, influencing new research that has more extreme implications than RLMs: We go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open-ended research at Sakana AI, speculative programmatic tool calling, capability overhang, Neuralese, and where Alex thinks the next big research opportunities may lie. We discuss: * Why AI-generated GPU kernels still leave substantial room for human exp...
- Latent Space: Academia is for Ambition — Alex Zhang, MIT
...ature the work of Alex Zhang of MIT. From GPU kernels and KernelBench to Recursive Language Models , Mismanaged Geniuses , and massive multi-agent swarms, Alex Zhang is exploring how much capability we’re leaving on the table by wrapping increasingly powerful models in primitive systems. RLMs took over the timeline early this year: and an RLM based harness was the first to ~solve ARC-AGI-3 before OpenAI’s Astra: and is even today, influencing new research that has more extreme implications than RLMs: We go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language mod...
- Latent Space: Academia is for Ambition — Alex Zhang, MIT
...go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open-ended research at Sakana AI, speculative programmatic tool calling, capability overhang, Neuralese, and where Alex thinks the next big research opportunities may lie. We discuss: * Why AI-generated GPU kernels still leave substantial room for human expertise * How one expert insight can potentially replace enormous amounts of brute-force token...
