Latent Space in 3 minutes

Here is The Daily FM summary of the Latent Space that aired on Wednesday July 8th. The episode featured Modal CTO Akshat Bubna, alongside cofounder Vibhu, in a wide-ranging conversation about how AI infrastructure is changing as agents, inference, and bursty GPU workloads become central to modern software. [1]

Akshat began by revisiting Modal’s origin story. The company did not start as a GPU inference platform, but as an attempt to build a better runtime than Kubernetes for compute-heavy, bursty workloads. The key idea was that developers should not have to manage piles of YAML or Kubernetes configuration just to run jobs with custom images, accelerators, or fast scaling. Modal’s answer was “self-provisioning workloads”: infrastructure requirements live next to the code, often as decorators, so the system can spin up the right compute automatically. [2]

A major theme was the shift from developer experience to what Akshat called agent experience. Modal has apparently reorganized its SDK thinking around how AI agents use infrastructure. His argument was simple: if it is painful for humans to read and modify hundreds of Kubernetes files, it is also painful for agents. Typed, colocated infrastructure definitions are easier for coding agents to modify, test, and observe. The hosts pushed on whether this still matters if humans are not reading code as much anymore, and Akshat said observability may now be even more important: agents can change code, but humans still need dashboards, logs, and judgment to understand what happened.

The conversation then moved into Modal’s current identity. Akshat described it as a cloud platform with primitives built from scratch for AI applications, covering inference, training, batch processing, and sandboxes. He emphasized that Modal is not trying to replace always-on web hosting. Its sweet spot is specialized compute that scales up and down fast, across GPUs, CPUs, regions, and unusual workloads.

One of the most interesting sections covered sandboxes. Modal built sandbox APIs as early as 2023, before coding agents had fully taken off, and even used an early “small developer” agent loop as an example. Back then, models tended to diverge after a handful of iterations. In hindsight, the hosts joked, the winning move would have been to collect all those failures, build benchmarks and RL environments, and turn them into a billion-dollar agent company. [3]

Akshat also explained that Modal’s biggest use case today is elastic inference for custom models, especially in audio, video, robotics, and computational biology. Companies like Suno and Runway may train models elsewhere but use Modal to deploy and autoscale them. Modal has gone deep on cold starts, GPU snapshotting, and regional autoscaling, because production inference is not just “find a GPU and run a model”; it involves tail latency, reliability, request delivery, and fast scaling across regions. [4]

A technical highlight was speculative decoding. Akshat explained that a smaller draft model predicts tokens ahead, while the larger model verifies them. If many drafted tokens are accepted, inference can become two to four times faster without quality loss. Modal’s open-source DFlash work uses block-based speculation, and its new Auto Endpoints aim to make optimized open-model serving easier while still letting users eject into full code when they need customization.

The episode also touched on distributed training, RDMA networking, private IPv6 overlay networks, and why Modal runs across 17 cloud and neocloud providers rather than owning data centers. Akshat framed Modal as a “supercloud” software layer, with its own reliability and scheduling layer on top of many providers.

Near the end, the discussion broadened to AI infrastructure trends: agent sandboxes, CI for coding agents, continual learning, auto-research, robotics, drug discovery, and whether future video generation may be orchestrated by agents rather than single video models. The final takeaway was that Modal’s bet on developer experience has evolved naturally into agent experience, and that the infrastructure primitives that once seemed niche now look central to the next generation of AI products. Thank you for listening to Latent Space in 3 minutes from The Daily FM. See you tomorrow!