My thoughts on the future of AI compute: GPUs, TPUs, LPUs, and heterogeneous systems

I've been doing some reading about ASICs for AI hardware and recently watched Julia Turc's video on GPUs, TPUs, and LPUs. It got me thinking about where AI compute is heading over the next several years.
My broad prediction is that GPUs aren't going anywhere, but they will increasingly become one component in a much more heterogeneous compute ecosystem.
The reason is that we're beginning to see different workloads develop very different hardware requirements. General-purpose accelerators are incredibly useful, but there's an opportunity to extract substantially more performance, efficiency, and responsiveness by specializing hardware around particular operations.
I think this will become increasingly apparent across three areas: inference, training, and agentic workloads.

1. Inference: specialization around different computational bottlenecks
   Inference is probably where I expect the most obvious hardware specialization to happen.
   Even within LLM inference, prefill and decode have very different computational characteristics. Prefill processes the input context in parallel and is generally more compute-intensive. Decode generates tokens sequentially and, particularly at low batch sizes, is often constrained by memory bandwidth and latency.
   These differences create opportunities to use different hardware for different parts of the inference pipeline.
   GPUs remain extremely attractive because of their flexibility, parallel compute capabilities, and large HBM capacity. They are particularly useful for high-throughput workloads and operations that benefit from substantial memory capacity and parallelism.
   TPUs take a different approach. They use specialized matrix-processing hardware, systolic-array architectures, and a compiler-oriented execution model to optimize neural network workloads. Google's latest eighth-generation TPU lineup is especially interesting because it explicitly separates large-scale training (TPU 8t) from high-concurrency reasoning and serving (TPU 8i). Google has also introduced specialized TPU 8tx and TPU 8i Max systems for confidential inference and extremely large-context workloads.
   Then there's Groq's LPU architecture, which takes specialization further. Its design emphasizes SRAM, predictable data movement, and compiler-scheduled execution. Rather than relying on the same dynamic scheduling model as a GPU, the compiler can plan computation and data movement ahead of execution. This makes the architecture particularly interesting for low-latency token generation.
   What's especially interesting is NVIDIA's integration of Groq LPUs into its Vera Rubin platform. The architecture combines Rubin GPUs and Groq LPUs, with GPUs handling operations such as attention over large KV caches and LPUs accelerating latency-sensitive feed-forward and MoE operations.
   This is a particularly important development because it goes beyond simply assigning prefill to one chip and decode to another. We're starting to see different components of the same inference iteration executed on different types of accelerators.
   My perspective is that inference will become increasingly specialized and disaggregated. We could see different hardware pools for prefill, decode, attention, feed-forward networks, speculative decoding, and memory or KV-cache management.
   I don't necessarily think every workload will benefit from this arrangement. Data movement, communication overhead, and scheduling complexity can easily eliminate the gains from specialization. But I expect more systems to be designed around matching individual operations to the hardware best suited for them.
2. Training: specialization, but not necessarily full disaggregation
   Training is where I'm considerably more skeptical about broad disaggregation.
   The core training loop has a lot of dependencies. We have the forward pass, backward pass, gradient computation, optimizer updates, and the synchronization required to keep distributed workers consistent.
   Then there's the distributed coordination problem. Tensor parallelism, data parallelism, pipeline parallelism, and expert parallelism all introduce different communication requirements. All-reduce, all-gather, reduce-scatter, and all-to-all aren't incidental operations. They're integral parts of the computation.
   I think this makes training fundamentally different from inference. There are opportunities to separate certain operations, but breaking the training system into independently operating computational services introduces synchronization, communication, and orchestration costs that can quickly become counterproductive.
   Where I see a much more interesting opportunity is specializing the coordination machinery itself.
   For example, Google Ironwood's SparseCores can offload collective communication operations such as all-gather and reduce-scatter, allowing the primary TensorCores to continue computation. In Google's published reproduction of Olmo 3 7B pretraining, SparseCore collective offloading combined with XLA optimizations increased reported model FLOPs utilization from 41% to 44.5%, without changing the training objective.
   NVIDIA is pursuing a similar direction. NCCL has introduced GPU-initiated networking and copy-engine-based collectives, allowing GPUs to initiate communication directly and dedicated hardware to handle certain data transfers instead of consuming GPU compute resources.
   Then there's NVLink 6 SHARP, which brings in-network computation into the fabric itself. Portions of collective operations such as all-reduce, reduce-scatter, and all-gather can execute inside the network switches rather than requiring every GPU to perform all of the associated work.
   Google's eighth-generation TPU 8i takes this idea even further with a dedicated Collectives Acceleration Engine (CAE), designed to accelerate reductions and synchronization. Google reports that it reduces on-chip collective latency by 5x compared with its previous-generation design.
   I find this direction particularly interesting.
   My prediction is that training will become increasingly heterogeneous at the component level, but the core training process will remain relatively tightly coordinated. We will see more specialized hardware for communication, memory movement, synchronization, and potentially other parts of the optimization process.
   I'm less convinced that the entire training loop will be broadly disaggregated into independent systems. The coordination and orchestration requirements are simply too substantial.
   I suspect the bigger opportunity is to make the coordination machinery itself more specialized and hardware accelerated, rather than trying to distribute every component of training across a different processor.
3. Agentic workloads: compute extends beyond model inference
   I think agents introduce an entirely different dimension to the hardware discussion.
   With traditional inference, we're primarily concerned with running a model and generating tokens. With agents, model inference becomes one component of a much larger execution pipeline.
   An agent might need to reason, call tools, execute code, search files, interact with APIs, manage state, retrieve context, launch subprocesses, and coordinate other agents.
   The model still needs GPUs or specialized inference accelerators, but much of the surrounding execution is fundamentally different.
   This creates a much larger role for CPUs, memory, storage, networking, and potentially DPUs and SmartNICs.
   CPUs are particularly important for tool execution, sandboxing, orchestration, and managing the many sequential and branching operations that make up an agent's execution. These workloads can be latency-sensitive and highly irregular, rather than the large, relatively structured matrix operations that GPUs excel at.
   NVIDIA's Vera CPU and its broader Vera Rubin platform are interesting examples of this direction. NVIDIA is explicitly positioning CPUs for agent orchestration, tool execution, and sandboxed computation, alongside GPUs and other accelerators.
   Google's latest TPU portfolio is also increasingly designed around agentic workloads, with different systems targeting training, reasoning, confidential execution, and long-context processing.
   My own speculation is that we may eventually see more sophisticated forms of predictive execution around agents.
   For example, while an inference accelerator is generating tokens, the CPU or another execution engine could predict likely upcoming tool calls, prepare the relevant sandbox, preload files, establish connections, or allocate resources in advance.
   Some safe and reversible operations might even be executed speculatively before the model confirms the next step. Obviously, irreversible operations would require appropriate safeguards and confirmation.
   The broader idea is that we should not think of an agent as simply a model running on a GPU. It's an entire distributed execution environment, with very different components competing for resources and introducing their own bottlenecks.
   I expect agentic workloads to accelerate the demand for heterogeneous systems substantially.
4. The overlooked piece: networking and coordination
   There's one additional area I think deserves considerably more attention: the interconnect and the hardware responsible for coordinating all these processors.
   InfiniBand and NVLink are both important parts of this picture, but they operate at different levels.
   NVLink is NVIDIA's high-bandwidth scale-up fabric, connecting GPUs within a system or rack. InfiniBand is commonly used for scale-out networking between nodes across a larger cluster. Ethernet is another important option for scale-out infrastructure.
   These fabrics are not simply interchangeable networking technologies. They provide different communication domains and help determine how efficiently a collection of accelerators can operate as one system.
   But I think the opportunity goes beyond increasing bandwidth.
   We have traditionally relied heavily on software libraries, runtimes, compilers, and host-side processing to coordinate distributed computation. XLA, TVM, NCCL, and similar systems are essential for determining how operations are scheduled and executed across hardware.
   However, we're increasingly seeing hardware take on some of these responsibilities directly.
   Dedicated communication engines, GPU-initiated networking, in-network aggregation, and specialized collective engines are examples of this transition.
   My perspective is that this will become a major area of hardware innovation. Rather than simply building faster accelerators, we will increasingly build systems where communication, synchronization, and data movement are accelerated independently of the primary compute units.
   I don't think this means software orchestration disappears. Compilers and runtimes will remain essential for deciding what should happen and when. But more of the actual execution machinery can be pushed into dedicated hardware.
5. Where I think all of this is heading
   My broader thesis is that AI compute will become heterogeneous across several dimensions.
   GPUs will remain important because generality and programmability have enormous value. TPUs and other ASICs will continue to specialize around particular computational patterns. LPUs and similar architectures will target low-latency inference and predictable execution.
   But the specialization won't stop at the accelerator.
   We'll see increasingly specialized memory systems, communication engines, networking fabrics, CPUs, DPUs, and other infrastructure designed around the actual requirements of AI workloads.
   In other words, I think the unit of optimization is gradually shifting from the individual chip toward the entire compute system.
   The interesting engineering problem is no longer just how to maximize FLOPs. It's how to minimize data movement, reduce synchronization overhead, keep compute units utilized, manage memory efficiently, and coordinate increasingly complex workloads within a constrained power budget.
   And I suspect future improvements will come as much from rethinking how these components work together as from making any individual accelerator faster.
   These are still speculative predictions, not claims that the industry has already converged on this architecture. But the direction of current hardware development makes me think heterogeneous compute will become increasingly important as AI workloads continue to evolve.