We wouldn't build a GPU today

The GPU was designed to render triangles for video games. It now runs the AI economy. But from qubits to thermodynamic chips, hardware is specializing by physics again.

A transistor is noisy analog matter forced to behave like a perfect digital bit. Enforcing that abstraction costs energy.

And we are not even using the energy we spend well.

It used to be cheap to throw compute at problems. In 2018-2022, I spent three months moving bits at the bare metal and saved $2 million a year in compute costs. Most companies did not bother because cloud instances were cheap and engineering time was expensive. But at the scale we were working, it mattered.

The amount of data only increased. Then AI started squeezing us. But there’s only so much we can win on better software.

The Silicon Data team benchmarked 3,500 GPUs across 11 cloud providers and found the “same” chip is not the same chip: memory bandwidth on identical H200s varied by up to 38%, compute on identical H100s by 34.5%, enough that two buyers renting the same model could see nearly 1.8x difference in tokens per dollar. And that variance is on top of what we already accept when we read manufacturer benchmarks: getting 50% of a GPU’s advertised FLOPs is doing okay, 70% is doing great. Where does it go? It goes to kernel efficiency, data movement, and memory access patterns rather than the silicon being broken (Chip Huyen, AI Engineering, 2024). The obvious next step: if utilization is 50%, fix the software before buying new physics. Orchestration and kernel work. Compilation.

At SuperCompute 25, people told me they have entire racks of GPUs sitting idle (maybe one version behind, since hardware lasts longer than the next generation ships), while at every AI conference, companies say hardware scarcity is killing them. The scarcity is real and the waste is also real. And we waste at the model level too, throwing frontier models at everything, like a trillion-parameter model answering “hi” to open your 5-hour token window once you wake up for the day (guilty). As Marvin the Paranoid Android put it: “Here I am, brain the size of a planet, and they tell me to take you up to the bridge.”

But a recent paper on inference economics finds something counterintuitive: there is an optimal cluster size for a given model, and past that point, performance can degrade because communication between devices dominates the workload. V100 to A100 to H100, tokens per second at fixed cost improve by only about 25% per generation. The bottleneck to fast decoding is not compute but latency, with network and NVLink latency the biggest culprits. Memory bandwidth matters more for long context inference; network bandwidth matters more for short context.

But where do we get outsized compute returns? The physics.

The GPU was designed to render triangles for video games. It got repurposed for matrix multiplication because that was the closest operation the hardware could do, and then the entire AI industry was built on top of that. If you were designing compute for AI inference, molecular dynamics, quantum chemistry, or combinatorial optimization from scratch today, you would not build the GPU.

Over 50% of DOE supercomputing time goes to simulating quantum mechanics on classical hardware (NERSC/LBNL, 2025). DFT alone eats 30.9% of all cycles. The UK’s national supercomputer has materials science as its single largest workload at 42%.

DFT is itself a cheap approximation: the method scientists settle for because exact treatment of electron correlation scales exponentially. The 50% measures what researchers can afford to run at today’s cost per simulation, not necessarily what they want to run. The 2D Fermi-Hubbard model, strongly correlated materials, the systems behind battery cathodes and high-temperature superconductors, mostly go unsimulated.

The semiconductor industry’s real achievement was not transistor speed, but layer on layer of abstraction, from device physics to transistor models to standard cells to compilers, and you get an industry where a single set of foundries, design tools, and talent serve every chip ever made. That made the explosion of the ecosystem possible, and the ecosystem drove cost down.

But the abstraction has a cost.

Making compute faster through different math and physics

The approaches range from conventional silicon running different number systems to devices that change the physics of the substrate itself. The further from digital you go, the bigger the potential efficiency gain for that problem, and the more of the ecosystem has to be rebuilt. A quantum processor does not simulate quantum mechanics; it is quantum mechanics. The qubit is the electron, the atom, the photon. The physics of the device matches the physics of the problem. A Perspective I co-authored calls these physics-based ASICs: thermodynamic chips, optical processors, memristive arrays, oscillator networks, each using its native dynamics as the computation rather than suppressing them.

Inference ASICs are the closest to conventional silicon. They still enforce the digital abstraction, but they strip away the generality, no graphics pipeline, no branch prediction, and the latest iterations focus on memory movement. A 70B-parameter model requires moving roughly 140 GB of weights per generated token; at 8 TB/s on a B200, that caps out around 57 tokens per second at theoretical peak, with the GPU waiting on data most of the time. The inference ASIC bet is that you fix this in the architecture. Tensordyne uses logarithmic arithmetic so multiplication becomes addition, shrinking the multiplier circuits and freeing die area for five times more SRAM than Blackwell, so the compute spends less time waiting on memory. Groq's LPU splits inference by memory type: SRAM for the decode phase where latency matters, Rubin's HBM for attention and KV cache, because NVIDIA concluded one chip can't serve both (maybe also to keep it from eating their market). Cerebras keeps the model on one enormous die. Etched hard codes the transformer. Taalas hard wires the weights, a chip that runs exactly one model and nothing else. The point is, every step trades market breadth for tokens per watt. The product in every case ships as a system, not just a chip.

Thermodynamic and probabilistic chips go further. They stop enforcing the digital abstraction at the device level and use the physics of the substrate as the computation. Normal Computing taped out CN101 in August 2025. Thermal fluctuations sample from probability distributions, potentially having applications for speeding up generative AI. Extropic's Z1 enters early access with networks of probabilistic bits at room temperature; their blueprint paper lays out the full stack on superconducting circuits driven by thermal noise, and they recently got $75M from the Department of Commerce. Both companies claim energy efficiency gains up to 1,000x. Of course, they are very early handwave-y estimates — more like goals, vendor-stated, on targeted workloads, against baselines the vendors chose, and no independent benchmark for this hardware category exists yet, with no integration considerations. See: the quantum benchmarking article.

Even if the claims don’t hit the 1000x, the potential has people excited. Some of these substrates may also be waiting for the workloads that need them: post-transformer architectures built on diffusion and energy-based models play to thermodynamic hardware’s strengths in ways.

Analog in-memory computing skips the bus. In a conventional chip, data moves between separate memory and processing units, and the movement is the bottleneck. Analog chips store the neural network weights directly as physical properties of the memory elements, tiny resistances set at each point in a grid, and the matrix multiplication happens through circuit physics when you send a signal through. IBM's analog AI chip does this with phase change memory. Mythic does it with flash memory, shipping analog processors for edge AI.

Optical processors send light through a chip and the matrix multiplication happens as the photons travel, no transistors switching. Oscillator networks solve optimization problems by physics rather than iteration: set up a system that naturally settles into the best answer instead of searching for it. NTT has run machines on this principle for years. And at the research frontier, the Penn groups of two of my co-authors have built electrical networks that learn without a processor, components that adjust their own resistance without any chip computing gradients, the learning itself performed by the physics.

Quantum processors seem even narrower, until you understand we live in a quantum world at small scales. A classical computer simulating a molecule through the Schrödinger equation burns energy enforcing digital logic on a natively quantum mechanical problem.

None of this physics is new. Analog computing predates digital. Ising machines trace to the 1980s. What changed is that the economics of digital silicon degraded enough, and the dominant workloads became probabilistic enough for the alternatives to be the economically viable option.

The ENIAC era is back

The device physics is genuinely hard for several of these, and years of work remain on the science and engineering, not to mention the ecosystem around the device, the test infrastructure, the design tools, the supply chains, and the benchmarks.

Before the GPU won, computing was specialized by physics. ENIAC was a specific machine for specific problems. Cray built vector processors for weather modeling. Signal processing ran on DSPs. The digital abstraction layer won because it let one architecture serve every workload, and the ecosystem that grew around it crushed every specialized competitor on cost and adoption.

We have seen this cycle break before when a boatload of money was dangling as a carrot. Bitcoin mining ran the specialization sequence very quickly: you could mine a bitcoin on a CPU in 2009, GPU in 2010, FPGA in 2011, then we needed to turn to purpose-built ASICs in 2013.

Everyone is still buying GPUs because they have to. But there is a shift in compute, with the call even coming from inside the house. NVIDIA holds roughly 80% of data center AI silicon, but the market share is going down. And NVIDIA bought Groq for $20 billion, 2.9x Groq’s last round, because it wanted the LPU inference technology inside its own AI factory. Cerebras IPO’d at $56 billion fully diluted. Custom ASICs are growing at 44.6% versus 16.1% for GPUs.

And the switching cost that was supposed to lock everyone in is weakening. A year ago, every investor I talked to had the same objection: nobody will rewrite their CUDA to a new architecture. The quote I heard from the quantum side: “everyone hates CUDA-Q, but it’s the best we got”. NVIDIA’s response, cuTile in 2025, built specifically as a response to the tile-based programming model. Each step down the list from GPU to TPU to QPU to physics-based ASIC trades more generality for more efficiency, and each step requires more of the ecosystem to be rebuilt.

The ecosystem needed to make alt compute stick

NVIDIA played the 20 year CUDA flywheel game. Libraries and developer tools created a lock in so deep that a chip with better specs could not compete because rewriting it was a pain. If it’s ain’t broke, don’t fix it. That worked for 20 years. Now Triton and LLM-based kernel translation (DRTriton, CodeRosetta) are showing that the switching costs are easier.

The computer best suited to the problem should be the computer that gets used. But if the software does not exist to program it, if the abstractions strip away the efficiency gains at the interface, if there is no compiler, no debugger, no PDK, no second-source supplier, then the physics advantage dies.

For example, a quantum computer is a system, and the system is mostly classical. The qubit, a patterned piece of superconducting metal, a trapped atom, a photon in a waveguide, costs almost nothing. The stack around it is expensive.

The dilution refrigerator that takes a superconducting chip to millikelvin runs on helium-3, an isotope so scarce that Maybell Quantum contracted with Interlune in May 2025 to source thousands of liters a year from lunar regolith starting in 2029 (of course, we don’t know contract terms - may be a non-binding MOU). Superconducting qubits need two to four coaxial cables per qubit. Trapped ions need laser systems. Every modality needs real-time error correction run on GPUs, and interconnects to join modules into one logical machine (at least, that’s the new roadmap for most of the leading modalities, though again, some new types of quantum computers, like electrons on helium and majoranas, say that if their physics works, the scaling is the easier part).

And no one can even really estimate costs well - the published cost estimates for a fault-tolerant machine span seven orders of magnitude, from a few million dollars to tens of trillions for the same computation. That tells you how many assumptions loaded into each financial model.

And the same “theoretical peak vs achieved performance” gap from the GPU world shows up in quantum, except worse. When we were testing our 8 qubit chips at Berkeley Lab, coherence time plummeted every day at 3 PM. We were baffled. Turns out, a window high up in the building let sunlight hit the water tubes running to the chiller at just the right angle to raise the temperature enough to matter. That is how delicate quantum systems are! The 38% performance spread on identical GPUs is a rounding error compared to the environmental sensitivity of a quantum processor, which makes the orchestration and system integration layer even more critical for quantum than it already is for AI.

The AI inference market is running workload changes at 100x the speed

Through 2024, training dominated the narrative, the capex, and the hardware design. Training rewards peak throughput and interconnect scale: massive parallelism across thousands of accelerators, huge memory bandwidth, a bounded compute campaign that runs for a defined number of steps and finishes. The entire GPU data center architecture, NVLink, InfiniBand, the NVL72 rack itself, was shaped by training’s requirements. Then the workload flipped. Deloitte estimated that inference accounted for half of all AI compute in 2025, rising to two-thirds in 2026, up from one-third in 2023. AWS and NVIDIA have both stated that inference accounts for as much as 90% of the cost of large-scale AI workloads over their lifetime, because inference never stops running.

And inference rewards the opposite hardware virtues: cost and latency per token, not peak throughput. The constraint is memory, and NVIDIA’s own roadmap confirms that Rubin delivers 5x more inference compute than Blackwell, but HBM bandwidth only scales 2.8x and capacity 1.5x. The compute keeps outrunning the memory that feeds it, which is why the metric that matters became tokens per watt, and why a hardware generation optimized for training keeps disappointing you on inference, but idle time for expensive hardware just eats into data center margins and raises costs more.

Every one of these devices has to live somewhere, and the somewhere is another constraint. We are still living in the NVIDIA data center designed world. They shipped a product most existing facilities are structurally and electrically unable to host, and the market is retrofitting billions of dollars of infrastructure because, what else are they going to buy?

Each step toward specialization buys another multiple of tokens per watt and narrows the market the chip can serve, and the companies that win will be the ones whose specialization matches where the workload actually can be used, the facility in can fit in, and play the game of filling capacity the best.

This is the question quantum has yet to answer: does the new compute fit into the data center as it exists, or does the data center get redesigned around the new compute? I sat on the ground in a lab on Heinz Avenue in Berkeley waiting for the train to come by to see what the vibrations would be like. A neutral-atom machine needs optical stability. But the same category of problem now applies to liquid cooled Blackwells or a Cerebras chip.

And the scale of spending on the supply side should concern anyone who remembers 1999. NERC projects 224 GW of new summer peak demand by 2035, the highest growth rate since tracking began in 1995. In the late 1990s, the EIA projected over 300 GW of needed generation. NRG, Calpine, Mirant, and Enron raced to build it, adding roughly 175 GW in five years. The dot com bubble burst, the demand did not materialize, and all four went bankrupt. Today, against a 16 GW announced pipeline for 2026, only about 5 GW is under construction, with 30-50% of planned projects expected to slip or cancel, and $2 billion campuses idle waiting on $40 million transformer orders. Amsterdam banned new data centers until 2030. Dublin blocked grid connections for four years. Singapore approves new builds only through competitive application.

The supply side answer is: build more generation — we all have to tokenmaxx, right? Imagine getting our Fable taken away after we spent weeks nurturing it?

The demand side answer is: waste less per computation, through hardware and software. The industry is pouring almost everything into the first one, but with things shifting fast, a breakthrough in alt compute can carry an overbuild risk like already happened in the dot com era.

The prep work for integration is already happening

Whatever the computation eventually costs, the machines will be bought from whoever can deliver them, at prices set by the holders of the bottleneck layers.

The interconnect layer is already being built for all of them. Copper fails at 224 Gbps per lane because the physics of electromagnetic propagation will not let it cross a server rack. Lightmatter raised $850 million and joined NVIDIA’s NVLink Fusion ecosystem. Marvell acquired Celestial AI for $3.25 billion. DARPA’s HARQ program is building photonic interconnect as the backbone for combining different qubit types.

Nobody debates the cost of a transistor across seven orders of magnitude. Not because transistor economics are simple, but because the semiconductor industry built the monster machinery and forced everyone into certain processes.

Every physics-native substrate needs the same missing infrastructure, and the companies that support the growth of the entire ecosystem. The physics can be ready and the product still years away because the ecosystem has to come up together, every layer at once, and the layers are expensive and boring and largely the same layers regardless of which device physics wins. That shared layer gets paid regardless of which device wins. The next compute era arrives when its ecosystem gets built and the math makes sense, but it needs to be done together.