Nvidia is building a chip that is not a GPU. Following its GTC 2026 unveiling, the company set a second-half-2026 launch for the Groq 3 LPU, a Language Processing Unit aimed purely at inference, and updated its data-center roadmap around it. It ships in liquid-cooled LPX racks of 256 LPUs with 128GB of on-chip SRAM and 640 TBps of scale-up bandwidth. The quiet admission underneath the spec sheet is significant: the company that sells the world's GPUs is conceding that GPUs are not the best tool for serving models once they are trained.

  • The LPU is a dedicated inference processor, not a general-purpose GPU, optimized for streaming tokens out of a trained model with low latency.
  • It leans on massive on-chip SRAM (128GB per rack config) instead of slower off-chip HBM, the memory bottleneck that limits how fast GPUs generate text.
  • It arrives in LPX racks of 256 chips with 640 TBps scale-up bandwidth, sold as a rack-scale inference appliance.
  • It sits alongside the GPU roadmap (Vera Rubin this year, Feynman later), so Nvidia is now selling separate silicon for training and for serving.
Training GPU versus inference LPU GPUs train models using large off-chip HBM memory; the LPU serves trained models fast using on-chip SRAM. GPU LPU general purpose inference only Job: train the model Memory: off-chip HBM Strength: raw throughput Weakness: token latency Job: serve the model Memory: on-chip SRAM Strength: low latency 640 TBps scale-up Training is a one-time cost; inference runs forever. Nvidia now sells a chip for each. genztech.blog
Fig 1 A GPU is built to train models and leans on large but slower off-chip HBM. The LPU is built to serve them, using fast on-chip SRAM to cut the latency of generating each token.

What is an LPU and why now?

Training a model and running it are different jobs with different bottlenecks. Training is throughput-bound: you push enormous batches of data through the chip and you care about total compute. Inference, the part that happens every time a user sends a prompt, is latency- and memory-bound: you generate one token at a time, and each token has to shuttle the model's weights through memory. GPUs use high-bandwidth memory (HBM) that sits off the chip, and that trip is the throttle on how fast text streams out. The LPU's answer is to keep the working set in on-chip SRAM, which is far faster to reach, so tokens come out quicker and more predictably. The name Groq is not a coincidence: this is Nvidia productizing the SRAM-heavy, inference-first design that the startup Groq popularized.

RelatedNvidia's RTX Spark Chip Storms the Windows PC Market

Why is Nvidia competing with its own GPUs?

Because the shape of AI spending is changing. For two years the money went into training ever-larger models, and GPUs owned that. But as those models get deployed into products used by millions, inference becomes the dominant, recurring cost, running forever instead of once. If a rival can serve tokens cheaper and faster with specialized silicon, that is exactly where Nvidia is most exposed, because inference is more commoditizable than training. Rather than defend GPU-only turf, Nvidia is planting a flag in the inference market itself. It would rather cannibalize a slice of its own GPU sales than let AMD, Groq, Cerebras, or the hyperscalers' in-house chips take the fastest-growing part of the market.

Where does it fit in the roadmap?

  1. GTC 2026LPU announced. Nvidia reveals a dedicated inference chip and reworks the data-center roadmap around it.
  2. H2 2026Groq 3 LPU ships. Liquid-cooled LPX racks, 256 LPUs, 128GB on-chip SRAM, 640 TBps scale-up.
  3. H2 2026Vera Rubin platform. Vera CPUs plus Rubin GPUs continue the training line.
  4. LaterFeynman GPU. Die stacking, custom HBM, and NVLink switches with co-packaged optics.

What it means for the market

For Nvidia (NVDA), this is a defensive land grab that doubles the addressable market: sell the training GPUs and the inference LPUs. The companies most exposed are the pure-play inference challengers, Groq, Cerebras, SambaNova, whose core pitch was that specialized inference silicon beats GPUs. Nvidia just co-opted that argument with a chip that plugs into its existing CUDA and NVLink ecosystem, which is the real moat. AMD's MI-series inference push and the hyperscalers' custom chips (Google TPU, Amazon Inferentia, Microsoft Maia) now face an Nvidia that competes on their turf while owning the software layer. The number to watch is tokens-per-dollar on the LPX rack versus a comparable GPU rack; if it is meaningfully better, cloud providers will buy it, and Nvidia keeps the customer either way.

Our take

This is the most telling hardware move of the year precisely because it is an admission. Nvidia does not build a non-GPU accelerator unless it sees the inference market becoming too big and too specialized to defend with general-purpose parts alone. Bundling it into racks with CUDA and NVLink is the smart part: the challengers had better chips on paper but no ecosystem, and Nvidia just neutralized their pitch without giving up its lock-in. The risk is complexity, two product lines, two optimization targets, and customers who now have to decide which silicon runs which workload. But strategically it is the right call. The company that wins AI hardware in 2027 is the one that owns inference, and Nvidia just refused to let anyone else have it.

Primary sources

Original analysis by GenZTech. Figures current as of July 2026.