Google's HEIR Compiler Runs AI On Encrypted Data

Google has open-sourced a compiler that lets a server run an AI model on data it cannot read. HEIR, short for Homomorphic Encryption Intermediate Representation, went public on August 14 as an MLIR-based toolchain that takes a pretrained model and rewrites it to operate on ciphertext. The input goes in encrypted, the computation happens without decryption, and the result comes back encrypted for the client to open.

Fully homomorphic encryption has been mathematically possible since 2009 and practically miserable ever since. The math works; the engineering has always been the wall. Getting a real model running under FHE has meant hand-tuning ciphertext layouts, picking encryption parameters that trade noise budget against speed, and rewriting your computation into a form the scheme can actually evaluate. That work took cryptography specialists months. HEIR's pitch is that a compiler should do it instead.

RelatedCohere Ships North Mini Code, a 30B Coder for One H100

How does a developer actually use it?

The intended workflow is deliberately unglamorous. You write a program in Python, annotate the types to mark which values are secret, and HEIR compiles the rest. Under the hood it defines layers of abstraction for homomorphic computation and a series of passes that lower from the higher layers to the lower ones, with each layer chosen to make one class of optimization tractable.

Three of those optimizations carry most of the weight. Ciphertext data layout, usually called packing, decides how your values get arranged into the slots of an encrypted vector, and a bad layout can cost you orders of magnitude. Parameter selection picks encryption parameters large enough to survive the noise your computation accumulates but no larger, since oversized parameters are pure overhead. Arithmetization rewrites operations the scheme cannot evaluate directly into ones it can. Each of these was previously a research problem you solved by hand for every new model.

Builds go through bazel with rules_heir, and you can target either OpenFHE or Lattigo as the backend. That backend choice matters more than it looks: it means HEIR is positioning itself as a layer above the existing FHE libraries rather than as a competitor to them, which is the same bet MLIR itself made about compiler infrastructure.

What has actually been demonstrated?

Google shipped four worked applications with partners, and the selection is telling. There is a deep learning recommendation model built with Belfort Labs, LG and NYU. There is credit card fraud detection with Niobium and hardshell.ai. There is network threat intrusion detection using the Kitsune system, also with Niobium. And there is hotword detection for voice assistants, again with Belfort Labs.

Every one of those is a case where the data is sensitive, the model is small enough to be tractable, and the party running the model has a regulatory or commercial reason not to want custody of the input. Fraud detection on card transactions is the clearest example: the bank wants the prediction, and increasingly does not want the liability of holding the raw data to get it. Hotword detection is the consumer-facing version of the same argument, since an always-listening microphone is exactly the workload people distrust most.

Why the missing benchmark numbers matter

Here is the part that deserves scrutiny. The announcement does not publish latency figures. It notes that latency numbers are presented for a single-threaded CPU and points at the source on GitHub, but the actual measurements are not in the post. For a technology whose entire history is a story about overhead, that is a conspicuous omission.

This is not an accusation of anything. Benchmarking FHE honestly is genuinely hard, because the cost depends on the model, the parameters, the packing strategy and the hardware, and a single headline number would be misleading in most directions. But the practical question every engineer will ask is how much slower encrypted inference is than the plaintext version, and right now you have to go measure it yourself. Anyone evaluating HEIR for production should treat that measurement as step one, not step five.

RelatedZhipu's GLM-5.2 Tops the Open-Weight Model Rankings

Who should care right now?

If you run inference on regulated data, this is worth a prototype this quarter. Healthcare, financial services and anything touching biometric input are the obvious candidates, because in those domains the compliance cost of holding plaintext is already a line item you are paying. Encrypted inference does not remove your obligations, but it changes the shape of them: you cannot leak what you never possessed.

If you are running a general-purpose large language model, this is not for you yet, and Google is not claiming otherwise. The demonstrated workloads are small models doing narrow classification. Scaling FHE to transformer inference at conversational latency remains an open research problem, and nothing in this release changes that.

Our take

The interesting thing about HEIR is not the cryptography, which is well understood and mostly not new. It is the decision to treat homomorphic encryption as a compilers problem. That reframing has worked before: GPUs were unusable for general computation until CUDA made them a programming target rather than a graphics device. If FHE ever becomes ordinary infrastructure, it will be because someone did exactly this and hid the parameter selection behind a type annotation.

The honest caveat is that we cannot yet tell you what it costs. A release that demonstrates four real applications and publishes no latency is a release that is confident about feasibility and quiet about performance. That is a reasonable place for a research-grade compiler to be. It is not yet a place from which you should promise your security team that encrypted inference is on the roadmap for next quarter. Go measure it, and let the numbers rather than the announcement decide.

Original analysis by GenZTech. Performance caveats reflect what the release does and does not publish.