AI Distillation: Theft or Competition? Experts Say Prove It

Anthropic says accounts tied to Alibaba ran 28.8 million queries against Claude through roughly 25,000 fake accounts between April and June. Treasury Secretary Scott Bessent says the U.S. is finding "watermarks" of American language models inside Chinese ones, and has put sanctions and Entity List designations on the table. And on Monday, Nvidia CEO Jensen Huang told CNBC that none of this is theft. "You're allowed to test somebody else's products all you want," he said. His advice to worried labs: "know your customers, and disable the service."

We covered the Huang and Bessent split when it broke. This time we asked a patent attorney, an attestation engineer and a founder who builds on the big labs' APIs where the line sits. They started in different places and landed on the same question. Not "theft or competition," but "can you prove it?"

RelatedNvidia's Huang Rejects Bessent's AI Distillation 'Theft' Claim

Is learning from a rival's model theft at all?

Michael Dilworth is a patent attorney and founder of Dilworth IP, formerly chief IP counsel at a publicly traded industrial company. He starts with a lawyer's checklist. "What were they taking? How were they taking it? Did they violate any contract? These are all valid questions to ask," he said. "What we call IP theft is typically stealing something someone else owns and has the right to protect."

It also doesn't require walking off with source code. "They can learn off of your output or even try to benchmark off of your models. It's not the same as them getting your code though."

So where does that leave a distillation campaign? Mostly in contract law. "If you signed a contract with them that says they can't then they have violated that as well. Whether they are using fake accounts or not. Illicitly gaining access can play into this too."

The part that should worry anyone hoping for a clean legal win is his next point. "You could probably determine if a model learned off another models output. But that would not make it IP theft. They still would have to be taking something that you can prove belongs to you."

On that narrow question, he gives Huang some credit. "It's understandable that Nvidia wants to keep its business in China. However, Huang makes a good point. Just because someone learns from your company doesn't mean they stole your IP. What you would need to show is that they took your IP. Or violated your contract."

Where does testing end and copying begin?

Mykyta Chernenko, founder of the AI book-writing platform AIWriteBook in Oslo, sits on the customer side of these APIs. "We run on several of the big labs' APIs and switch between them, so I spend a lot of time testing other people's models," he said. "That makes me sympathetic to Huang's first point and skeptical of his second."

His line is volume and intent. "Testing a rival's model is normal. We compare outputs across providers every time a new model ships, and nobody thinks that's theft," he said. "The line gets crossed at volume and intent: a few thousand prompts to see how a model behaves is evaluation, tens of millions of prompts designed to harvest its answers as training data is copying the product."

Then the catch: "The outputs look the same either way, which is why it's so hard to prove from the outside."

That's also why he doesn't buy Huang's fix. "'Know your customer, disable the service' sounds simple and isn't. An API that serves millions of accounts can't vet each one, and the abuse he's describing arrives spread across thousands of fake ones. Rate limits and pattern detection catch some of it. None of it catches a patient actor."

Anthropic's own numbers make his point. Spread 28.8 million queries over roughly 25,000 accounts and each one averages about 1,150. That's closer to the "few thousand prompts" of an ordinary evaluation than to anything that trips an alarm on its own.

Can a watermark settle it?

Serhii Nikolaichuk, an attestation engineer and founder of The Capital Index in Austin, thinks both men have a point and neither has evidence. "Huang and Bessent are both right, and both are missing the same thing: proof," he said.

"Huang is right that learning from a rival's product is how technology moves," he said. "Bessent is right that siphoning a model through millions of automated queries, against the terms it was offered under, crosses a line. The problem is that today nobody can prove which side of the line anyone is on. Distillation fights are arguments without evidence."

He thinks that's fixable, starting with the outputs themselves. He pointed to research from Meta's AI lab, and it checks out: the 2024 paper Watermarking Makes Language Models Radioactive found that when as little as 5% of a model's fine-tuning text is watermarked, the trace can be detected with high confidence. There's a condition worth knowing. That result is for suspect models whose weights are open, which happens to describe many of the Chinese models in question, but not every model a lab might want to accuse.

RelatedOpenAI's GeneBench-Pro Exposes AI's Genomics Judgment Gap

"Train a student model on marked answers, and it carries the fingerprints," Nikolaichuk said. For full disclosure, he and his partner Roman Oliinyk have developed a method for proving which parts of a text an AI generated, and he says they have secured priority for it at the U.S. Patent and Trademark Office. He has a stake in this answer.

Dilworth, who has no such stake, accepts the idea with a lawyer's caution. "You could argue that there are watermarks that would allow you to know that the model used your model as input. This could be used as some sort of proof that they are learning off your model," he said. "But you still need to know that this technique will work. No false positives, etc."

That's a direct challenge to Bessent. Finding "watermarks" is only evidence if the test survives someone who wants it to fail.

What would evidence look like in court?

Nikolaichuk's second tool comes from his own field. "Confidential-computing hardware, including Nvidia's own GPUs, can produce signed evidence of what ran and on which machine," he said. "That turns 'we think they copied us' into records a court or a regulator can check."

His third is the least technical and closest to Dilworth's argument: "Write the rules into the terms of use, then enforce them with evidence, not sanctions by suspicion."

Together, the contract defines the violation, the watermark shows a student learned from your outputs, and attestation shows where it happened. Traffic alone, as Chernenko warned, won't show intent.

"The question isn't 'theft or competition,'" Nikolaichuk said. "It's 'can you prove it?' Until we can, every accusation is just an opinion with a lawyer attached."

Our take

Huang is right that evaluation isn't theft, and our sources agree with him more than you'd expect. Nobody we heard from thinks querying a rival's model is wrong in itself. His fix is weaker: knowing your customers doesn't help when a campaign is spread across 25,000 accounts averaging about a thousand queries each. Bessent has the opposite problem. And "we found watermarks" isn't yet evidence anyone outside government can examine. The useful outcome of this fight would be a public standard: terms of use that plainly ban training on outputs, watermark tests with published false-positive rates, and enough logging to connect the two. Slower than a sanctions threat, but it holds up when the accused pushes back. For how the models in this fight actually compare, see our AI coding leaderboard.

Quotes gathered directly by GENZ TECH from sources who volunteered to comment on this story, with full attribution.