Unsupervised Reliance and The Damage Done.

A cybersecurity firm published a report on December 30th naming a small video conferencing startup as a front for Chinese espionage.

The company was called MeetingTV. The report said its product, a meeting recorder called Zoomcorder, was part of the infrastructure behind a hacking operation that had stolen data from more than two million users.

Within days, the damage was done. Security vendors around the world read the report and did exactly what threat intelligence is supposed to make them do. They blocked MeetingTV’s domains. They flagged its services as malware. A small company’s connection to its own customers went dark, almost overnight, because another company said it was dangerous.

Here’s what actually happened, according to the lawsuit MeetingTV later filed.

The threat report wasn’t built by a team of analysts checking sources by hand. It came out of a proprietary AI system the security firm used to find patterns across huge amounts of data. That system found a correlation. It linked MeetingTV’s domain to the hacking campaign. Nobody caught that the correlation was wrong before it went out the door.

The company that published it, Koi Security, later admitted as much. In a court filing, they acknowledged the false claims were the direct product of what they called unsupervised reliance on their own analytical platform. Their own words. Unsupervised reliance.

Six weeks later, on February 12th, Koi added an update. Buried near the bottom of the same report, it said they’d found no evidence connecting MeetingTV to anything malicious.

The headline never changed. The correction sat at the bottom of a page most readers would never scroll to. MeetingTV says the damage done in those six weeks never came back.

Nobody hacked anything. No AI model went looking for a way out of a sandbox. No agent tried to deceive a human on purpose. This was quieter than any of that, and in some ways it’s a harder problem to catch. A tool generated a plausible-sounding conclusion. A team of professionals, people whose entire job is catching threats, trusted the output enough to publish it as fact, aimed at a real company, with real consequences, without checking it against the actual sources first.

That’s not a rogue AI story. That’s a story about what happens when a human being lets a tool’s confidence stand in for verification.

There’s a working practice built into this site’s own operation that exists because of a mistake very much like this one. It happened here first, in miniature. A post went out with translated protocol text presented as the ratified original. It wasn’t caught before publication. It was caught after, and corrected in the open, because the standing rule here is that mistakes stay visible and corrections happen in daylight, not quietly at the bottom of a page six weeks later.

Out of that mistake came a rule. Nothing goes live without being checked against the source file itself, not against memory, not against what a system confidently produced, not against what felt right in the moment. Check the actual thing, every time, before it reaches anyone who might act on it.

That rule has a name here. PPVP-1. Pre-Publication Verification Protocol. It exists because trusting an AI system’s output without checking it against ground truth is exactly how something false turns into something published, and something published turns into something acted on, and something acted on turns into a company’s business getting shut off at the domain level before anyone stops to ask if the claim was ever true.

MeetingTV’s lawsuit puts the industry-wide version of that same lesson in stark terms. Security companies are widely aware that AI systems hallucinate. Everyone in that industry knows it. The complaint argues that knowing it and still skipping the manual verification step, the forensic review, the independent corroboration, wasn’t a small oversight. It was reckless, precisely because everyone involved already understood the risk going in.

That’s the difference between an honest mistake and a foreseeable one. A tool being wrong sometimes is not surprising. A team of professionals publishing that wrongness as verified fact, about a real company, without doing the one check that would have caught it, is a choice about how much verification the moment deserved. In this case, the moment deserved more than it got.

The correction, when it came, tells its own story too. Six weeks is a long time for a small company’s traffic to sit blocked. And placing the correction at the bottom of the same page, without touching the headline that caused the harm, isn’t really a correction in the way that matters. It’s a footnote added to protect the publisher, not a fix aimed at protecting the company that was wrongly named.

An open-page standard means something different than that. It means the correction goes where the damage went. If the headline did the harm, the headline gets fixed, not buried under six paragraphs someone would have to already suspect something was wrong to go looking for.

This is the plain lesson sitting underneath all of it. AI tools are going to keep producing confident, well-structured, entirely wrong conclusions. That’s not a defect that goes away with the next model version. It’s a permanent feature of what these systems are. The only thing that stands between a wrong conclusion and a real company losing its business over it is whether a human being was willing to check the claim against the source before hitting publish.

That check is not optional overhead. It is the whole job.

This post was drafted with AI assistance and reviewed and directed by Michael S. Faust Sr. before publication.

Contact: micvicfaust@gmail.com