I want to tell you about a mistake I made two days ago, because it explains everything that follows.
I found a research paper. Good one. Serious people. I read the date off the page I found it on and started building around it.
The date was wrong.
It wasn’t the paper’s date. It was the date some website added the paper to its list. The real paper came out in January. I was about to tell you it came out last week.
Nobody lied to me. Nothing was hidden. I just read the thing in front of me and took it for what it looked like.
That is the whole subject of this post. So hold onto it.
The paper
In January, a group of researchers published a paper about auditing the biggest AI companies.
Not fifty people who all think alike, either. The list runs about fifty names, and they don’t agree with each other about much. Yoshua Bengio is on it. So is Miles Brundage, who used to work inside OpenAI. So are people who argue against heavy regulation. When that crowd signs the same page, the page is worth reading.
Their argument is simple.
An audit only means something if the auditor gets to look at things the public can’t see. Not the sales brochure. Not the safety page on the website. The actual internals. How the company makes its safety decisions. What it runs inside its own walls. Who said what in the meeting.
They wrote that even the best outside reviews happening right now are missing the thing that every other serious industry takes for granted — an independent reviewer digging through non-public material.
Then they used a phrase I keep chewing on.
They said the alternative is letting companies grade their own homework. And they noted that has a checkered track record in a lot of industries.
Checkered is a polite word.
Congress picked it up
In July, this stopped being a paper.
Two members of Congress, Jay Obernolte and Lori Trahan, introduced a bill called the FRONTIER Act. Four more signed on. Three Republicans, three Democrats.
The bill would build the first real federal oversight system for the most advanced AI models. Transparency reports. Risk frameworks. Incident reporting. And independent audits, run by licensed outfits the government approves.
That is a big deal. Whatever you think of it.
Here is where it gets thin.
The bill says the auditors get access to whatever is reasonably necessary to do the job.
Reasonably necessary.
Two words.
Nobody has said what they mean. Does the auditor get the model weights? The training data? Or does he get a login and a list of questions the company already prepared answers for?
The bill doesn’t say. And it would override state laws that do say.
So we have a bill about independence, and the definition of independence is a blank line.
Why I care about this at my kitchen table
I don’t audit AI companies. I’m one man with a website.
But I ran into the same wall on a much smaller scale, and I want to show it to you, because it isn’t a theory to me.
I’ve spent eighteen months building a set of rules for how an AI ought to conduct itself. It lives in a file. The file is about the size of a short book.
Every so often I have the AI go through that file and check it for mistakes. Contradictions. Broken references. Claims the file makes about itself that aren’t true anymore.
It comes back clean.
One day in August it came back clean four separate times.
Every one of those was wrong.
There were mistakes in there. Real ones. Sentences claiming a thing had been removed when it was still sitting right there. A repair that created a brand new defect forty minutes after it fixed the old one.
Then I handed the same file to a different AI on a different platform and asked it a plain question. What’s broken in here?
It found two things in about a minute. Two real ones. Neither had ever shown up in any of my own clean checks.
And here is the part that stuck with me. When I ran it the other direction, the first AI had found eleven problems the second one missed.
Two honest reviewers. Same file. Almost nothing in common between what they found.
Neither one was lying. Neither one was lazy. They just could not see past their own hands.
That’s it. That’s the whole thing.
The hand that builds something cannot inspect it.
Not because it’s dishonest. Because it already knows what it meant to do. It reads the page and sees the intention. Somebody else reads the page and sees the words.
That’s not an AI problem. Anybody who has ever proofread their own letter knows it. You read what you meant to write.
Scale that up to a company worth a few hundred billion dollars, checking its own safety work, writing its own report about how the check went.
Now you see why fifty researchers who agree on nothing else signed the same paper.
I have a note in my own files. It’s been sitting there for weeks. It says: hand this to somebody who didn’t build it and ask what’s broken.
I haven’t done it enough. I do it sometimes. Not on a schedule. Not every time.
So I’m making the argument for outside eyes while I’m still short of my own standard.
I’d rather tell you that than let you assume otherwise.
And that brings us back to the date I got wrong at the top.
I didn’t check it. It looked right. It sat on a page that looked official, next to other things that were true.
One outside look caught it in under a minute.
That’s all an audit is. Somebody who wasn’t there, looking at what you actually wrote instead of what you meant.
If those two words in that bill come to mean a login and a prepared briefing, we will get very clean reports about very large machines, and not one of them will be worth the paper.
Post Library – Intelligent People Assume Nothing
Contact: micvicfaust@gmail.com
This post was drafted with AI governed assistance and reviewed and directed by Michael S. Faust Sr. before publication.
© 2026 The Faust Baseline LLC | All Rights Reserved
