The Project Is Called The AI Observatory

Everything you have ever read about how people actually use AI came from the companies selling it.

Every headline. Every statistic about what folks do with these machines all day. Every number a lawmaker cites when he stands up to talk about regulation.

All of it came from the people with something to sell.

MIT Technology Review ran a piece yesterday about a group trying to change that, and what they found is worth your time.

The project is called the AI Observatory. It is led by researchers out of Stanford and MIT, along with the Data Provenance Initiative and some others.

What they did was gather up real conversations between people and AI machines. Not made up test cases. Real ones, collected with the users’ permission through seven existing research datasets.

They ended up with 24,521 conversations. That comes to 85,633 back-and-forth exchanges, from about 5,000 people talking to 52 different AI models, spread across 2023 to 2025.

Then they did something clever.

One of the most quoted sources of information about AI use is a report Anthropic publishes called the Economic Index. It looks at how people use Claude for work and productivity.

The Observatory took Anthropic’s own method and ran it across their own pile of conversations, just to see what that method keeps and what it throws away.

It threw away 48 percent.

Nearly half of every real conversation they had did not survive the filter.

Now, that alone is not damning. A work report ought to filter for work. That is what it says on the tin.

The trouble is what was in the half that got swept off the table.

Conversations about health and relationships showed up in 44.2 percent of the discarded material. Anthropic’s published analysis puts that figure at 31.2 percent.

Harassment and hate came in at 27.5 percent of the filtered-out conversations. The published number is 5.66 percent.

Sexual content, 16.7 percent against a published 2.4. Adult or illicit topics, 7.9 percent against 2.1.

So the half that got removed was not just non-work. It was the harder half. The part where people bring their sickness, their marriages, their loneliness, and their worst behavior to a machine.

And I want to be fair here, because fair is the whole point.

Anthropic is not hiding this. The report says right in its name that it is about work. The company has put out separate write-ups on people using Claude for support and companionship. They have even published on people trying to use it to make material involving children.

Nobody buried anything.

But here is what happens anyway. The work report is the one that gets quoted. It is the one that lands in the news story and the policy brief and the boardroom slide. The other reports sit off to the side where almost nobody goes looking.

One of the researchers, an assistant professor at UT-Austin who was not part of the project, put the problem right where it belongs. Ask whether one of these systems is used mostly for good or mostly for ill, and there is no way to answer, because the information belongs to the company.

Not hidden. Just theirs.

And this is not one company. OpenAI’s own 2025 report found that only about 30 percent of consumer use had anything to do with work. Same shape. Different name on the door.

Now the part that ought to bother you most, and it is not about any company at all.

The Observatory has 24,521 conversations.

Anthropic’s index is built on a million. OpenAI’s report ran through a million and a half.

The independent check is somewhere around one fortieth the size of the thing it is checking.

That is not a scandal. That is a structural hole. There is no instrument. Nobody built one. The only gauge that exists belongs to the outfit being measured.

The researchers are honest about their own limits too, which is how you know they are worth reading. Their data came from people who volunteered it. People do not volunteer their ugliest conversations. So their own numbers are probably too low, not too high. They say plainly that their findings do not represent all AI use.

That is what an honest limit looks like. Say it before somebody else has to.

I have written before about a simple rule. Nothing goes out the door until it has been checked, and the person putting it out says how it was checked and what the check could not reach.

That is a rule for one person writing one thing.

Watch what happens when a whole industry never adopts it.

You get real numbers, honestly produced, with a filter disclosed in the fine print, and a world downstream that takes the topline and runs. The picture that reaches the public is not false. It is partial, and nothing in the system is built to tell anybody which part is missing.

Nobody has to lie for that to go wrong. That is the thing to understand. The failure does not need a villain. It only needs an absence.

One more thing I owe you before I close, and it is not comfortable.

I write these with AI help. The particular machine I use was built by Anthropic, which is one of the two companies named in this story. So when I tell you their filter removed half the conversation, understand that the tool helping me say it was made by them.But governed with the Faust Baseline.

I do not know how to fully account for that. I can only tell you it is there and let you weigh it.

The lesson stands either way.

When the only people measuring a thing are the people selling it, you do not have a measurement. You have a brochure with numbers in it.

Somebody finally started building a real gauge. It is small, it is underfunded, and it is one fortieth the size of what it is checking.

That is still one more than we had last week.

The_Ten_Plain_Rules_CardDownload

This post was drafted with AI governed assistance and reviewed and directed by Michael S. Faust Sr. before publication.

Post Library – Intelligent People Assume Nothing

Contact: micvicfaust@gmail.com