A lab In Jerusalem Ran The Asrologey Test

A lab in Jerusalem ran an experiment this month that ought to bother more people than it will.

Three researchers at Hebrew University took two books and fed them to an AI.

The first book was the DSM-5. That is the manual psychiatrists use to diagnose mental illness. Decades of clinical research went into it. It is about as serious as a book gets.

The second book was a popular astrology guide. The kind you find in an airport.

They asked GPT-4 to build a personality questionnaire out of each one.

The AI did it. Both times. Statements you rank from one to five, same as any personality test you have ever taken.

Then they handed both questionnaires to six hundred people, along with the Big Five inventory. The Big Five is the gold standard. It is the one psychologists trust.

Here is what they expected to find.

They expected the astrology questionnaire to fall apart. And in one way, it did.

Inside the astrology test, the answers did not hang together. Traits that were supposed to cluster because they shared a zodiac element did not cluster in real people. The structure was incoherent. The lead researcher said the astrological elements do not reflect real psychological dimensions.

So far, so good. The control did its job. The fake test failed.

Except it did not fail.

Both questionnaires predicted real outcomes in the people who took them. Depression. Anxiety. Well-being. And they did it at levels comparable to the Big Five.

The test built from an airport astrology book predicted people’s mental health about as well as the instrument psychologists spent decades validating.

The frame was false. The readings were fine.

Now sit with what that means.

If you had been handed that astrology questionnaire with the source stripped off, you would have taken it. You would have gotten a score. The score would have told you something true about yourself.

And you would have had no way to know the whole thing was built on zodiac signs.

Not from the questions. They read like real questions.

Not from your results. Your results were useful.

Not from the numbers. The numbers held up.

There was no tell. Anywhere.

The only reason anybody found out is that somebody ran a control.

That is the entire story. Not the AI. Not the astrology. The control.

Three researchers did the boring thing. They built a version they expected to fail, so they would have something to measure the real one against. And the boring thing is the only reason we know any of this.

I have been saying a version of this for a long time now, and I have never had it demonstrated this cleanly.

A thing can work and still be built on nothing.

Output quality is not evidence of a sound foundation. It never was. We just got used to treating it that way because most of the time the two travel together.

They do not have to.

This is why my framework has a rule called CES-1, the evidence floor. It says you check the ground under a claim before you stand on it. Not the claim. The ground. Because a claim that is standing up tells you nothing about what is holding it.

And it is why there is a rule called SVP-1 that says you state a thing at the weight it actually holds. Not heavier. Not lighter.

Which brings me to the second half of this.

Watch what happened to this finding as it traveled.

The paper says GPT-4 estimated how a population would answer, on average, before the surveys went out. That is a real result. It is also a modest one. Population averages. Not you. Not any individual person.

The newspaper I first read it in ran the headline as a question. Is AI clairvoyant.

A tech site picked it up and ran with predicting human thoughts.

Three hops. From aggregate survey averages to mind reading.

Nobody lied.

Every hop was defensible on its own. Somebody wrote a headline with a question mark in it. Somebody else read that headline and wrote a livelier one. No single step was a fabrication.

The distortion did not come from any one person. It came from the distance.

The newspaper version also left out the astrology result entirely. The part where the fake test worked. That is the single most important finding in the study, and it did not survive the trip.

I would not have known that either, except I went and checked the article against the study it was reporting on.

One more piece, and then I will let you go.

The same lab said what they are working on next. They want to read personality directly out of a person’s natural language. No questionnaire at all.

No form. No ten questions you agreed to answer. No moment where somebody hands you something and you decide whether to fill it out.

Just your words.

I am not going to tell you that is sinister. It might be useful. There are places in this world with no psychologists at all, and something is better than nothing.

But I will tell you this.

When the questionnaire goes away, the control goes away with it.

Right now, somebody can hand you a test and you can ask where it came from. You can ask who validated it. You can ask what happens if you answer honestly.

When the assessment is just made from how you talk, there is nothing to hand you. Nothing to ask about. Nothing to refuse.

And based on what these researchers just found, the reading might be pretty accurate.

Built on anything. Or nothing.

You would not be able to tell the difference.

Neither would they.

This post was drafted with AI governed assistance and reviewed and directed by Michael S. Faust Sr. before publication.

Post Library – Intelligent People Assume Nothing

I post four a day. Leave your email and it comes to you.

Contact: micvicfaust@gmail.com