Here’s a good rule of thumb on how to treat news articles on psychology and religion: Guilty until proven innocent. There are too many ways to get things wrong, and journalists have an amazing way of hitting them all. Sometimes it’s entirely journalists’ fault for misreading research reports; sometimes researchers share the blame. In my current example, it’s both.

The example comes from PsyPost’s article last month on research that “indicates religious individuals are more likely to cheat,” but that “this tendency can be diminished by prayer.” Non-believers, in contrast, cheat more after they’ve prayed.

Yes, yes, “more research is needed,” it says, and all that business. That line’s in there. Buried. The lede says what it says, though. So does the headline: “Prayer makes non-believers more likely to cheat, study finds.”

So is that so unbelievable? Where does it go wrong? Let me count the ways. Take heart: You don’t need to be a social scientist to spot most of these. You just need to know something about your own religion, and use some common sense.

Misunderstanding Prayer and Religion

First, the researchers assigned half of the subjects to “compose a prayer,” whether they believed in God and prayer or not. The assumption, apparently, is that everyone is doing the same thing when they “compose” their “prayers.” But believers who pray aren’t just “composing.” They’re connecting with God. Unbelievers? Not so much. The act of composing a prayer means something entirely different from one group to another.

Second, the researchers treat “religiosity” as a single up-down scale, “measured as supernatural beliefs.” But which beliefs about which God, god, or gods? Doesn’t it matter who or what a person thinks he or she is praying to? Or do the researchers think religion is all the same? It isn’t. Not even close.

Third, this research assumes that no one prayed unless they were told to pray. How do they know that? What prevents people from praying without being told to? Believers don’t always need psychologists to “prime” them with thoughts about God!

Fourth, the study concludes that non-believers are more likely to cheat if they pray. What about the alternative answer: “Non-believers are more likely to cheat if they’ve already been instructed to act dishonestly?”

I’m making no slam on non-believers here. I’m saying the researchers told them to do something — praying — they don’t believe in. So if they go on to do something else they consider wrong, why conclude it was because they had God on their minds? Why not think it was because they were sitting in an environment that encouraged dishonesty?

Effect Size and Sample Size

Fifth, this report tells us nothing about effect size. It’s entirely possible for a study to show statistical significance while meaning absolutely nothing in actual practice. Suppose then, or rather imagine, these other problems didn’t exist. We still wouldn’t know how much prayer affects cheating. A little? A lot? It doesn’t say. Not even a hint. (The original research article undoubtedly reports on this, but not this PsyPost version.)

Sixth and last, the sample size (251 total) is too small, too homogeneous (only Americans), and otherwise too undefined to say it “indicates” anything about people in general. I don’t care if these researchers found statistical significance. It’s still not real until it’s been repeated. Often. In many places, with many different groups.

Not that anyone should bother, given everything else that’s wrong with this study. They could confirm it a hundred times, and unless they vastly improved the design, I’d still have the same first five or six issues with it. The study would still be worthless.

These Errors are Typical

But here’s the kicker. I’ve identified six major problems with this study. (I could have included at least two more, but I would have had to subject you to the article’s psych-speak.) The researchers and the writer were undoubtedly aware of the last two issues. The other four? I didn’t read the original research — would you pay $43 for it? Based on this PsyPost article, though, neither the researcher nor the reporter shows the slightest sign of knowing how badly they got it wrong.

And I can tell you from lots of other experiences, this is typical. There’s too little self-policing among social science journalists, ensuring that they only write what they understand. Or helping them see that their track record is weak in matters of science and faith. The researchers themselves tend to make different kinds of mistakes, but as you see here, they still make them.

So when it comes to matters of faith and psychology, the rule isn’t, “Don’t believe everything you read.” It’s, “Don’t believe anything you read” — not until you’ve analyzed its assumptions, as I’ve done here. This is especially crucial when reading research that has to do with sexuality.

Once in a while you’ll run across a good one. There’s decent research (here’s an example) supporting a connection between faith and well-being, for example. So I’m not saying you should be unwilling to find an article to be honest, well-informed and balanced on matters of psychology and faith or religion. I’m just saying you should prepare to be surprised if you do.

Tom Gilson (@TomGilsonAuthor) is a senior editor with The Stream, and the author of A Christian Mind: Thoughts on Life and Truth in Jesus Christ and Critical Conversations: A Christian Parent’s Guide to Discussing Homosexuality with Teens, and the lead editor of True Reason: Confronting the Irrationality of the New Atheism.