10 Ways “Scientific” Studies Are Used to Deceive

Last week a press release describing the results of a study that questioned the effects of marijuana on certain regions of the brain was published and about, oh, every media outlet ran the story to lets us all know, in no uncertain terms, that pot was going to turn us into zombies. Have a look at some of the headlines from the day this study hit the media:

What should be an embarrassment is proudly trotted out for public display.
What should be an embarrassment is proudly trotted out for public display.

There’s a link to the study itself below.

I discovered this media deluge of irrationality right before hopping a plane to the Vegas ArcView/NCIA event and, what with a four-hour flight and two free drink coupons on my hands, wrote this post. In it, I am venting a little. Also, there were the Bloody Marys…


1. Negative studies make news and are memorable. Positive studies are less likely to make news and are less memorable. Why? Perhaps it’s because negative studies tell us what to avoid. Positive studies tell us that we need to change habits. We like the former, dread the latter. Whatever the reason, it does seem to be a fact of human psychology.

2. The imprimatur of science influences perception. Consider: Pharmaceutical advertisements feature two images. One, regular folk having a blast doing cartwheels on the beach, around a campfire, or doing whatever advertising companies think people do in all their many hours of leisure time with unlimited funds for beach houses, and, two, they feature a group of “scientists” in lab coats hard at work, with looks on their faces that let us know, they are Dedicated. Yeah, that’s what truth looks like. It is pure and white. If a fairly good-looking person with a clean complexion, conservative haircut and white lab coat tells us something, so it shall be.

A scientician delivers us some Truth. By Lauri Kulpsoo (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons
A scientician delivers us some Truth.
By Lauri Kulpsoo (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

3. Media outlets and reporters lack integrity. It does not take a genius to call into question a study that includes a mere 48 subjects. It takes lots of curiosity to read the study and take the time to understand it. Most importantly, it takes zero journalistic integrity to pull a quote out of context or write a sensational headline. Here’s how easy it is to come up with two headlines suggesting exactly opposite messages from the same study. First, one I just pulled out of my ass:

Scientistical People Conclude Absolutely Nothing About MJ’s Effects on Brain

Here, in support, I could quote the study author: “[This study] does not conclusively address whether any or all of the brain changes are a direct consequence of marijuana use.” Overall, pretty accurate reporting from an admitted hack.

Now, one the LA Times pulled out of their collective asses:

Regular Pot Smokers Have Shrunken Brains, Study Says

Holy shit!? I thought only dead Amazonian tribesmen and people in England who ate beef tainted with BSE had shrunken brains! The LA Times story continues:

…the authors of the study acknowledge that they cannot discern whether a pot smoker’s smaller orbital frontal cortex is the cause or the result of chronic marijuana use. A 2012 study found that subjects with a smaller orbital frontal cortex at age 12 were more likely to start using marijuana by age 16, suggesting that deficits in this crucial region may predispose one to substance-abuse behaviors.

Researchers noted that the IQ of the marijuana-using group was significantly lower than that of the non-using group–not a finding of the study, but an incidental factor that might be indirectly linked to marijuana use.

Go ahead and read that again. These scientists do not know which came first, the “shrunken” brain or the pot smoking (see #7, below), nor did they standardize the subjects’ IQs. What is this study worth? Bupkes.

And yet, what does the headline reader take away from this? The negative information. It’s stickier (see #1, above.)

Your brain on pot.
Your brain on pot.

Media outlets stopped caring about journalistic integrity a long, long time ago. I scarcely need to support this assertion with an argument, as it has become axiomatic. We now have sites promoting stories simply for how they make you feel (I am thinking of the Hallmark-card sentimentality promoted by UpWorthy and such sites. And while these sites at least announce at the outset what they are offering, the point is that we now select our stories like we would choose a book at the library: based on entertainment value.) So few outlets retain any semblance of journalistic integrity, that they can be counted on one hand—provided that your hand is closed into a fist. It’s abso-fuckin-lutely pathetic and we all know it.

This poses an epistemological problem: How does one sort the clickbait from the factual? Today’s reader must become savvy about the narrative mind and how susceptible it is to stories that are neatly congruent with one’s own perspective or designed with virality in mind. In other words, don’t believe everything you think. Stephen Colbert was onto this idea when he coined the term “truthiness.” Be on the lookout for truthiness. Our tendency to seek stories that support what we believe is known as confirmation bias and it’s something we are all, to some degree, susceptible. We seek evidence to confirm what we hypothesize to be correct and we ignore evidence to the contrary.

Most media outlets are after readership, not honest discourse. That means clicks. That means anything to get clicks. The best of them will disclose their biases. Most do not.

4. Follow-up studies almost always call into question the lead study’s methodologies and conclusion(s), but go unreported. It won’t be long before another group publishes a criticism of that tiny study, pointing out its flaws. By then the news of the first study will have spread and the message will be cemented into the public psyche and discourse. Reversing a widely-held belief is a much more daunting task than shaping the belief in the first place. It’s literally a race to shape the discourse. Early movers win. That’s sad, but there you have it.

5. Small sample size makes a study statistically worthless. Small studies are designed to help guide further scientific inquiry and that’s about it. There are seven billion people on planet Earth, lots of them smoke pot. What about studying four dozen of them makes anyone think something meaningful has been done? Most people do not have the slightest grasp of statistics or the implications of extrapolating from small populations to large ones. Instead, people rely heavily on anecdotal evidence. It’s another aspect of the narrative-loving mind. We love stories because they are useful. They grip us because they allow us to learn from others’ experience without the exposure to risk such experiences ordinarily entail. Our minds want to believe that our vote matters, that the lottery ticket we bought could be The One, that the marble could land on red this time, your kid could be President, or that 48 similar-looking MRI’s actually means something about all pot smokers.

6. Confusing correlation with causation is a common tactic/mistake. Every pregnant dog has a tail. Every star shines. Myopic people are more intelligent than the general population. All true. All are correlated phenomena. But, while tails usually are attached to them, they don’t cause dogs to get pregnant, light doesn’t create stars and near-sightedness does not make people smarter. That one thing can be correlated with another says little about why such a correlation obtains and it cannot be asserted with any logical certainty that one causes the other. It could, yes, but in most cases an underlying mechanism must be elucidated, experiments devised and hypothesis tested before causality can be said to have been proven.

7. Studies often disclaim their results. This particular study’s author even goes as far as to warn that nothing can be concluded. Only that a phenomenon has been observed. That, plus $3 will get you a cup of coffee at this one café I know.

8. Know when funding is and is not suspicious. NIDA funded this study. Hmmm… Doesn’t NIDA have an anti-drug bent? Why, yes. Yes, it does. In fact NIDA is under congressional mandate to study drug abuse.

Scientific studies are their own industry. But does this mean that all studies are subject to this criticism or that all study authors are in the pockets of their payers? No, not at all. To suggest so would be cynical and unfair to many ethical researchers. But is the funding organization of a study beholden to publish the results of every study it purchases? No it is not. This is why peer-review is so important. Some scientists are for sale, but not all scientists are for sale to the same payer at the same time. Peer review will pick out dubious studies 99 times out of 100. This is why it is so important to not react strongly to any study until it has had time to be subjected to peer review and repetition.

9. Is the study relevant to the discourse or is it a distraction? The marijuana debate in the United States is not a health care debate. It is a political debate. Framing it as a health-care debate is a disingenuous distraction. Even so-called medicinal marijuana is not subject to this debate. Nobody ever went to their polling place to vote on whether marijuana works for glaucoma or epilepsy, and that will never happen. We already accept that marijuana provides relief for some sick people for whom conventional medicines are not as effective or are altogether ineffective. The debate happening across America is whether or not to permit these people to have access to marijuana. That is a political decision, not a medical one.

Allow me a quick word on the “marijuana is bad for kids” ploy: it is a cynical, bullshit argument that amounts to nothing more than socio-moral blackmail because everyone already agrees that kids should not have marijuana except in certain, exceptional cases (e.g. Dravet Syndrome.) People who make this argument are in a corner and merely wagering that their opponent will not want risk being seen as anti-childhood-is-sacred (which is a kind of religion in America).

10. When does the opposition deserve a fair hearing? People making arguments that stray from the political debate are either confused or they are not fighting fairly. The pubic health debate is irrelevant. The sanctity of childhood is not threatened by what adults are legally permitted to do. People such as Kevin Sabet, Patrick Kennedy and Bill O’Reilly should not be invited to the table to speak unless they are willing to adhere to the topic.

Often a panel will include speakers who bring up such topics because the host/presenter wishes to have pro and con arguments in the interest of “fairness” or “balance.” But the very notion that every topic has equally valid arguments on either side is itself intellectually suspect. Some ideas do not deserve a hearing. I am not saying that prohibition is one of these ideas, but it is awfully close. Any “con” argument would have to proceed along the lines of a “compelling pubic interest” or similar argument that remains politically relevant. I don’t think such an argument can be made to be convincing, obviously, or I wouldn’t be so adamantly pro decriminalization. There is nothing about prohibition that has worked, so why efforts for its intensification are so routinely submitted as functional ideas simply escapes me. But—and this is the key point—that is the argument anti-pot people will have to make. All the other arguments are beside the point.

Add a Comment

Your email address will not be published. Required fields are marked *