A new study has hit the newsstands that purports to explain observed differences between genders. If you’re interested in seeing a thorough explanation of that article, I highly recommend looking at Social Psychologist Sharon Presley’s amazing analysis of the piece and its many faults. However, the widespread proliferation of this study and the articles surrounding it present an important topic for discussion. Namely, that there is a lot of bad science reporting out there, and people need to be aware of how to spot it.
1. It doesn’t cite its sources
I understand that much science is published in journals that aren’t widely available to the public, but almost every journal makes abstracts available for people to look at. However, many publications do not bother to to use the great technology of hyperlinking to share it with us. For example, Salon recently published an article, citing Mother Jones, that reported that a French manufacturer recently warned women that the emergency contraceptive pill would not work for women over 176 pounds. Neither Salon, nor the MJ article it cites, even mention who ran the study (studies?), whether it was repeated, or the size of it. There is no way for a concerned citizen to analyze the results or know more. We’re just expected to take Salon’s word for it.
2. It assumes scientists are experts on everything
This phenomenon holds true for anyone with a PhD. The idea is that if they have a doctorate, they must be a worthy source for talking about anything. This is an important fallacy to correct. People with PhDs have an expertise in not just one field, but one subset of that field. They hold a specific subset of knowledge with very narrow parameters. While they may have greater knowledge than the layman in related fields, to assume they have expertise is false. This leads to articles citing radiologists on matters of sex psychology.*
3. It cites only one study to make trend arguments
I have a mantra whenever I read research that’s exciting or supports a trend that I agree with. “One study a trend does not make.” Any study that’s worth its salt is going to focus on one variable for a particular set of circumstances. The claims of that paper only apply to what the data says based on those variables. Scientists can and often do make conjectures about the implications of their research, but those conjectures are not a part of the science they’ve done. Only their data are. More importantly, critical review of research is essential to the scientific process (and indeed any academic process). When scientists submit their papers to a journal, others almost immediately critique the research. Others attempt to repeat the study. This process provides an insight that articles miss by only including one experiment and citing it as fact. Peer review could very well prove the experiment deeply flawed and inconclusive.
4. It involves gender differences
I’ll be frank. If you’re reading an article about a study and the article is claiming that it confirms or debunks differences between the sexes, that article is probably going to commit some of the problems above. The Internet positively eats up the so-called gender studies articles, because, due to their controversial nature, they get a lot of hits. The frenzy keeps the bad science coming; it’s just where the incentives line up. To be fair, though, it’s not just sex/gender differences that fall into this trap. Some other topics to be immediately skeptical of are: diet advice, violence in video games, a lot of evolutionary psychology’s claims, and pretty much anything about cancer.
Strive to be a Scientist
For folks who traverse the Internet looking for knowledge through science, I would impart the following important factoids on you: Science is slow. Science is meticulous. Science is cautious. Science necessarily requires repetition, reproducibility, and exactness. You will not find all the answers to the world’s problems by reading one study. That’s not how science works. What conclusions we can definitively draw about our world, if we get any from science, are based on an amalgamation of endless multitudes of data across years and hundreds of thousands if not millions of experiments and research studies.
If you want your life to be guided or informed by science, then make sure to take that inspiration to your own methods of inquiry. Question everything based on a structured method. Do your research. Be cautious, careful, and meticulous—just like science.
* In fairness, Dr. Verma, who was cited in the article linked, is a radiologist and is the last author on the paper the article is discussing. However, Verma’s claims and conjectures fall outside of the bounds of the research she participated in and her fields of expertise, from what I can tell from the abstract and her comments.