by Raywat Deonandan
Feb 4, 2019
A version of this article first appeared as a blog post.
We educators, when feeling bored and troublesome, often pass the time both by complaining about the failures of public education and by making bold and unreasonable suggestions about how best to reform education for all. While I have always erred toward the essential skills of numeracy, literacy and even history, my good friend, statistician Dr Nicholas Barrowman, once offered something more intriguing.
He felt that what everyone really needs is to understand a single deceptively simple concept: that a good scientific investigation needs a control group. If everyone could wrap their brains around why a control group is needed, he argued, then society as a whole would be more rational, less divisive, and better able to navigate life’s boundless waters of chaos and uncertainty. He used fewer words and less poetic allusion, but the wine was flowing as that point.
My first reaction to his position was friendly dismissal. Clearly, Nick was stuck in his own world, his particular professional paradigm, which involves the design and critique of medical studies. But years later, I finally begin to see the true weight and value of his position, especially in the current era of fake news and anti-science. How often are we now subject to tweetstorms and listicles of political or ideological positions buttressed by convenient observations and studies whose lengthy methodologies are too laden with jargon for most people to dissect?
Often, the first step in assessing the quality of such evidence is the ability to identify the control group, and then to assess whether the conclusions made from the presented results are indeed warranted.
The importance of a control group to science is, I hope, logically obvious to everyone reading this article. Without a control group, one has nothing to which to compare empirical observations. Absent such relativity, one can reach many unsupported conclusions, and therein construct any number of personally appealing, though empirically unsupported, realities.
In short: the absence of a control group allows for the ascension of ideology over facts.
Decades ago, I was at a dinner party at which a vocal young woman, passionately opposed to the proliferation of pornography in society, declared to the group: “Do you know that 100% of convicted sex offenders have looked at porn?” She clearly hoped to demonstrate an inescapable causal link between pornography and sex crime.
As I had not yet learned the virtues of silence, I foolishly responded: “100% of convicted sex offenders have presumably also eaten bread. What’s your point?” This, of course, did not result in the friendly dinnertime chit-chat for which I’d been hoping.
The rare gluten-intolerant sex criminal aside, I could have also offered that 100% of men who are not sex offenders also watch porn. But my point was made, however clumsily. Absent the control group, this young woman’s personal political values had driven her toward one –and only one– interpretation of a given observation, even though her conclusion was not necessarily supported by that observation.
Recently, Amnesty International published its report on women’s experiences of abuse on Twitter, titled, Toxic Twitter: A Toxic Place for Women. The report concluded that “the violence and abuse many women experience on Twitter has a detrimental effect on their right to express themselves equally, freely, and without fear.”
No rational person in 2019 can doubt that the Twitterverse can be a horrible place to live. But it’s the phrasing of the report’s conclusion that raises eyebrows. The word “equally” suggests that the study was done in comparison to the experiences of men. But, as Suzie Mulesky pointed out, Amnesty did not assess the Twitter experiences of men, the natural control group. This omission calls into question whether the phrasing of their conclusion is supported by the very data that they presented.
In many sciences, a control group is not always present. But in such cases, care is taken to not overstate the implications of whatever data result.
In 2014, a University of Toronto study showed an improvement in ADHD symptoms among subjects receiving homeopathic treatments. Homeopathy is largely considered to be quackery for a host of very good reasons. So for a respected institution to put its name behind a clinical trial showing some benefits to such a controversial approach is, shall we say, newsworthy.
Physician-scientists joyfully criticized the study, and an open letter signed by leading scholars, including two Nobel laureates, chastised the university for having backed it. And, unsurprisingly, the Faculty of Pharmacy’s link to the study is now dead. But one of the methodological factors that should have immediately cast some doubt onto its merits was –you guessed it– its lack of a control group.
The study is what we call an “uncontrolled trial”, in which a series of subjects are observed over time to see if an intervention (in this case an homeopathic treatment) caused any change. Uncontrolled trials have a purpose in science, including, for example, the measurement of survival rates after a surgical intervention. But it’s very dangerous to make strong conclusions about therapeutic effectiveness from such trials.
A control group is not always necessary in an experiment, if proper care is taken to limit one’s conclusions. But it behooves us all to understand why a control group is indeed important, and how interpretations driven by ideology can flourish in its absence. I still don’t think this understanding is ultimately more crucial than numeracy and literacy. But Dr Barrowman’s position is well taken: in an increasingly technocratic world, the ability to understand the basic philosophies of science becomes evermore important.