Random Acts of Medicine: The Hidden Forces That Sway Doctors, Impact Patients, and Shape Our Health, by Anupam B. Jena and Christopher Worsham (Doubleday, 320 pp., $30)
Random Acts of Medicine, by Anupam Jena and Christopher Worsham, is a breezy tour through the use of natural experiments in medicine. As doctors, the authors occasionally confront questions that are nearly impossible to answer with randomized controlled trials, so they try to find other ways to do so. Natural experiments have a long history in medicine, going back to the famous contaminated water pump that John Snow removed to show how cholera was transmitted.
The combination of sophisticated analytical methods borrowed from econometrics, as well as large datasets, have allowed this type of work to flourish in recent decades, addressing all sorts of questions. One striking study: in school districts with an enrollment cutoff on September 1, children born in August are much more likely than those born in September to be diagnosed with ADHD and put on medication. To ensure that there was nothing inherent in birth seasonality that caused this difference, the authors tested for differences in rates of asthma and diabetes, which are diseases with objective diagnostic criteria (and found none). How does this happen? Children born in August, just before the cutoff, start kindergarten younger than their classmates, while their September-born classmates are the oldest in their grade level. Because executive function improves with age, August-born kids have worse self-control than their September-born classmates. This relationship disappeared in states that have different age cutoffs, such that August and September-born classmates are merely one month apart in age, instead of 11 months apart in age. The upshot is that some portion of ADHD diagnoses are effectively the medicalization of normal childhood behavior.
Another study found that patients of older hospitalists (doctors who specialize in hospital medicine and are assigned almost at random to patients, based on their shifts) had worse outcomes than patients seen by younger hospitalists. This relationship disappeared, however, when the older hospitalists had high patient volumes, meaning they saw many patients. The authors speculate that this may be the result of younger doctors being more up to date with changing medical guidelines. By contrast, older surgeons appeared to have better outcomes than younger surgeons—hinting that, for surgeons, experience and technical mastery trump knowledge of the latest guidelines.
Jena and Worsham are to be commended for their lucid explanations of study design and results, as well as for keeping things interesting by weaving in the personal experiences that motivated their investigations. However, much like the behavioral economics research that the authors uncritically cite, they fail to explain the subtle ways that their research paradigm can go wrong.
One problem with natural experiments is what is known as “researcher degrees of freedom.” This refers to the choices a research team makes when it decides to study a question. In a study on the effect of nearby marathons on delays in emergency medical care, for example, the researchers had to decide which marathons to include, which days would be the comparison days, which hospitals would get included, and which diseases to study. All these choices could affect the size and significance of a given effect. A researcher fishing for an effect could find the perfect combination of analytical choices to demonstrate a result, without readers being any wiser. To their credit, the authors report a number of null results, such as the finding that political affiliation did not predict differing rates of birthday celebrations during the Covid-19 pandemic. But they miss an opportunity to educate readers on this broader phenomenon.
Sometimes, a seemingly perfect natural experiment turns out to be deeply flawed. For example, the highly publicized finding that hungry judges give much harsher judgements on parole boards has been subject to significant criticism.
Another problem with natural experiments involves choosing what types of questions to investigate. In a revealing section, the authors decry the lack of gender balance in several medical specialties, while highlighting how female physicians, across various studies, tend to have better patient outcomes. But does anyone really believe that a study reporting better patient outcomes for male physicians these days would get published as easily? Bias in the types of questions studied and published (“publication bias”) is a real possibility. Consider “stereotype threat”—the idea that an individual’s performance metrics can be harmed by stereotypes about the group to which they belong. A comprehensive meta-analysis on stereotype threat and women’s performance on math and science tests found a much smaller effect than originally claimed, as well as evidence of publication bias.
Analytical tools exist to combat some of these biases. In preregistered studies, the methodology and research questions are fixed ahead of time. Funnel plots and p-curves are tools for checking publication bias. Some institutional changes may help, too: more ideologically diverse academic fields may serve to check politically biased research; publications like the Journal of Controversial Ideas can serve as a backstop for politically disfavored topics and conclusions. Perhaps prediction markets could serve as another way to aggregate expert opinion on a topic, as has been attempted in psychology.
As Jena and Worsham venture closer to more partisan topics, they lose some of the thoughtfulness that makes the first three-quarters of their book so educational on study design. They write: “Women physicians are also impacted by the inequities that affect professional women across many industries. They are paid less than men, such that over the course of their entire careers, women have been estimated to make an average of $2 million less than their male counterparts.” After so many pages of careful caveating of studies and thoughtful analysis of results, this simplification is surprising. Instead of describing how this pay gap might be mediated by, say, differences in specialty choice, work hours, work environment (academic center versus private practice), or discrimination, the authors blithely present these assertions as simple facts.
Their section on the effects of a physician’s race reads similarly: “The glaring inequities in health and health care that have resulted from this mistrust may be best summed up with a single statistic: in 2020, the life expectancy at birth of a Black American was estimated to be about six years less than that of a white American.” The authors appear to attribute the difference in black-white life expectancy at birth to mistrust of the health-care system. This is a startling claim—and almost certainly false, in light of the more powerful impact of non-medical choices and environment (often referred to as “social determinants of health”).
Overall, though, Random Acts of Medicine is an excellent introduction to the creative ways researchers are answering especially tricky medical and health-system questions and chock-full of facts ripe for cocktail parties. Just be sure to take some of it with a pinch of (substitute) salt.
Photo: Totojang/iStock