There were only so many moments of absolute confusion—punctured by the occasional dominatrix, feces gobbling scientist, and rocket-predicting erections—I could take before I was compelled to admit that I simply don’t have what it takes to enjoy this book.

Who does have what it takes? I don’t know, and I’m not sure I want to meet them. Pointsman’s conditioning of Brigadier General Ernest Pudding started to feel an awful lot like what Pynchon was trying to do to me.

But Gravity’s Rainbow is considered one of the great works of modern literature. It’s a national book award winner, sits on Time’s list of All-Time 100 Greatest Novels, and to quote wikipedia, “is considered by many critics to be one of the greatest American novels ever written.” There are, apparently, a great many people who do actually enjoy reading this thing.

So why didn’t I like it? One possibility is that I’m simply not a good enough reader. Reading is a skill, after all. Maybe I entered the big leagues too soon. I thought I was being responsible—I prepared myself with Wallace and Woolf, Joyce and Nabokov. But even Wallace’s Infinite Jest comes nowhere close to the complexity of GR. Each sentence is a misadventure with eight words you’ve never heard of before, and when you get to the end of a page you realize you’re not sure whose perspective you’re reading from anymore, let alone what’s going on in the scene.

Again, maybe this is a skill issue. Or maybe this book is simply meant for people with a much higher IQ than mine. But I confess that I have a hard time believing that anyone is having a good time during their first read through of GR. No matter how good a reader you are, I’m skeptical that you find this enjoyable.

But maybe this doesn’t matter. Maybe a great book can be—or even should be—confusing the first time through? Maybe it should make you wrestle with it, gaining more and more insight and clarity with each read, until finally it clicks into place. Maybe the fact that GR is nearly unreadable at first pass is a feature not a bug.

There’s surely something to this. Crime and Punishment isn’t considered great because it’s easy. It’s considered great because each time you revisit it you can learn something new about the psychology of guilt.

But how far can we push this? Can a book be radically inscrutable at first read and still be considered a masterpiece?

Reading GR (trying to, rather) has made me realize that I don’t have a well-developed theory of literature. What makes a book great? If I had to give you my best theory right now, it would be something like the following: The best book should be stimulating at every level of analysis, giving you endless opportunities for further exploration.

That is, a great book should be interesting on the first pass as just a story. But it should also leave you with near endless opportunities to investigate further, raising questions about the plot or character motivation or ethics or psychology and so on. And you should be able to follow these threads, which in turn lead to even more questions and puzzles. The more you read, the more there is to notice.

Pale Fire is a good example. Nabokov is a beautiful writer, and regardless of whether you’re following the intricacies of the plot the first time you pick it up, it’s fun. The main poem is magnificent and it’s fun to realize that the narrator is a maniac. But once you’ve finished it, you realize you’ve stepped into a game of 14 dimensional chess. What actually happened? Who is real? Who is lying? One layer of mystery gives way to the next, and there are always hints waiting for you in the writing, leading you onwards.

Infinite Jest by David Foster Wallace is, in my view, an even better example. Modulo some difficult writing, the story is both hilarious and moving on the first pass. Coming to the end of the book you feel like you need to go right back to the beginning to start working out what actually happened. There are plots and subplots within plots and subplots and entire storylines buried in the footnotes, with just the right amount of detail at each level to help you along the way.

Stoner, The Brothers Karamazov, Crime and Punishment, Lolita, and Anna Karenina are more examples that score well by this theory.

I’m undecided about whether it’s possible for fiction to be truly open-ended. Can there actually be an infinite amount to investigate in a novel? Maybe not. But even if not, how much exploration it generates, coupled with how fun and novel and insightful that exploration is, is a useful metric by which to judge a book.

The problem is that this metric can be gamed. What if you just write a really complicated book full of gestures towards intriguing questions replete with hundreds or thousands of arcane references? This can seem like it’s full of insight and rabbit holes, but really it’s just throwing shit at the wall and hoping the audience does enough work on their own to make something out of it.

It’s a fine line. A book needs to offer enough structure to lead you to genuinely new territory, but not so much as to lead you by the hand. This means a bad book can cosplay as a good one by just being sufficiently impenetrable, pretending that a confused reader simply isn’t reading closely enough.

And yeah, sorry, but this is sort of what Gravity’s Rainbow feels like to me. And the conversations I’ve listened to about it since giving up don’t do anything to alleviate my suspicions. They’ll touch on big themes—world wars, paranoia, operant conditioning, fear of death, sadomasochism. Fine, but a great work can’t just list a bunch of interesting topics. What new thing does it say about them? What sort of insight is it offering? Where is the book actually taking you?

When it comes to GR, I have no idea, and quite frankly I’m not sure anyone else does either.

This was another book club pick. You can listen to us optimistically (?) tackle part I here, and then give up here.

Subscribe now

For those of you who don’t consume Andy Matuschak‘s content like he’s some sort of productivity demigod, anki cards look like this:

They are digital flashcards that are shown to you at a cadence defined by spaced repetition, which is simply the idea that you should review information at increasing (geometrically spaced) intervals to best remember it. To fight the “forgetting curve,” you need to review new information more frequently after just learning it, and less frequently as time goes on.

One unanticipated consequence of using anki is that I’ve become more interested in more things. While anki is typically recommended as a technology that lets you memorize what you’re already interested in, I’ve found that memorizing brute facts has actually increased the surface area of my interests.

I was, sadly, never good at history. I had a second-order desire to be the kind of person who knew a lot of history, but my first-order preferences would always betray me.

But I bootstrapped an interest in history by learning some basic facts. Just knowing the dates and dynasties of the Delhi Sultanate made me more interested in early India. And then I got interested in how the last dynasty gave way to the Mughal empire, which in turn led to an interest in the East India Company. And now, all of a sudden, I can read William Dalrymple’s The Anarchy at an almost-not-embarrassing pace.

We have theories of learning that partially explain what’s going on here, all of which are along the lines of “knowledge begets knowledge.” It’s easier, obviously, to learn new things when you have the relevant background. You need something to be able to hang your hat on, so-to-speak. You need to be able to relate new things to old things.

But while this explains why it’s easier to learn about the Mughal Empire after I know about what led to its founding, it doesn’t explain why it became more interesting. Why was it fun all of a sudden?

Because we enjoy things that are challenging, but not too challenging. If I’m presented with information and have no context then I’ll fall behind and lose all interest. This is why so many courses feel miserable. If you’re not keeping up with the readings, then you have no idea what’s going on and being in class is a waste of time. Likewise if things are too easy. Nobody does the same crossword over and over again.

But when things are challenging yet achievable, we love it. In fact, these are the kinds of tasks that get us into flow state. And learning, even just learning “basic facts,” involves puzzle solving. You’re constantly asking, “does this make sense?”, “does this accord with everything else I know?”, “if this is true, what else has to be true?”.

Once you lay a foundation, filling in the gaps is fun. And then everything on the periphery of what you’re learning becomes that much more interesting. And before you know it, your anki deck contains a disturbing amount of information on the history of the Indian subcontinent.

Subscribe now

I’m increasingly puzzled by analogies between evolution by natural selection (a biological phenomenon) and optimization in machine learning (a branch of mathematics). This is a particularly common idea among the small but growing contingent of people who are concerned that AI will kill us all (when, how, and why it will kill us all is unknown—but by Jove are they ever sure it will.) Eliezer Yudkowsky repeatedly drew such analogies on both The Lunar Society and the Lex Fridman Podcast. Robert Wiblin made the analogy on twitter. Roko Mijic made similar noises on Robert Wright’s podcast.

I think the analogy between evolution and optimization obfuscates much more than it illuminates. It is occasionally helpful as an intuition pump, but should not be relied upon as the premise of detailed arguments about AI. At the very least, any such analogy needs to be argued for, not merely asserted. Something-something Hitchens and his shaving routine something-something.

Let’s take a minute to lay the groundwork for those not neck deep in vexing arguments about AI on twitter.

1. Brief recap of optimization and evolution

Evolution by natural selection involves blind variation of genes followed by the selective retention of those changes which were advantageous for the organism. Blind variation refers to the fact that the variation at the gene-level is effectively random (they are due to chance mutations or copying errors during replication). Consequently, the process is undirected: genes have no goals. They are not trying to mutate in any particular direction—they are not trying to do anything. But different genes have different effects, and different effects lead to differential rates of survival among organisms. And bada-boom bada-bing we have evolution: genes whose effects proved useful are more likely to be found in the next generation. Now simply iterate for millions and billions of years.

Mathematical optimization, meanwhile, involves … well, math. At a high level, we have some function (called a loss function) that we’re trying to minimize. If you’ll forgive the temporary intrusion of some algebra, suppose the loss function is f(x) = (x - 2)^2. This function is minimized at x=2 (you can convince yourself by drawing a picture, doing a little calculus, or simply noting that f will be positive everywhere except at x=2 due to the square). The field of optimization focuses on developing efficient computational methods to solve these problems (this only gets interesting when the loss functions are significantly more complicated).

A particularly successful optimization technique is gradient descent, which is a method based on the geometry of the loss function. Imagine the function as a landscape, and consider walking downhill from wherever you begin. If you cannot walk downhill any longer, then stop. The function f above has the shape of a parabola, so it’s easy to see that this strategy will indeed eventually lead you to the minimum regardless of where you started. The same intuition holds for higher dimensional functions, although one has to imagine higher dimensional landscapes. This can be challenging without pharmaceutical intervention.

In machine learning, we choose the loss to measure how well the model is performing on a specific task. It compares the true label to the predicted label and penalizes the prediction according to how wrong it is. For the much discussed large language models (LLMs), the label is (roughly) the next word in the sequence. For instance, given the input “Harry Potter is a,” the predicted word “wizard” will score better than the word “toad.” The model is trained by minimizing the loss over training data (e.g., a bunch of text from the internet), using optimization to efficiently tweak the parameters of the model until the loss is sufficiently small.

(In our example above x is the only parameter, but large neural networks have billions of parameters. Optimization tells us how to tweak each one to minimize the loss.)

2. Evolution as optimization?

Biological evolution can certainly be seen as some kind of optimization process. The accumulation of adaptive mutations over long periods of time create organisms that are especially well-suited to a given niche. Depending on the niche, this may involve gills and eyes. It may involve echolocation or the ability to photosynthesize. Or, as in the case of humans, it may involve the evolution of minds capable of art, language, and reason.

That said, there is some distance between the notions of mathematical optimization and evolution. Yudkowsky, however, recently tweeted out the following, which suggests that he—and probably the many people he’s influenced—are relying on the parallels as arguments about the inevitability and imminent arrival of artificial superintelligence

The claim is that because evolution by natural selection gave rise to human brains, so too can any kind of mathematical optimization provided the problem is sufficiently “open-ended.” Yudkowsky goes on to define an open-ended problem as a “complicated problem where you can go on getting better and better at it.”

Obviously, this all hinges on what exactly he means by a “complicated problem.” Consider calculating the decimal expansion of pi. You can get better and better at this; progress will never saturate because the expansion is infinite but non periodic. But a computer tasked with calculating this expansion for the next twenty billion years will not develop general intelligence.

To be fair, I’m sure Yudkowsky would take issue with this example. Perhaps he would say that this calculation isn’t sufficiently complicated. But this imprecision is all part of the problem. The AI-is-going-to-kill-everyone conversation is dominated by grandiose claims that are hard to prove wrong because they’re so vague. And then when they’re not proved wrong, an army of people on twitter begin yelling that we’re all about to die, journalists start to take them seriously, and the general public develops a severely skewed understanding of this technology.

(And then, to complete the cycle, some people use this public opinion as evidence that they are correct and that they can bow out of the conversation, safe and assured in their victory. Pooh and Piglet have almost caught the woozle.)

3. Real versus artificial worlds

Machine learning takes place in a highly artificial world. We carefully select training data and structure it so that learning is possible. The goal is well-specified (minimize the loss), and so is the procedure for reaching this goal (gradient descent). The space of possible outputs is known beforehand (e.g., the set of words constituting the corpus). Models are being trained in what Jimmy Savage would call “small worlds,” which I’ve described previously as “worlds where all inputs and outputs are known, as well as the rules governing all interactions. There are no unknown unknowns.”

This is why math, inductive reasoning, and probability work so well in these domains. We have crafted the artificial world such that they are useful. We have parceled the world into legible chunks, told the model which chunks are relevant, and given it a mass of examples to learn those chunks.

This artificial world is the opposite of the real world which, of course, is where natural selection occurs. This distinction is responsible for several substantial differences between mathematical optimization and genetic evolution.

3.1. What’s the loss?

If one is to analogize evolution and optimization, the first step is to identify a loss function in the former. What, exactly, is evolution optimizing? One may be tempted to say that evolution has “inclusive fitness“ as a loss function, i.e., the number of descendants and close relatives that an organism shares in the next generation. But this would be misleading in several ways.

First, evolution is not goal-oriented—there is nothing it’s trying to do. It is not proceeding “in the direction of steepest descent” in the same way that gradient descent is (even if that phrase were to mean anything in the context of natural selection).

Second, inclusive fitness is a more abstract notion than a mathematical loss function. It is constantly changing depending on the organism and its niche, whereas a loss function is not. Third, loss functions in machine learning include the desired output. That is, they compare the predicted output with the target output. This target output is selected by humans. Inclusive fitness has no target output; there is no superhuman figure dictating whether a genetic change was good or bad. Therefore, the means by which you’re judging progress is different in the two cases.

Finally, genetic evolution is discrete (the “parameters” of the “model” are the ACGT base pairs). This is distinct from machine learning, where we choose the loss to be continuous and differentiable (mathematical properties which make the optimization process tractable).

3.2. Hardware versus software

As Quintin Pope points out, the “optimization” of natural selection and the optimization in machine learning are working at different levels. Natural selection changes our genome, which gives rise to our brain structure—what connects to what, how many connections there are, the physical shape of our brains, and so on. Optimization in machine learning takes the model architecture (typically a neural network) as given and optimizes over the parameters of the network. The number of parameters in evolution (the size of the genome) is therefore constantly changing, whereas the number of parameters is fixed in machine learning.

More generally, natural selection is an embodied process. It involves directly interacting with the world. The claim that (super)intelligence can be evolved in an artificial, digital world is not at all obvious. In fact, there is a direct conflict between evolution and optimization here: natural selection in humans first produced motor functionality and only then evolved higher cognitive capacities. (Arms and legs predate the human brain. The former evolved in Hominidae.) Machine learning is attempting to go directly for the latter. Even if you believe this is possible (which I do), the argument as to why it’s possible cannot follow evolutionary logic.

3.3. Novelty is punished

Optimization in machine learning works, as we’ve (tediously) discussed, optimizing the parameters of a model by minimizing the prediction error over the training data. This implies that the model will be penalized for new “ideas” (sequences of words) that are not part of the training set, regardless of whether they are true or false. More precisely, it will be penalized for sequences of words whose conditional frequencies do not match those of the training set.

To make this concrete, imagine training a language model on all text prior to Galileo’s famous 1610 pamphlet. The sentence “the sun rotates around the earth” is more consistent with this corpus than the statement “the earth rotates around the sun.” A language model will thus be explicitly discouraged by its training to output the latter. It is for this reason that these models are excellent at summarizing and compressing known ideas (including code), but rubbish at generating truly novel insights which contradict the status quo.

Random genetic mutations, by contrast, do not have to be consistent with any “training data.” New phenotypes are judged only according to whether they prove advantageous at gene replication in the current environment.

4. Having it both ways

The analogy between evolution and optimization tends to be applied when convenient and ignored when not. If evolution optimized humans to be smart, then it also optimized us to be social, curious, sexual, and all the things that fit into the messy bucket labeled “human nature”. Oddly, I never seem to hear about a possible existential risk due to social, self-conscious, and status-seeking automata.

Meanwhile, we often hear the AIs will undergo some sort of recursive self-improvement. They’ll start modifying their own code, proceeding from human-level intelligence to superintelligence within a matter of months, days, or minutes. Humans, however, don’t do this. If we’re going to appeal to the equivalence of gradient descent and evolution as an argument for why deep learning will produce general intelligence, let’s at least be consistent about it. We can’t begin arbitrarily layering on abilities never seen in any organism produced by natural selection.

5. Optimization is not manna

It may seem unfair to pick on this single tweet. (I’d feel worse if its author didn’t think he was the smartest human alive. Perhaps he’ll deign to explain himself next time.) But I think this view is widely held (though perhaps not stated in these terms), and is fomenting huge amounts of confusion. For instance, the philosopher David Chalmers recently gave a talk1 in my department. One of his slides read:

An algorithm that truly minimized text prediction error (subject to constraints) would presumably require deep models of the world and genuine thought and understanding.

If so: sufficiently optimizing text prediction error in a language model should lead to world-models, thought, and understanding.

And here is Ilya Sutskever, chief scientist at OpenAI, on the Lunar Society Podcast discussing the power of next token prediction, and why being able to predict the next token well enough implies all sorts of abilities:

[W]hat does it mean to predict the next token well enough? It’s actually a much deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It’s not statistics. Like it is statistics but what is statistics? In order to understand those statistics to compress them, you need to understand what is it about the world that creates this set of statistics? And so then you say — Well, I have all those people. What is it about people that creates their behaviors? Well they have thoughts and their feelings, and they have ideas, and they do things in certain ways. All of those could be deduced from next-token prediction.

Both Chalmers and Sutskever are making the same mistake. They are treating optimization as an abstract process (truly minimized prediction error, predicting the next token well enough) and arguing about what sorts of abilities it will have. But to connect these arguments to modern machine learning, you have to argue about the actual optimization process in these systems (gradient descent or variants thereof), as well as what they’re optimizing over (structured training data selected by humans).

Alas, “optimization” is not a magical fluid that can be poured onto problems of your choosing to inexorably dissolve them. Arguments which claim insight into machine learning based on evolution are, more often than not, playing a language game. The word optimization means many different things. Simply because you can use the word optimization to describe a process does not mean that it will result in fully formed malicious minds launching themselves into the ether.

Of course, none of this is a knock down argument against AI risk (even of an existential nature). In fact, I agree there are things to be worried about. But I find the connections between optimization and evolution to be tenuous at best, and relying on such connections as arguments as to why deep learning will inevitably create superintelligence is careless.

Thanks to Vaden for comments.

Subscribe now

It was fun! You can’t talk about whether abstractions are real, the correspondence theory of truth, the Duhem-Quine thesis, and the demarcation between science and metaphysics too often in daily life without severely hampering your dating prospects, so it was refreshing to be surrounded by people who couldn’t get enough of these topics. There were roughly 70 attendees—small enough to feel intimate, large enough for there to always be someone new to meet.

But despite the fun, I’m worried. I’m worried because I’m not sure if “critrats” (what proponents of critical rationalism like to call themselves) can avoid the hard problem that most communities face as they grow. A problem that we might call ideological ossification.

You would be hard pressed to find someone at RatFest who doesn’t believe in the many-worlds interpretation of quantum mechanics, or isn’t a libertarian, or thinks school is good, or takes evolutionary psychology seriously, or thinks AI poses a danger. Most people there would agree with the physicist David Deutsch, himself a student of Popper, about practically everything. Dissidents were there—John Horgan was one such voice—but they were few and far between.

This dynamic is to be expected, at least to some extent. Critrats are a group of people inspired by a particular set of ideas, so naturally their worldviews will be similar. And to be at all effective in the world, a community needs to share certain views. The libertarians won’t get very far if they start recruiting communists; unionists are unlikely to be anarchists; factory farming abolitionists probably won’t start questioning whether animals are sentient.

On the other hand, it’s easy for a community to become an ideological monoculture, wherein everyone thinks the same way and there are shared unquestionable, often unmentioned, premises. We’re all aware of this dynamic now, having been yelled at about echo chambers for several years. There are enough examples from across the political spectrum for you to pick your favorite one: BLM, MAGA, environmentalists, AI doomers, AI accelerationists, anti-natalists, anarcho-capitalists, communists, whatever.

Most communities are, I think, aware that its members tend to think the same way. Some praise this dynamic, convinced that they’re correct and that anyone who disagrees is a blasphemous moron. Political communities tend to be like this, as are most groups fighting for (what they’d describe as) “social justice.” And for some communities—those concerned with things besides politics and science and truth-seeking—this dynamic isn’t a problem. The local bowling community can be as much of a monoculture as it wants, and nobody is the worse for it.

Other communities recognize that groupthink is bad, and they do their best to fight it. In theory, a community can get around this problem by promoting particular epistemic values. They can support internal and external criticism. They can value humility and fallibility, discourage hero worship, avoid argument by authority or identity, and invite disagreement.

Of all communities I can think of, effective altruism (EA) comes the closest to promoting and upholding these kinds of norms. They pay people to criticize them. They hold essay contests asking people to change their minds. They litigate niche disagreements publicly on the EA forum. The comment section of Scott Alexander’s blog is like the garden of eden for calm and rational disagreement.

But despite all these healthy epistemic norms, EA has, from my perspective, effectively lost their minds over the past five years. They have gone from promoting distributing bed nets in subsaharan Africa to prioritizing shrimp welfare, death by AI superintelligence, trying to predict and influence the world one billion years from now, and unleashing a modern day Bernie Madoff onto the world.

Some people will disagree that these are bad things, or that EA is at fault for them. Fair enough, maybe I’m just wrong. But insofar as you agree with any of these examples, this should make you worried! To repeat: This community puts substantial effort into fostering healthy epistemic norms, more so than any other community I can think of. Plus they’re nice people! They’re thoughtful and smart. If they are susceptible to weird ideological trends like this, what hope is there for anyone else?

I find this genuinely troubling. Of course, I expect any community of people to get things wrong. But to my eye, EA has gone massively wrong. And more worryingly, they’re quickly becoming incorrigible.

Again—maybe you disagree with me about EA. Maybe you think they are perfectly undogmatic. But even then, you must marvel at how much effort is required to strike this balance between openness to criticism, and having a cohesive enough community to successfully do things in the world. Even if EA has solved this problem, it’s clearly a hard problem to solve.

Building a community requires cohesion among the members, and cohesion most easily comes from having similar ideas. But having similar ideas pushes you towards ideological conformity. That’s a difficult dynamic to navigate, and I hope the critrats are up to it.

Thanks to Cam and Kasra for feedback on earlier drafts.

Subscribe now

Ben Recht, professor at Berkeley and author of the argmin substack, has also been trying to figure out what statistics is up to as a discipline. I've learned a lot from his blog (I particularly recommend his collection on Paul Meehl's course) and I share many of his criticisms of statistics run amok. We're both, I think, wary of the over-mathematization of systems that are difficult to mathematize, and the ensuing misguided enthusiasm to throw mathematical optimization at everything.

Recht recently wrote a paper on a bureaucratic theory of statistics, which advances his view of what statistics is, and should be, all about. It's a refreshing take and adds a new dimension to the (surprisingly lively) debate about the role of statistical inference. But I find that I’m not entirely convinced by his thesis. This is me thinking through it.

Recht's view is that statistics is less in the business of truth-seeking than in the business of providing clear, transparent rules for decision-making in large organizations. That is, statistics helps facilitate decision-making in bureaucracies. He writes:

From its inception, statistics has been the mathematics of bureaucracy. It provides a numerical foundation for governance by clear, transparent, aggregate rules. Statistics helps governments measure what experts on the ground see and create reasonable metrics for consensus to move forward with policy.

Recht calls the specification of these rules ex ante policy. Concrete examples of such policies are:

• The FDA deciding that a drug can enter the market if a clinical trial with 10,000 participants shows a treatment–control difference with estimated Cohen’s d at least 0.4, and the difference is statistically significant at the 0.01 level

• Google committing to changing its logo if an A/B test shows that the click-through rate increases by at least 0.1%

• The IRS using doubly robust confidence sequences on audit outcomes to give estimates on the tax gap each year

• A genomics company searching for associations between genetic variants and a disease commits to applying the Benjamini–Hochberg procedure to control the false discovery rate among the variants it tests

"Ex ante" here means "before we run the experiment." So ex ante policies are statistical commitment devices that organizations adopt to be transparent about their decision-making.

Recht differentiates ex-ante policy from ex post inference, which concerns "drawing conclusions about the verisimilitude of theory or the nature of material reality from empirical evidence." In other words, ex post inference is what everyone thinks statistics is in the business of doing: helping us discover things about the world. (There's a reason we call it statistical inference after all!)

(The vocabulary "ex post" will automatically trigger statisticians. Recht doesn't mean ex post in the sense of illegally choosing your parameters after analyzing the data (also called peeking, p-hacking, harking, etc.) but rather trying to infer parameters from the data, which necessarily occurs after the data are in.)

Examples of ex post inference are:

• Helping determine that exposure to asbestos increases your risk of mesothelioma (example)

• Helping determine that smoking increases cancer risk (example)

• Every discovery of a subatomic particle at CERN or any high energy particle collider (eg bosons, quarks, neutrinos, and so on) (example)

• Helping detect the existence of gravitational waves (example)

• Helping determine that ibuprofen reduces pain (example)

Recht's paper highlights the importance of ex ante policy, and argues both that (i) statistics has, perhaps unbeknownst to some of its more starry-eyed practitioners, always been in the business of ex ante policy making, and (ii) that it should more openly embrace this role.

What's unclear to me about Recht's thesis is the extent to which he thinks ex ante policy is the only role for statistics. Should we dismiss ex post inference entirely? On page two, Recht says seems to say no:

Rulemaking is certainly not the singular valuable application of statistics, but it has been a revolutionary and uncelebrated use. Causal inference researchers, whether potential outcomes advocates for whom all causation stems from hypothetical RCTs, or do-intervention scholars who still argue RCTs should have less relevance, should reckon with this underappreciated role of statistical theory. (emph mine)

But he is also quick to say that ex post inference doesn't work very well. He argues that the so-called scientific "discoveries" made by ex post inference are never clear cut, always messy, and more sociological than logical.

While statistics has admirable aspirations to help answer questions about ex post inference, it’s hard to find grand scientific discoveries solely enabled by RCTs or other causal inference methods. Scientific inference is rarely numerical and always cultural and heuristic.

And he frames the paper as trying to answer the question of what statistical tests actually do, since he's unconvinced that the "lofty goals" aspired to by many statisticians, which include ex post inference, are tenable:

Despite a century of rigorous development of statistical hypothesis testing and causal inference methods, it is not at all clear that Statistics actually does any of these things. We see something different when we look at the practice of statistical testing and statistical inference. Cyberneticist Stafford Beer famously asserted that “the purpose of a system is what it does.” If, after 100 years of fighting and browbeating, we see that statistical testing consistently fails to provide strong epistemological guarantees or determinations of causation, then it’s counterproductive to continue teaching our students that this is the purpose of statistics. But then, what exactly do statistical tests do?

I agree with Recht that ex ante policy is an important part of statistics. I agree that differentiating ex ante policy from ex post inference is a good move, and I agree that statisticians should remain cognizant of this divide.

But I don't think ex ante policy is the only role for statistics. So insofar as Recht is dismissing ex post inference, I disagree with him. Ex post inference will always be an important part of the game for two reasons: (i) it does help us discover things and (ii) you can’t sensibly evaluate a proposed ex ante policy without discussing its properties as a tool for ex post inference.

Regarding (i): dismissing ex post inference too quickly puts you in an awkward position. You have to start denying that statistics played any helpful role in the five examples listed above. Do we actually not know if asbestos or cigarettes are bad for you? Does none of our medicine work? All of these discoveries relied on statistical inference because the effects are difficult to register with your own senses. (Try and feel a gravitational wave.) They require sorting out signal from noise, comparing two datasets and trying to determine if there’s any significant difference between them. This is precisely the domain of statistics.

Elsewhere, Recht has indeed disputed some examples of ostensible ex post inference. He questioned whether we really discovered the Higgs Boson, for instance. His criticism seems to rely on the fact that the process was messy and the statistical model very complicated. Not everyone was immediately convinced, and there are still doubters. As he put it above, the inference wasn't purely numerical but also cultural and heuristic.

But this isn't particularly damning of statistics. Scientific discovery is always messy, even in domains that rely little on formal statistics. Darwin’s theory of natural selection took decades to gain acceptance; evidence slowly dribbled in from fossils, embryology, comparative anatomy, and biogeography. So too with plate tectonics: Wegener’s idea of continental drift was mocked for half a century until mid-20th-century oceanographic data made it undeniable. And again with the germ theory of disease. And again with the double-helix model of DNA. And so on and so forth.

Recht is correct that “it’s hard to find grand scientific discoveries solely enabled by RCTs,” but that’s too high a bar for statistics. Statistics, on its own, will rarely enable discovery. But statistics in conjunction with a detailed theory of the phenomenon of interest does enable discovery.

Statistical testing exists as a mechanism to keep us from fooling ourselves. It’s the first line of defense against naive realism—the tendency to accept uncritically what your eyes or gut tell you. It’s easy to come up with a theory of why something works, or of how to cure a disease. It’s harder to get your theory to pass muster in a clinical trial. And while doing so is not proof of correctness, it’s an important bulwark against nonsense.

If the government tells me that I have to inject myself with a virus every year, then they damn well better be able to show the efficacy of doing so in a clinical trial. I’d rather not rely on RFK Jr’s magical ability to diagnose “mitochondrial challenges” by sight. If you feel the same way, then you also value statistics as a tool for ex post inference.

Regarding (ii): If one's only consideration is ex ante policy, then the precise details of the statistical procedure don't matter. Should we set the significance level to 0.05 or 0.01? Should we use asymptotic or non-asymptotic tests? Should we use outlier-robust methods? Should we report error probabilities or risk control? Do we use FDR control or family-wise error rate? Should we report p-values or e-values (a question close to my own heart)?

From the perspective of ex ante policy, what matters is only that the procedure is transparent, well understood, and fixed ahead of time. But an infinite number of procedures satisfy these criteria. How do you justify the use of one statistical method over another? That is, on what basis do you advocate that ex ante policy be set?

Suppose we set totally arbitrary regulatory rules on the adoption of a new drug. One hundred drugs get proposed, and our rule is: select one at random. This is clear, unbiased, and transparent—everything you could want in an ex ante policy qua policy. It is also clearly unsatisfactory in nearly every way imaginable, most notably because the adoption of a new drug under this scheme has no relationship to its actual effect.

I doubt Recht would advocate for such a policy. But why? Because the policy has terrible properties as a tool of ex post inference!

When we adopt one statistical method over another, we appeal to the method's properties as a tool of inference, i.e., its ability to infer various parameters from the data. Any justification of using one statistical method over another cashes out in some reason like: "we think this gets us adequately close to the truth of the matter." Different methods have different properties, and they are more or less efficient at doing different things. We obviously do, and we obviously should, take that into account when setting policy.

Overall, Recht's take is a refreshing reminder of an important side of statistics. I like the differentiation between ex ante policy and ex post inference, and I'll be using these terms going forward. But I think his focus on ex ante policy leaves an unwary reader with the sense that ex post inference either doesn’t work or doesn't matter. (Again, it's unclear to me to what extent Recht himself actually believes this.) And that's importantly wrong.

Subscribe now

You can also watch me ask Ben about some of this in our conversation:

William Stoner entered the University of Missouri as a freshman in the year 1910, at the age of nineteen. Eight years later, during the height of World War I, he received his Doctor of Philosophy degree and accepted an instructorship at the same University, where he taught until his death in 1956. He did not rise above the rank of assistant professor, and few students remembered him with any sharpness after they had taken his courses.

If you’re worried about an uninspiring read at this point, the second paragraph does nothing to assuage your concerns:

An occasional student who comes upon the name may wonder idly who William Stoner was, but he seldom pursues his curiosity beyond a casual question. Stoner's colleagues, who held him in no particular esteem when he was alive, speak of him rarely now; to the older ones, his name is a reminder of the end that awaits them all, and to the younger ones it is merely a sound which evokes no sense of the past and no identity with which they can associate themselves or their careers.

True to this summary, Stoner is a book in which nothing much happens—outwardly, at least. It’s the world's most subdued bildungsroman, following a quiet youth turned quiet professor who does not become famous, get the girl, or always make the right decision.

And yet, in a NYT book review, Morris Dickstein said of Stoner: "it is a perfect novel, so well told and beautifully written, so deeply moving, that it takes your breath away." If you search for reviews online, you'll quickly learn that the book has something like a cult following, especially among men. "A masterpiece", "devastatingly moving", "only book to have ever affected me like this", "one of the most impactful books I've ever read in my 68 years," are some of the responses you'll stumble across.

I endorse the praise. Stoner is my favorite book, and I expect that I will reread it every few years until I die. It is, however, frustratingly difficult to articulate why it's so good. But a clue is contained in the title of Dickstein's review, which is called "The Inner Lives of Men."

A developmental arc that most of us experience is realizing that we are devastatingly ordinary. We won't be famous. We won't change the world. Buildings and ideas won't be named after us and we won't be talked about much after we're gone. While we may leave our mark in some small way, we will be left out of history’s grand narrative.

But within an ordinary life there is still love and loss and dignity and virtue. There are difficult choices to make and principles to stand up for (or not). Stoner is the portrait of an undistinguished professor who faces the harshness of the world and doesn't always win. The portrait of someone who, like most of us, can’t satisfactorily navigate all the tensions in his life. And those tensions are not something out of Dostoevsky or Homer. Stoner is not embroiled in murderous plots or lost at sea for a decade. His problems are disarmingly mundane and recognizable.

Stoner’s wife, Edith, turns vindictive and controlling, slowly draining the warmth from his home life and poisoning his relationship with their daughter. At the university, he is confronted with the decision of whether to remain in the cocoon of academia during WWII. Later, he faces an administrative battle over a graduate student, Charles Walker, who is supported by another professor, Hollis Lomax. Walker is a fraud—verbally gifted but unwilling or unable to put in the necessary work to pass his qualifying exams.

The reader watches Stoner stoically decide which problems to focus on. He does not go to war, but later learns that one of his two best friends was killed on the front lines. He speaks up once or twice about his daughter, but Edith has him outmaneuvered. He witnesses the erosion of his relationship with his daughter who, by the end of the book, has lost her first husband and descended into alcoholism.

The battle he elects to fight is with Walker and Lomax. He stands up for the integrity of the institution, choosing to fail an undeserving student despite personal and professional repercussions. Lomax is the head of the English department and makes Stoner's academic life hellish afterwards. But Stoner was not thinking of the consequences for himself. The university introduced Stoner to his first true love—literature. It is a sacred space, deserving of his protection.

It's easy to criticize Stoner’s choices. In fact, I think Williams wants us to criticize his choices. Surely he should have fought harder for his daughter?

But life is rarely so clean. It’s unusual for there to be one grand moment where you can Fight The Power and stand up for your beliefs. There are instead a long series of small, often inconsequential, decisions whose aggregate sum defines your convictions. It's easy to have regrets in hindsight. It's harder to realize in the moment that your passivity is weakness masquerading as wisdom and will cause you great anguish later on. Stoner makes his choices and, like all of us, makes some of the right ones and some of the wrong ones.

Despite Stoner's mistakes and losses, he is not a tragic figure. His life is good in many ways. In John McGahern’s introduction to the book, he quotes a passage from John Williams’ Denver Quarterly interview:

[Stoner] had a better life than most people do, certainly. He was doing what he wanted to do, he had some feeling for what he was doing, he had some sense of the importance of the job he was doing. He was a witness to values that are important... The important thing in the novel to me is Stoner's sense of a job. Teaching to him is a job—a job in the good and honorable sense of the word. His job gave him a particular kind of identity and made him what he was... It's the love of the thing that's essential.

The quiet dignity that Stoner brings to his studies and his teaching is part of why the book feels so precious. Here is an otherwise ordinary man, whose life is neither great nor terrible, who experiences both love and despair, but who takes duty seriously. Once he has chosen his battles, he is unwilling to cave even in the face of immense pressure. The ordinariness of Stoner's life makes him relatable but his perseverance makes him a hero.

Stoner finds love in his work, but also in his relationships. A fellow graduate student, Gordon Finch, remains a loyal friend for life, eventually becoming dean of the school and helping Stoner navigate Lomax’s revenge. And, while Stoner’s marriage with Edith is cold, he finds deep romantic connection with a younger colleague, Katherine Driscoll.

Stoner and Katherine first meet as teacher and student, with her attending his graduate seminar. She stays on to teach at the university where she and Stoner carry out a delicate courtship, one we’re continually terrified that Stoner is going to bungle. But he doesn’t, and their time together is some of the happiest that either have ever known. Their pleasure in each other’s company is summed up by Katherine's comment "lust and learning, that's all there really is, isn't it?" which is possibly the best line that has ever been uttered in the English language.

Williams paints a picture of a relationship which is intoxicating for someone with my temperament:

For their life together that summer was not all love-making and talk. They learned to be together without speaking, and they got the habit of repose; Stoner brought books to Katherine's apartment and left them, until finally they had to install an extra bookcase for them. In the days they spent together Stoner found himself returning to the studies he had all but abandoned; and Katherine continued to work on the book that was to be her dissertation. For hours at a time she would sit at the tiny desk against the wall, her head bent down in intense concentration over books and papers, her slender pale neck curving and flowing out of the dark blue robe she habitually wore; Stoner sprawled in the chair or lay on the bed in like concentration.

Sadly, it doesn’t last. Lomax vindictively derails the relationship. Katherine moves away and she and Stoner never speak again, though Stoner later notices that she dedicated her first book to him.

Just as Stoner found romantic love later in his life, he came late to his love of literature. He grew up on a farm with no thought of doing anything other than farming until his father encouraged him to attend university. He went to study agriculture—to learn modern techniques which he could on the family farm when he returned. But his attention is inadvertently captured by a live reading of Shakespeare's Sonnet 73 during a required English class:

William Stoner realized that for several moments he had been holding his breath. He expelled it gently, minutely aware of his clothing moving upon his body as his breath went out of his lungs. He looked away from Sloane about the room. Light slanted from the windows and settled upon the faces of his fellow students, so that the illumination seemed to come from within them and go out against a dimness; a student blinked, and a thin shadow fell upon a cheek whose down had caught the sunlight. Stoner became aware that his fingers were unclenching their hard grip on his desk-top.

Stoner isn't particularly good at English literature. You certainly wouldn't call him gifted. But he can't help himself: he's transfixed by the subject. He spends his free time roaming the library, pulling out books here and there, taking a selection home to read at night.

This fascination leads him to drop his agricultural studies and switch his major. He can't justify his decision and he doesn't tell his parents. It takes Arthur Sloane, his first year English teacher, to point out the obvious:

Sloane leaned forward until his face was close; Stoner saw the lines on the long thin face soften, and he heard the dry mocking voice become gentle and unprotected.

"But don't you know, Mr. Stoner?" Sloane asked. "Don't you understand about yourself yet? You're going to be a teacher."

Suddenly Sloane seemed very distant, and the walls of the office receded. Stoner felt himself suspended in the wide air, and he heard his voice ask, "Are you sure?"

"I'm sure," Sloane said softly.

"How can you tell? How can you be sure?" : "It's love, Mr. Stoner," Sloane said cheerfully. "You are in love. It's as simple as that.”

We're often fed stories of the famous academic who was ferociously precocious and gifted from a young age—the Richard Feynmans, the Terence Taos, the Scott Aaronsons of the world.

But John Williams wrote an ode to a different kind of learner: the late learner. Stoner wasn't surrounded by books when he was young. He had never heard of Shakespeare before that first English class. It never occurred to him to study literature; he didn’t know that it was a subject that one could study.

But the subject duly transforms him and, alongside Sloane, the reader recognizes that literature is the first love of Stoner’s life. Until his death, the best hours of his life are spent at his desk reading.

My love for this part of the book is largely personal. While less romantic than Stoner's story, I also came late to my love of learning. In high school, I wasn’t racing home to read or watch lectures on youtube. I was trying to fit in and failing to have girls notice me. Even once university rolled around, I was only there to get a job. I started in biology, hoping—like 999 out of the other 1000 students in biology 101—to go to medical school. Education was an exercise in pragmatism, not something one did for its own sake.

Luckily, I was forced to take some math classes. (Thank God for those stupid breadth requirements.) This sent me down a rabbit hole from which I've yet to emerge, opening up a whole new world for me.

The book begins its ending with Stoner’s merciless evaluation of his own life:

Dispassionately, reasonably, he contemplated the failure that his life must appear to be. He had wanted friendship and the closeness of friendship that might hold him in the race of mankind; he had had two friends, one of whom had died senselessly before he was known, the other of whom had now withdrawn so distantly into the ranks of the living that . . . He had wanted the singleness and the still connective passion of marriage; he had had that, too, and he had not known what to do with it, and it had died. He had wanted love; and he had had love, and had relinquished it, had let it go into the chaos of potentiality. Katherine, he thought. "Katherine.”

Stoner’s reflections mirror those of the readers' throughout the novel. He, and we, begin with disappointment, wondering if his life has been a failure. But then, for the same reasons we fell in love with him, Stoner finds a self-respect:

A kind of joy came upon him, as if borne in on a summer breeze. He dimly recalled that he had been thinking of failure --as if it mattered. It seemed to him now that such thoughts were mean, unworthy of what his life had been. Dim presences gathered at the edge of his consciousness; he could not see them, but he knew that they were there, gathering their forces toward a kind of palpability he could not see or hear. He was approaching them, he knew; but there was no need to hurry.

There was a softness around him, and a languor crept upon his limbs. A sense of his own identity came upon him with a sudden force, and he felt the power of it. He was himself, and he knew what he had been.

Stoner closes his eyes a final time. The book he was holding, the only book he wrote during his academic career, falls to the floor. Very few people will ever read it. And that’s fine.

Subscribe now

We talked about this one in book club. You can listen to the episode here (this was an early one, my audio quality sucks unfortunately).

Math as discovery starts with a set of assumptions and examines the consequences, whatever they may be. You begin exploring mathematical structure like one explores a dark room, feeling your way around, bumping into an edge here and there, fumbling your way towards the light switch, slowly understanding how everything is connected. You often don't even know the flavor of the kinds of things that will be true before you start.

Math as engineering is the kind of math most of us become familiar with at school. You're asked to calculate—to find the flow rate of water into a tank, or how much hydrogen you need to balance the reaction, or the allele frequency in a population. In engineering mode you ask questions like "how can I show that?" or "how do I make this happen?"

In math as discovery, you might begin with a set of assumptions, or an intriguing definition, and then see what's true as a result. Perhaps you start with the natural numbers, and then notice that only each second number can be divided by two. Why is that? So you prove it. Are there as many even numbers as odd numbers? How do you reason about the size of infinite sets? And you're off to the races. In discovery mode you ask questions like "what is true here?" and "could it be that?" and "I wonder if?"

Math as engineering vs math as discovery somewhat tracks the distinction between pure and applied math. Applied problems tend to lend themselves to the engineering mode, whereas discovery mode is typically the domain of pure math. But the distinction is not only a feature of the problem at hand. It's also an attitude. Even in pure math, once you believe something is true, you often enter engineering mode and start asking yourself how you can prove it. Once you prove it, or fail to prove it and start wondering if it's false, you enter discovery mode again.

Math isn't the only endeavor with both a discovery mode and an engineering mode. All kinds of research, and daily life in general, consists of alternating between modes of investigation. But the idea that there’s a discovery side to math is foreign to most, because that’s not how math is taught at school. This is very sad, because that’s the magical part. It's the feeling of discovery resulting from the exploration that students of math are addicted to, not calculating derivatives.

Subscribe now

To say that a farm boy knows how to milk a cow is to say that we can send him out to the barn with an empty pail and expect him to return with milk. To say that a criminologist understands crime is not to say that we can send him out with a grant or a law and expect him to return with a lower crime rate. He is more likely to return with a report on why he has not succeeded yet, and including the inevitable need for more money, a larger staff, more sweeping powers, etc. In short, the degree of authentication of knowledge may be lower in the "higher" intellectual levels and much higher in those areas which intellectuals choose to regard as "lower." (KaD, pg 9)

A related concept is talent authentication: How is talent judged, and success allocated, in different domains? Comedians are judged by different standards than mathematicians, mechanics differently than artists.

There are often straightforward metrics for talent authentication. You're a talented sprinter if you're fast, a talented mathematician if you prove novel and interesting results, a talented mechanic if you can fix issues that others cannot. Measuring speed, checking if an engine works, or verifying whether a proof is correct are relatively straightforward (verifying correctness is easier than generating the proof to begin with).

Other domains such as dance, music, and comedy, have fewer clear, objective metrics. But there's no doubt that some dancers are better than others, or that some people can hold a note while others cannot. And while people may have different comedic and musical tastes to some extent, there are no famous comedians who consistently fail to make audiences laugh. In these domains there is still lots of objective skill determining who becomes successful.

Suppose an unknown person claims to have extraordinary talent in domain X. How are they treated? In any of the above areas—from sprinting to dancing—they will simply be asked to perform the skill. This makes breakthrough cases possible.

And we see breakthrough cases! Shows like "American Idol" and "So You Think You Can Dance" are possible precisely because there are better and worse singers and dancers. Likewise, Srinivasa Ramanujan is a famous breakthrough case in mathematics. Initially unknown outside of India, he became world renowned by sending a letter to G.H. Hardy at Cambridge. Even though Ramanujan's notation was unfamiliar, Hardy was able to judge the quality of the work because the mathematics were, at the end of day, extremely insightful.

Meanwhile, at the other end of the spectrum, there seem to be fields of endeavor which lack any objective metrics for judging talent. As my first exhibit I summon the rectangle:

John McLaughlin was a famous abstract expressionist active in the post WWII art scene. This painting, named #17 (one wonders what was wrong with the first 16), appeared at the Smithsonian. The art critic Christopher Knight wrote in the LA times that "McLaughlin occupies the top tier of 20th century American art" and that "his work rivals in imaginative depth and beauty any produced in the undisputed art capital of New York."

What I find fascinating about McLaughlin's success is that everyone can draw a rectangle (I daresay most can even draw a pair of rectangles). Why are his rectangles hanging in the best galleries in the world, while others are ignored?

Some art is perhaps susceptible to the founder's effect: we are more impressed with the person who does something first. We cherish the Beatles more than cover bands because we admire the talent required to conceive of the music, not only to play it. It's easier to copy something than to generate the original idea.

But rectangles existed before John Mclaughlin. Similarly for a lot of other abstract expressionism. I'm sure someone had urinated on metal before Andy Warhol, thrown random bits of paint on a canvas before Jackson Pollock, or hung a snow shovel on a wall before Marcel Duchamp.

Before you yell at me, let me state for the record that I'm not passing judgment on the validity of abstract expressionism as "true art." I don't doubt that it's possible to experience awe, wonder, and profound insight while staring at a rectangle (anyone who has taken psychedelics knows this is all too possible).

I'm simply interested in how the elite artists of abstract expressionism became the elite in the first place, seeing as everyone after second grade would seem to possess the requisite skillset. A "Britain's Got Talent" for abstract expressionism would not work so well.

If the bottleneck to achieve success is not objective skill, then success must be allocated in more arbitrary ways. In the art world, this power resides with the curator. He or she will decide which artists get shown; which rectangles will be praised. In the absence of objective skill, this will involve social connections, whims, and signaling your sophistication to other curators. Stephanie Barron, curator of one McLaughlin exhibition, describes him as an artist's artist, "held in high esteem by a coterie of artists, critics, collectors, and curators but otherwise virtually unknown." In other words, this is part of a status game among the cultured—the plebs simply cannot appreciate a good rectangle.

I'm picking on abstract expressionism because I think it's a particularly clear example of a domain which lacks an objective criteria for talent. But it's fruitful to analyze other areas from this angle. Take politics: It's hard to imagine testing for political talent, so we might expect political success to be governed by social connections and status hierarchies (which it certainly seemed to be, until the rise of a certain orange colored outsider).

In general, as less objective measures of talent are available, cultivating an elite will require that talent be judged by more subjective metrics, and such metrics often rely on social status.

There will be fewer Ramanujans in such fields.

Subscribe now

A false but intuitive view of science is that we run around and collect evidence in favor of various theories. The theory that has the most evidence in its favor wins and is our go to theory until the evidence starts to favor another hypothesis. This is the confirmationist view of science: We look for evidence that supports our theories, and those with more evidence in their favor are more likely to be true.

The confirmationist view sounds good until you start to formalize the notion of evidence. There are several issues, but a particularly devastating one is Hempel's paradox.

Suppose we have the theory: All monkeys have prehensile tails. (A prehensile tail is a tail that can hold objects.) The logician would write this as the conditional statement: If x is a monkey, then x has a prehensile tail. The contrapositive of this statement is: If x does not have a prehensile tail then x is not a monkey. The contrapositive of a statement is logically equivalent to the original statement, so evidence for one is evidence for the other. (Release your inner Wittgenstein and draw a simple truth table if you're unconvinced, or read wikipedia's intuitive explanation.)

Hempel's paradox is the observation that we can get endless evidence for the statement If x does not have a prehensile tail then x is not a monkey by simply noticing that things that don't have tails are, in fact, not monkeys. No tail on my air conditioner? Check. No tail on my laptop? Sweet, sweet evidence. No tail in my coffee? I think we have a pretty damn good theory on our hands!

This is extremely unsatisfying. We can get arbitrary confirmation of our theory concerning monkeys and tails by examining non-monkeys with no tails. The race to confirm the theory thus becomes a race to observe the most non-monkeys.

So where do we go from here? Except for a few rogue philosophers who are still attached to the notion of confirmation, the scientific community has solved this problem by declaring defeat on a strict, formal notion of evidence. You will never open a scientific paper to find a number that purports to be the degree of confirmation of a theory. Watson and Crick never said "our degree of confirmation in DNA being a double helix is 79.4." (You will however open papers to find precise numbers on statistical theories implied by a general theory. I'm getting to this.)

Instead of formally comparing the evidence in favor of various theories, we compare theories in other ways. We first ask that they satisfy certain criteria—such as not being post-hoc, not being more complicated than necessary, and not invoking essences.1 And we also demand that they make predictions beforehand that we can go and test. When there are multiple live hypotheses on the table we try to design tests that will distinguish them from each other, ideally refuting all but one. But we don't have a way of formally comparing the evidence for one versus the other. We might use the word evidence (in fact we use this word a lot) in reference to scientific theories, but it's always in a squishy, subjective sense.

This, it's worth noticing, is what makes science fun and interesting! If there was a clear cut way to measure the evidence for our theories, a scientist's job would be mechanical and boring. We would simply generate a bunch of hypotheses and find the one with the most evidence in its favor. Hardly exciting enough to spend five years in graduate school losing your hair and making next to no money.

To recap: you run into big issues when you try to formalize the notion of evidence for a scientific theory. So, naturally, we don't do that. We instead compare theories in other ways—both by designing tests that distinguish between different theories, but also by imposing constraints on the types of theories we consider in the first place.

II. ... did we just invalidate statistics?

But there is one discipline where the formal notion of evidence is alive and well. That discipline is statistics. Leveraging probability theory, statistical theory has developed a huge mathematical toolkit for precisely this purpose. We quantify how surprising data is under particular hypotheses (p-values, e-values), say how likely one theory is relative to another (likelihood ratios, Bayes factors), make claims about the probability of making errors in scientific discovery (type-I error rates, FDR control, FWER control), and generally provide guarantees or estimates related to how well-supported various theories are by data (confidence intervals, posterior probabilities, etc).

Statistics, in other words, is in the business of providing the kinds of formal guarantees that we just argued was doomed in science.

This should make us uneasy. What's going on here—is statistics trying to do the impossible? This seeming paradox often leads to two reactions, both of which are wrong, and both of which confuse the boundaries between science on one hand and math on the other.

The first is to believe that if statistics can do it, then science can do it as well. This leads to a misguided belief in the possibility of quantifying the precise state of our scientific knowledge at any point. (I promised myself I wouldn’t name names here …) The second reaction is the inverse of the first. It holds that if science can't do it, then surely statistics can't do it either. That is, the second reaction is to become skeptical of statistics.

Neither of these reactions is correct. The correct reaction is instead to notice that there is no paradox, because science and statistics are playing different (but closely related) games.

Statistics gets out of Hempel's paradox by fiat. The currency of statistics is mathematical models, simplifications of reality that allow evidence to be precisely quantified by construction. A model declares a priori that it only cares about looking at monkeys, not coffee cups, and also makes assumptions about how many monkeys we've seen so far, how we're sampling monkeys, how many monkeys there are in the world, and what is possible to observe every time we look at a monkey.

These restrictions are what allow us to make precise statements about evidence. They're also what makes correctly applying statistical models tricky: we want to make sure that the model matches reality close enough to be useful. But the model also needs to be simple enough that we can actually do things with it. It needs to be amenable to analysis. Finding useful but manageable models is a fine line, and constitutes the art of doing science.

III. The role of statistics in scientific discovery

To more formally delineate the world of science and statistics, it's helpful to notice how they interact. In science, we're after general theories of nature. These theories are objective and time-independent, such as "DNA is a double-helix" and "hydrogen has one proton and one electron".

A theory, let’s call it T, implies things about the world. It has consequences. Logically we can write this as T → O, where O stands for "observation", and T→ O means that if T is true then so too is O. If T is "light travels at a constant speed of 299,792,458 m/s in a vacuum," the O might be "radio waves, X-rays, and visible light should all travel at exactly the same speed (in a vacuum) despite having different frequencies."

(This is different from our monkey and prehensile tail example. There we phrased the theory itself as a conditional statement, whereas now we're discussing the implications of a theory.)

When you make implied observations precise enough, you realize they too usually have the form of conditional implications. That is, O can be written as O1 → O2 (if O1 then O2) and we get the deductive chain T→ (O_1→ O_2). In the speed of light example, O can be rephrased to be the conditional statement: if we measure the speed of radio waves, X-rays, and visible light in a vacuum, then they will all have the same speed. If T is "hydrogen has one proton and electron," then O1 → O2 might be "If we heat hydrogen gas to approximately 3000 Kelvin, then it should emit light that includes a spectral line at exactly 656.3 nanometers."

Okay, why does this matter? Because statistics is in the business of testing O1 → O2, not T→ O. We try to carefully craft our experimental conditions so that O1 is met (we set up an apparatus to send and receive radio waves, X-rays and visible light; we heat hydrogen to 3000K), and then check to see if O2 holds.

But there's lots of error and uncertainty in this process! Our measurements will never be perfect—there will be human error, instrumentation error, and overall imprecision. Have you ever tried measuring the speed of an X-ray? It's not all pancakes and blueberries.

As our theories get more sophisticated, the effects we're looking for are often miniscule. When LIGO was looking for evidence of gravitational waves, it was looking for an effect that would stretch and compress the detector's 4-kilometer arms by less than 1/10,000th the width of a proton. The detectors have to be so sensitive that they'll pick vibrations from distant traffic, farming activities, and humans just walking around. LIGO has to pick out the tiny effect of a gravitational wave from all this background noise.

Or take the discovery of the Higgs Boson in 2012 at the LHC. We often say we "found" the Higgs. What does that mean? It means that we saw the blue bump on the left side in the plot below and concluded that it was evidence of the Higgs. We made a bunch of statistical assumptions on what particle collisions would look like without the Higgs (themselves the result of previous theory and experiments) and looked at what "normal behavior" would be under these assumptions. These are the green and yellow bands. Then we saw that the observed data was significantly outside these bounds. Welcome to the party, Higgs.

You can argue that we used the wrong statistical model when detecting the Higgs, but it's clear we need statistics. Imagine if all you had was the blue dots in the plot (the "observed" data, not the "experimental" data which, confusingly, is itself part of the statistical model). Would you know that the bump on the left was significant in any way? What about the bump on the right, or the bumps in the middle? Maybe we discovered 17 new particles, actually!

The give and take between science and statistics involves generating implications O1 → O2 which are amenable to statistical models. The conclusions will, of course, always be uncertain—both because statistical conclusions themselves are never certain, but also because we're making simplifying assumptions about reality in order to do statistics in the first place.

So when we discover, or fail to discover something, it's really a statement about the statistical model. And then we hesitantly apply these conclusions to reality if we think our model is good enough. Or maybe we discovering that our simplified model of the world behaves differently than we expected, which then sends us back to refine our theories. Science progresses not by accumulating confirmatory evidence, but through this iterative dance between theoretical insight and statistical testing, none of which is ever on logically pure and rock-solid ground.

Subscribe now

Like a set of Russian dolls, Gabriel García Márquez's One Hundred Years of Solitude consists of nested storylines all following the same pattern:

(a) Innocence and curiosity leads to (b) aspiration or modernization which leads to (c) a fall of grace or corruption which ends in (d) failure, decay, and erasure.

The first doll is the town Macondo itself, founded as a utopian, isolated village where "no one ever died." But the town can't help but modernize, first by trading with gypsies who bring new technology and then by becoming the manufacturing center of the Banana company. This interaction with the outside world ends in the "banana massacre," in which thousands of villagers are slaughtered by the company's hired mercenaries. The government erases evidence of the massacre and eventually the town of Macondo itself disappears.

Next there is the Buendia family line, which begins with the marriage of José Arcadio Buendia and Ursula—the founders of Macondo. Their descendants engage in politics, commerce, and science but all fail in time, and the family eventually folds in on itself via violence and incest. The last child in the family is born with a pig's tail, symbolizing their ultimate decline. Everyone in the family is eventually lost to history.

Then there are the arcs of the individual characters, all marked by tragedy. Inspired by the gypsy Melquíades, José Arcadio Buendia becomes curious about science and develops a fascination with magnets and ice. But his thirst for knowledge becomes obsessive and he abandons his family and his responsibilities. He goes mad, and ends up tied to a tree for many years until he dies.

The colonel Aureliano Buendia, José Arcadio Buendia's son, pursues alchemy alongside his father as he's growing up. Like many young men, he's eventually seduced by a politically righteous cause which results in him joining a civil war. The 32 battles he fights (all of which he loses) see him transformed into a sadistic and cruel general. Embittered, he nearly orders the execution of his friend and then shoots himself in the chest. But even that fails. He lives out the rest of his life sad and lonely.

José Arcadio, José Arcadio Buendia's elder son, is well-intentioned as a young man and leaves Macondo on an adventure. He returns as tattooed and decadent, throwing orgies. (One cannot help but think of Judge Holden in Blood Meridian.) He is eventually murdered.

Amaranta’s childhood is full of love and potential. She courts Pietro Crespi but ends up rejecting his love and succumbs to bitterness. She deliberately chooses to sabotage her own chance at happiness, engaging in a pseudo-incestuous relationship with Aureliano José, and eventually dies in isolation.

One can keep going. Pilar repeatedly falls in love with different members of the Buendia family but each relationship withers and dies. Gaston attempts to bring aviation to Macondo but ultimately abandons the project. Renata finds joy in music but, because of her mother's violence, eventually withdraws into a lonely life of silence.

Finally, the arc appears in various little subplots. The colonel learns to craft little metal goldfish as a boy, but this activity later becomes compulsive and meaningless as he melts them all down to make more. An attempt by Visitacíon to escape war leads to the insomnia plague and the town's collective loss of memory. Remedios' beauty corrupts the hearts of men and leads to infighting. The railroad is an initial attempt to connect with the outside world, but gets abandoned after the banana massacre.

Given the appearance of this pessimistic pattern at every level of analysis, it's hard not to read the book as a commentary on the futility of aspiration. Any sort of novelty ends in disaster. We are treated to never ending cycles of hope and agency stifled either by the whims of the world or the flaws of the characters themselves. There are no silver linings. The lesson—represented by Melquíades' parchments which hold the prophecy of the destruction of Macondo—seems to be that we're destined for a miserable decline.

As a Rosling-Pinker-Deutsch pilled optimist who thinks that the world has gotten, and can continue to get, a lot better, this thesis is hard for me to swallow. And even though your enjoyment of fiction should not depend on agreeing with the author's politics, it was hard for the pessimistic themes to not affect my reading.

That said, given this thesis, I can appreciate some of the stylistic choices Marquez made. The book is hard to read—not because of the vocabulary or the DFW-style page-long sentences, but because everything is blurred together. You jump from one event to the next, one character to another, one generation to its successor, too quickly to track. Pages and chapters bleed together; page 100 feels the same as page 300.

It doesn't help that nearly all male Buendia family members have one of two names: José Arcadio or Aureliano. This is on purpose—the point isn't the difference between the characters, it's their similarities. Regardless of the specific details of their life, we're meant to notice that whatever they try ends in tragedy. The reader who gets caught up in mapping out each character, figuring out exactly who is related to who and who is sleeping with who and who has which hobbies and which vices is missing the point.

The magical realism deployed by Marquez serves a similar purpose. With it, he can introduce disasters freed from the constraints of a realistic physics. José Arcadio Buendia can be tied to a tree for dozens of years; the entire town can go without sleep for weeks; it can rain for four years and eleven months; Remedios the beauty can literally ascend to heaven; José Arcadio's blood can weave through town to his mother's house; Melquíades' parchments can foretell the history of Macondo down to every detail.

That is, Márquez uses magical realism to highlight and heighten the corruption and decay phase of our arc. It allows him to make the erasure of various characters, and eventually Macondo itself, more devastating and symbolic than reality would allow.

But after reading 400 pages of magical realism, I'm of the firm opinion that it should be a device used very sparingly by authors. The cudgel it becomes under the pen of Marquez leaves the reader exhausted—you stop having expectations about the book because anything could happen. And, as Rich points out in the episode, it limits your ability to empathize with the characters, because you don't know what kinds of solutions are available to them to solve their problems. Maybe they can use flying carpets like Aureliano, or use a magic mirror like Melquíades, or have a self-playing piano like Pietro.

Somebody should have introduced Gabriel García Márquez to Jorge Luis Borges. Borges was also a master of magical realism, but had the instinct to never stretch it across an entire novel. He knew that it was best dealt in smaller chunks. One Hundred Years of Solitude could have been condensed into a short story without losing much. If you write a book with the lesson that the details don't matter much, then adding 300 pages of unnecessary detail seems a bit aggressive.

If you're going to read Márquez's magnum opus, read the first 100 pages, understand the tone of the book, and then put it down.

Subscribe now

You can watch our beautiful faces discuss the book if you haven’t had enough of my complaining:

About 15 minutes into the conversation, Arnold makes the following disconcerting observation about the philanthropic evaluation pipeline:

[T]here was this realization we had that everybody in the chain was incentivized to find the positive result, whether it was the funder, the academic, the university, the journal, or the popular press.

Everybody wanted to find and publicize the positive result, and so there were a lot of positive results that were coming out that were either done with low bars of integrity and quality, or at the extreme end — and this is relatively rare — but bordering on absolute fraud in the industry.

It's unsurprising that there are bad incentives in philanthropy. There are bad incentives everywhere! But Arnold's point is more subtle. He's noticing that everybody in this evidence-based-philanthropy chain has incentives that point in the same direction. The researchers studying the interventions want positive results so they can publish fancy papers, the evaluation agencies want positive results so they can claim a positive impact, and the donors want positive results so they feel good about where their money is going.

The only people who are not incentivized to get positive results are the recipients of the aid themselves. No one in the village where the infamous PlayPump was installed was tempted by a system that would break down and result in grandmas pushing a heavy merry-go-round to get water.1 But the recipients are typically not a part of the evaluation pipeline, since it’s their outcomes we’re monitoring.

We might call Arnold’s observation an example of an incentive cascade: Incentive alignment among all the actors in a decision-making loop. This, surely, is not a new observation. But incentive cascade is a nice concept-handle for a problem that is annoyingly common. Consider:

• Grade inflation: Kids want better grades. Parents want their kids to have better grades. Teachers want to give good grades because they get evaluated both by the students themselves and the school on their students’ performance. Schools want their students to have good grades so they get good jobs after graduation and donate to the school. And employers want their new hires to have good grades so they can boast about employees that graduated with various latin adjectives attached to their names. Is it surprising that the median grade at Harvard is A-?

• ESG scores: Environmental, social, and governance (ESG) scores are given to companies by third-party rating agencies. Companies with good ESG scores get access to ESG-focused investment capital (e.g., Blackrock). Companies, therefore, want good ESG scores. Rating agencies want clients, so they're incentivized to give good ESG scores. And investors want more investment opportunities so they want more companies to have good ESG scores. Under this scheme, will ESG scores accurately reflect *mumbles something indistinct about whatever ESG ostensibly measures*?

• About half of academic publishing: Academics want to publish eye-catching, counterintuitive studies. Academic journals want to publish eye-catching, counterintuitive studies. Science journalists want to report on eye-catching, counterintuitive studies. Peer-review is supposed to be critical of eye-catching, counterintuitive studies, but peer-review consists of individual reviewers who (1) often know whose study they're reviewing, (2) would rather spend time doing their own research than reviewing someone else's work, and (3) know that if the standards aren't that high for getting something published then their own work has a higher chance of getting published. Astonishingly, sometimes peer-review does still work. But often it doesn't, and the result is whatever psychology has been up to for the last 30 years.

Incentive cascades are produced by aligned incentives across multiple actors. So, to break incentive cascades, you need to introduce an actor whose incentives point in another direction. You need to introduce adversaries, or at least disinterested third parties. Ideally you introduce actors with an incentive to be as truth-seeking as possible. (Cue Nassim Taleb yelling "skin in the game!")

Here are some systems which deliberately avoid incentive cascades by introducing friction:

• The Legal system. The defense and prosecution are deliberately pitted against one another, and both are incentivized to win the case.

• Open source software. Each individual developer wants to contribute to the repository. All other developers, especially the maintainers, want the contributed code to work, and to solve a particular problem. What has evolved is a robust set of norms and protocols (issues, pull requests, unit tests, code review) to ensure that the code which makes it into the repo is useful and as bug-free as possible.

• The other half of academic publishing. OK, there are parts of academia that still work well. If an academic knows that any claims will be heavily scrutinized and replication will be attempted quickly (e.g., math, physics, biology, chemistry), then this is pitting the incentives of each academic against the others.

• Capitalism?! Buyers and sellers obviously have different incentives. This ensures that prices, in a well-functioning market, accurately reflect supply and demand. Market distortions, on the other hand, can lead to incentive cascades. (Imagine if the government decided to guarantee the student loans provided by private organizations. That would be wild.)

Incentive cascades exist on a spectrum. Perfectly aligned (or perfectly orthogonal) incentives don’t exist in practice. There’s always some friction in the system—not every student has an A+, not every company has an ESG score of 100%, and people eventually noticed that PlayPumps are a terrible idea. But the idealization of an incentive cascade is a useful tool for diagnosing why some institutions work better than others.

Thanks to Cam for comments.

Central to fiction is world building: creating the set and setting of the novel and the rules of the game. Sometimes the fictional world is much like our own and the world building consists of creating new characters and social relationships. And sometimes the fictional world is radically different, involving space travel, new physics, or waking up for your sales job only to notice you’re now a giant cockroach.

As readers, we get to engage in a prolonged thought experiment as we are exposed to the world that the author has built for us. From Dostoevsky we learn what it's like to suffer the psychological consequences of committing wanton violence. From Ursula K Le Guin we learn what it's like to live in an anarcho-syndicalist society. Ted Chiang has us explore a world in which we have instant video access to anything in our past, and Nabokov puts us inside the head of a pedophile (and makes us enjoy it).

How much should an author explore the world they’ve built in the text? Fiction, after all, is not math: we shouldn't want the author to deliberately lay out their axioms and then dutifully deduce all the consequences. Much should remain hidden from the reader, with hints given only as necessary. This is why good fiction requires interpretation. It's also why some interpretations are better than others—some hew to the spirit of the world more successfully.

As readers, we shouldn’t expect the author to divulge everything, but we should expect that there is an underlying logic to the world we're exploring. We should expect that the authors know the rules they're operating with, even if those rules are not made explicit. We should, at minimum, expect that the world is self-consistent. Raskolnikov shouldn't wake up one morning, guilt-free, and decide to go to Ibiza on vacation. Odysseus shouldn't stop trying to get home. Ahab shouldn't decide to open up a pizzeria—white whale be damned.

This creates a pact between reader and writer. The writer creates a rich, consistent new world and shows part of it to the reader. They promise that they’ve thought deeply about the ideas, the logic, the structure. The reader takes the world seriously, exploring and interpreting it. They develop theories about the work, based on the premise that what happens on each page isn’t entirely random.

Or so I thought. Philip K Dick apparently never agreed to this pact. The writers at Time magazine also have not heard of this pact, seeing as they ranked Dick's novel Ubik, his 1969 classic, among the top 100 all time novels of the past 100 years.

Ubik reads like the first draft by a brilliant but drug-addled high school student. In fact, in many ways, this is probably accurate. Philip K Dick was broke, addicted to amphetamines, and wrote a 1000 page exegesis about his divine interventions after getting his wisdom teeth removed. He published 44 novels and more than a hundred short stories before dying in early fifties.

In Ubik, Dick throws a torrent of ideas at the wall and doesn't even wait around to see if something sticks. He doesn't make sure the ideas are consistent, let alone explore their consequences. The ideas are certainly interesting—Dick was arguably the first to introduce many concepts that are common in sci-fi today, like living in a simulation—but they are not packaged into a satisfying story.

Ubik takes place in a world where psychic powers are commonplace. There are at least two types of psychics: "precogs" which can see the future and "telepaths" which can listen to your thoughts. The story follows Joe Chip, a grumpy and debt-riddled employee who can’t leave his apartment because his front-door is coin operated and he doesn’t have the cash. (One imagines that Joe is a stand-in for Dick, given his preoccupation with speed.) Joe works at Runciter associates, a firm which specializes in training "inertials," people who can combat the powers of other psychics, and hiring them out to companies.

Joe meets Pat Conley, a psychic with a new and apparently unique ability: she can undo events by changing the past. Pat is the sexy femme fatale of the novel. Joe is skeptical of her abilities until she takes a shower at his apartment, then he’s sold. At one point she changes the past so that she and Joe are married. Their nuptial status is unclear for the rest of the book. If this is confusing, no problem. Pat Conley is irrelevant for the plot, and can safely be forgotten.

Joe Chip is recruited along with twelve others to go investigate some shifty psychic behavior on the moon. The man who recruits them turns out to be a human bomb, floats up to the ceiling and explodes, killing their boss Glenn Runciter, the head of Runciter associates. The team scrambles to get Runciter back to earth in order to get him into half life—a technology that preserves the recently deceased in a state of cold-pac so their consciousness can survive and communicate with the living.

But odd things start happening to Joe and team. The coffee goes bad instantly, their cigarettes go stale, their technology reverts to earlier forms. They start receiving messages from the presumed-to-be-dead Runciter at the bottom of urinals. Surprise: they're the ones in half life. Surprise number two: They're being hunted by a psychotic-or-perhaps-simply-tragically-misunderstood teenager named Jory (mentioned once in the story previously before turning out to be the arch-villain). Jory has learned to eat other half-lifers to extend his own half-life, and by God Jory is hungry.

Thankfully, Jory has been kept at bay by Runciter's extremely young and attractive wife Ella, who herself has been in half life this whole time. Ella has been using Ubik to battle Jory in some undisclosed way, a product which halts the decaying process. Us readers have seen this mysterious substance before. Each chapter opens with an advertisement for Ubik as different products: hair spray, instant coffee, car wax, deodorant—usually promising to preserve or freshen whatever you put it on.

The novel ends with Joe Chip taking over for Ella in the eternal battle against Jory because Ella has pressing business elsewhere. Initially Joe can't find any Ubik but then, always quick on his feet, he thinks really really hard about it and a can of Ubik appears. Close call.

Ubik is certainly fun. But a great novel should be fun, consistent, and thought-provoking. The best a reader can do with Ubik is to nod at some of the ideas and themes. Radical ontological uncertainty that almost certainly inspired the matrix? Nice. References to gnosticism and an eternal battle between good and evil? Check. The pervasiveness of advertisement and the commodification of death? Great.

But there’s a reason appreciation for Ubik tends to stop here. You can’t push the analysis much further, because you can’t trust that Dick himself thought any more about the structure of the world beyond what’s on the page. How does the economy function in a world full of telepaths? There goes the stock market. What was the point of Pat Conley? Why introduce psychic powers at all when they have nothing to do with the half-life battle? Who rigged the human bomb? Why send thirteen people to the moon at all? That entire subplot vanishes. Why do some things decay but not others? What the hell is Ubik?

If readers are unable to have a debate about the interpretation of a book because no one trusts that the world building is coherent, then you haven't written one of the top 100 books of all time. You've written a sci-fi pamphlet, albeit an entertaining one.

One of the most significant costs is a lack of deeply rooted community—the web of relationships that result from intimate long term friendships. The kinds of friendships that are forged by witnessing someone's journey through life's victories and defeats; seeing them fall in and out of love, get hired and fired, have children, and experience death.

Deeply rooted community is easier to create in person. Maintaining relationships from a distance is challenging, and growing relationships is even harder. You're not around to drive friends to the hospital in the middle of the night, or to rush to their house to help fix a leak. You can try and create closeness with weekly zoom calls and spontaneous audio messages, but it’s the rare friendship that can sustain emotional proximity in the absence of physical proximity.

Leaving a community gets a little harder every time it happens. You leave behind another cohort of friends, recognizing that if you had stayed, many of them would have been your closest friends for the rest of your life. And while some of them will stay close friends, many won’t. Many of them will drift perilously close to "acquaintances," that dreaded word that signifies the impossibility of good and heartfelt conversation.

When you move, you will grow a new community. But it’s hard. It takes time to get to know people, and it’s easy for new relationships to become shallow and ephemeral. And the further people are along in their lives, the more their routine is set. They might not be looking for new friends. A certain malaise sets in as you look to build community over and over again, in different cities and countries across the world.

I’ve had several sets of friends who I thought would be groomsmen at my wedding, only for those relationships to become too distant for that to make sense. I’ve lost several serious girlfriends, because one or both of us moved too far away. The slow accumulation of these casualties has made me realize that those who choose not to move might be onto something. I’m starting to understand the wisdom of staying in one place.

None of this is to say that moving is a net negative. There is tremendous upside: meeting new people, encountering new cultures, and chasing opportunity. Opportunity is not equally distributed across the world, and odds are that pursuing the next exciting job, or the best education, will require moving somewhere new.

There are also less obvious benefits. Moving around forces a certain amount of independence and maturity—both social and financial. You have to make choices about savings and investments, about rent and groceries, about banks and insurance which vary from city to city and country to country. You learn to make friends quickly, otherwise you'll be unbearably lonely. You learn to do things alone, like walk into a bar and talk to strangers. All of this creates a general sense of agency—you discover that the world is navigable and intelligible.

I don’t wish I had lived in only one place as an adult. But I do wish my 20 year old self had been more aware of the costs of moving so much. Unfortunately, my 20 year old self was acting under asymmetric information—the benefits were apparent but the costs were hidden. I barely knew what community was yet. And 20 year olds have more leverage over their futures than their 30 year old counterparts.

This lesson is particularly sharp for people like me—people who are addicted to ideas in their twenties and want to go wherever they can to most easily study those ideas. It takes some time to realize that you should be equally as addicted to people.

Thanks to Fran and Cam for comments.

This obsession resulted in someone who would only drink instant coffee because anything else takes too much time.Someone who would eat only simple foods that didn't require preparation—like apples and carrots—so that he didn't waste his time cooking. He exercised on a stationary bike so that he could read philosophy. His only question when meeting someone new was: What is your book about? In the last decade of his life he all but refused to socialize, moving to a house in the countryside so that he could work in peace. Meetings with students, when they happened, would famously last anywhere from four to 18 hours as he investigated their ideas with the ferocity of someone who thought those ideas had to save the world.

I read Parfit with the goal of understanding how such a mind works, how it develops. What motivates someone to dedicate so much of their waking life to ethics? How does someone become so possessed by the questions of moral philosophy?

Parfit, not for lack of trying, is unable to fully grapple with this question. I don’t view this as the fault of the book—I don't think a biographer could do a better job than Edmonds. Parfit's life was a mystery that he kept well hidden. There's only so much a biographer can do.

In fact, the biography makes the mystery of Parfit's obsession even more poignant. Edmonds highlights that he was not always so monomaniacally focused on ethics. In fact, his younger days were marked by a dazzling array of interests. At Eton (a boys only boarding school in the UK), he played the piano and the trumpet and participated in orchestra competitions. He was the president of the debate society and was the editor of the Eton chronicle. He was a member of the chess club, the literary society, and a bebop group. He was part of the drama club, playing Antony in the school's performance of Antony and Cleopatra. He was social and cheerful.

After Eton and before starting at Oxford, Parfit was offered a position with New Yorker magazine and took it, where he worked as a researcher for the "The Talk of the Town" feature column. Once at Oxford, Parfit opted to study history instead of philosophy. Clearly, he was not yet convinced that life’s biggest questions were in meta-ethics. (Oddly, Parfit never actually received a formal degree in philosophy, resulting in some criticizing him for a lack of philosophical depth later in his career.) Overall, Parfit's early life was marked by a surprising engagement with areas outside of philosophy. As Edmonds writes, "Those who knew him only as a narrowly and feverishly focused adult would be startled by the range of his interests [in his youth]".

One thing was clear from Parfit's early days: he was brilliant. He was consistently at the top of the class at the Dragon School (a prep school for pre-teens), winning the top scholarship to study at Eton. Then, at Eton, Parfit "collected an almost embarrassing array of prizes in his final year: the chess prize, the essay prize, the Latin prose prize, an art prize, the school prize for French, the reading prize, the English literature prize, and the overall Gold Medal prize." All in all he won sixteen awards while there, an unprecedented number according to Eton's archivist. He proceeded to be awarded the Brackenbury, Balliol college's top scholarship to study at Oxford.

At Oxford, Parfit won the HWC Davis Prize for the university's best performing history candidate, and the Gibbs Prize for the highest average mark on a set of history essays. Writing in support of his application to Oxford, his history teacher at Eton said "I can say, unhesitatingly, that Mr Parfit is the best historian I have come across in my fourteen years’ teaching experience." After Oxford, Parfit applied for a Harkness fellowship, the American equivalent of the Rhodes Scholarship. In support of his application, his Oxford tutor described him as "exceptionally gifted", "the ablest pupil I have ever had," and "an absolutely first rate man, brilliant, charming and yet disarmingly modest." Unsurprisingly for anyone, he was awarded the fellowship, which allowed him to spend two years at universities in the US.

Multiple US universities fought to have him visit. In the end, he decided to spend most of his time at NYU and Harvard. Edmonds hints that it was during this time in the US that Parfit fell in love with philosophy and decided to dedicate his life to it. He arrived in the US wanting to learn more about various subjects, including sociology, psychology, and philosophy. But he emerged from his two years in the US hooked on philosophy and, encouraged by some of the best philosophers in America, returned to Oxford to pursue a BPhil—a two year philosophy degree.

This was in 1967 and, unbeknownst to Parfit at that time, marked the beginning of his lifelong journey as an Oxford academic. He won an All Souls fellowship—widely considered Oxford's most prestigious fellowship—at the beginning of his BPhil, granting him seven years of fully-funded time to pursue his own research with no other obligations. After those seven years he won an All Souls Junior Research fellowship which gave him another seven years of freedom. At some point along the way Parfit's degree status was lost, and he became the equivalent of a full professor without ever receiving anything higher than an undergraduate degree.

It was during his time at All Souls that his personality began to change. He became a recluse, valuing only time he spent working on philosophy. He became "intensely uncomfortable in the normal social world," and avoided social events unless they were work-related. When an ex-student invited him to his wedding, he "wrote a nice note to say he couldn’t come because, firstly, at these occasions there was no time to have meaningful conversations and, secondly, that he had to maximize his work time."

While most of his old friends found it in their hearts to forgive his quirks, his antics made him increasingly difficult to put up with.

His social awkwardness grew more debilitating. If there was a dinner after an event, he would fret about seating arrangements and whom he would have to converse with. A friend from his undergraduate days, Deirdre Wilson, who knew him sufficiently well to have spent a few weeks living at 5 Northmoor Road after her Finals, found herself sitting next to him at an All Souls dinner. She had been looking forward to reminiscing about the past and reminded him of their old friendship. ‘He said, “I don’t talk about the past,” and he turned away and didn’t address another word to me for the rest of the evening.’

As his obsession was deepening, Parfit found small talk increasingly difficult. When faced with a new conversation he had two signature openings: What are you working on, and what are you thinking of as the title of your book? When non-philosophers were elected as Prize Fellows at All Souls, Parfit would suggest they abandon their discipline and take up philosophy because he felt it was more important.

Somewhat astonishingly, Parfit did get married. His courtship of the philosopher Janet Radcliffe Richards was “protracted and decidedly strange.” It began with a screening process: he read her book, The Skeptical Feminist, to judge the quality of her arguments. She passed. But it’s hard to say the relationship was a happy one. Philosophy remained Parfit’s primary, if not only, interest:

Derek and Janet were together for over three decades, but Janet is clear-eyed about her position in the hierarchy: ‘I was a side show in his life. The real show was philosophy.’ She also wrote, ‘I can’t think of anything we did together that wasn’t what he wanted to do.’ All the concessions in the relationship were made by her.

They were together for three decades before deciding to get married. This was Janet’s idea, and was mostly for pragmatic reasons. Parfit didn’t invite anyone and there was no honeymoon because he was working; it was “out of the question.” In On What Matters, his magnum opus, Parfit acknowledges hundreds of people for help with the book. Janet’s name is absent.

It was not only his social attitudes that changed while he was at All Souls, it was also his work style. The early Parfit longed for passionate disagreement and debate. But the later Parfit stopped listening to opposing views. He became rigid and dogmatic:

It happened gradually, but, as a philosophical interlocutor, Parfit slowly ceased to be a receptive listener who would give a sympathetic hearing to an opposing position. The gymnastic imagination, with its shape-shifting flexibility, receded. Its place was taken by a more rigid mind. This was manifest in the narrowing of his interests. There had been a time when he would become engaged with a myriad of philosophical questions that his colleagues were grappling with. Now he was only interested in the few issues that mattered to him. Almost everything he did became an extension of his own project.

What was the cause of Parfit's descent into intense monomania? Edmonds discusses several possibilities. Perhaps it was the pressure he faced at All Souls when trying to get promoted. This forced him to complete his first book, Reasons and Persons, on a deadline, resulting in two years of intense work. Or perhaps he had always been like this, but had hidden his real personality from the world until he received a Senior Research Fellowship in 1984, securing a position at All Souls for life. Maybe this allowed him to finally give in to his authentic self. Or perhaps Parfit was autistic, a diagnosis Edmonds had believed when he started writing the book, but later disavowed.

Or perhaps it was the work itself. Parfit became focused on "unifying ethics"—showing morality is real and objective, and that the major philosophical theories of deontology, consequentialism, and contractualism are in fact one and the same once their kinks are ironed out. Parfit thus came to focus more and more on “meta-ethics”—the study of the foundations and nature of morality—instead of practical ethics or any other area that used to interest him, such as personal identity or population growth.

He spent the last twenty-five years of his life anguished by philosophical disagreements he had with other philosophers. In particular, he grew increasingly upset that many serious philosophers believed that there was no objective basis for morality. He felt that he had to demonstrate that secular morality— morality without God— was objective, and that it had rational foundations. Just as there were facts about animals and flowers, stones and waterfalls, books and laptops, so there were facts about morality.

He genuinely believed that if he failed to show this, his existence would have been futile. And not just his existence. If morality was not objective, all our lives were meaningless. The need to refute this, the need to save morality, was a heavy emotional as well as an intellectual burden. How he came to bear this burden, and how it shaped him from being a precocious and out going history student into a monastically inclined philosopher obsessed with solving the toughest moral ques-tions, is the subject of this book.

Whichever hypothesis you prefer, it's hard not to read Parfit as a tragedy. It's the story of a brilliant young man who could have used his astonishing mind to tackle any number of important questions but instead became lost, both personally and intellectually, in the miasma of meta-ethics. Many moral philosophers will shudder at this description, attributing to Parfit some sort of God-like status in their field. Indeed, Edmonds himself compares Parfit to other philosophical giants such as Immanuel Kant.

I think such comparisons are misguided, and are often inspired more by Parfit’s legendary personality than by a sober reflection of his life’s work. I think we can conclusively say that Parfit's project failed. His final tome, On What Matters, running more than 1,900 pages, is rarely mentioned by anyone, let alone read. Has it solved ethics? Do people treat consequentialism and deontology as the same moral theory? Not even close.

Those searching for the impact of Parfit’s work will often locate it in the Effective Altruism movement. But beyond a general encouragement to do the most good possible, it’s hard to say precisely what this influence was. Edmonds, for one, thinks that Parfit encouraged the movement to “swivel its orientation to the long term, rather than focusing solely on the present.” This would make Parfit one of the grandfathers of longtermism which, admittedly, I don’t find to be particularly admirable.

But even if you think the focus on longtermism is a good thing, Parfit’s support of such ideas came about as a result of his views on population ethics, which was work done earlier in his career. In fact, Parfit’s influence was nearly inversely correlated with his focus on meta-ethics and his embrace of social seclusion.

Parfit’s first book, Reasons and Persons, is the more influential of his two major works (17,000 citations compared with 3,000). And it was written on the basis of ideas he had at the beginning of his career. Perhaps his most well-known work overall is on the nature of personal identity, which is the first philosophy paper he ever wrote. In other words, as Parfit became ever more obsessed with philosophy and cut himself off from the world, both he and his philosophy suffered.

Was his sacrifice worth it? At the end of his career, All Souls threw him a retirement party and told him he could invite friends. He didn’t invite anyone because, as he told Janet, ‘I don’t really have any friends.’ It seems clear that, in the end, Parfit did not figure out what mattered.

Thanks to Philippe-Antoine Hoyeck for catching a typo in Parfit’s book title, and to Royal Quaye for correcting “New York magazine” to “New Yorker magazine.”

Being dogmatic bears personal costs. For one, you can lose friends because you're an annoying conversation partner. Second, it torpedoes your ability to see the world clearly, making it harder to navigate. A mother convinced that vaccines cause autism denies herself and her child the benefits of modern medicine. A teenager convinced the world is about to end might dedicate their life to radical, counterproductive policies. Whatever idea you hold dogmatically (there is at least one, dear reader) people do not envy you. You are cutting yourself off from exploring new ideas, ideas that might dramatically increase your quality of life.

So here is the paradox. While dogmatism is negative for an individual, it can be socially beneficial. A dogmatic person will find the strongest version of their arguments and find all the flaws in their opponents' arguments. This makes it easier for third parties to evaluate the strongest arguments on both sides and come nearer the truth.

This is, after all, precisely how a court of law works. We don't assign neutral observers to sift through the evidence and come to an unbiased conclusion. We pit the prosecution and defense against each other, two teams which refuse to give an inch to the other. And we hope the jury—who ideally have no personal connection to the trial—will be swayed by the strongest arguments. We trust that truth will emerge as a byproduct of the ruthless battle of ideas.

Dogmatism functions as nearly the opposite of a tragedy of the commons, a situation in which the reasonable thing for each individual person to do results in a worse outcome for everyone (e.g., overfishing a local bay). Being dogmatic, on the other hand, is something like a boon of the commons: it is bad for the individual but good for the group.

Of course, it's an exaggeration to say that dogmatism is always bad for the individual and good for the group. Dogma can be used to gain membership to a group that will provide social and emotional support, often genuinely making your life better. A common example is religious communities. A religious person is not (typically) suffering the consequences of their metaphysical beliefs and gains a sense of community and solidarity. Holding a dogmatic religious belief is often benefitting someone more than it is harming them. (This is what rational-irrationality is all about: when it is cheap to believe something it is rational to believe it.) Religion is only one example—politics and activism have similar dynamics.

Likewise, dogmatism can be bad at the group level if it causes affective polarization (increased hostility to those who disagree with you) in the group. Such polarization leads to lower trust societies, which is generally a bad thing. If everyone becomes dogmatic, then there is nobody left to benefit from the resulting battle of ideas. Dogmatism works best when a few voices are steadfast on either side of an issue, and the majority of people are capable of having their minds changed based on the resulting conflagration.

Still, I think that dogmatism is underrated in general. Dogmatists provide the best arguments in favor of their position because they are determined to prove the other side wrong. My personal views on a variety of subjects can be traced back to arguments between dogmatists—whether atheism vs theism, environmental pessimism vs optimism, the woke vs anti-woke on critical theory, Israel and Palestine, US foreign policy, the dangers of AGI, the benefits and harms of social media, libertarianism and socialism, or critical rationalism vs Bayesianism.

Do I think the people publicly participating in these debates are neutral arbiters of the evidence? Not at all. In fact, I would be extremely surprised if any of them ever changed their mind. Paul Ehrlich, famous environmental pessimist, has been writing books and papers about the end of the world for more than 50 years. But his dogmatism on this point has been beneficial for me. In particular, examining in what ways he's been wrong has influenced my views on the limitations of mathematical models.

But surely, you scream, it's possible to marshall good arguments for a position without being a dogmatist about it? Surely we can have civil disagreements led by rational, open-minded, good-willed people who just want to get to the truth and aren't wed to a particular position? This is, of course, possible to some extent. But I think this optimistic picture misunderstands what we know about human psychology. Who has the most incentive to seek out every possible argument in their favor, and to spot every possible flaw in their opponent's position? People who are emotionally and tribally attached to a particular view.

Dogmatists lay anchor points for public conversation, helping the rest of us navigate the complex world of ideas more efficiently. They generate the strongest arguments for their side so that the rest of us can feel confident that we are hearing the best case possible—a public service.

Thanks to Cam for comments.

The constrained vision holds that human nature is fundamentally limited in terms of knowledge, morality, and foresight. It can be bent by incentives but not erased. Individuals will always be self-interested, tribal, and have bounded knowledge. Social policy is an endless series of trade offs between imperfect solutions. Progress is possible, but it comes from the slow evolution of decentralized processes—markets, common law, and cultural tradition—which aggregate the tacit knowledge of millions of participants.

The constrained vision is wary of our ability to design new institutions from scratch. The accumulated societal wisdom embodied in traditions is superior to any individual's reasoning. Grand schemes that redesign social systems from the ground up inevitably produce unintended consequences. The constrained vision therefore emphasizes incremental change, empirical evidence over theory, and process over intentions.

The unconstrained vision holds that human nature is malleable and perfectable. The goal of social policy is to provide comprehensive solutions (eliminate poverty, war, racism) rather than to merely manage trade-offs. The unconstrained vision places its faith in the expert application of reason, favoring the deliberate design of social systems and institutions. It is more comfortable with centralized problem-solving, and skeptical of tradition and decentralized decision-making. Outcomes can and should be optimized, and we should judge reform efforts by how close reality approaches an ideal.

Sowell uses the constrained and unconstrained vision to explain political disagreements. Those set on radical change often cannot fathom why there are those who are opposed—especially when those people have trouble articulating explicit reasons beyond simply citing tradition. Those skeptical of change cannot understand why some are so disposed to risk what we have built to chase some abstract ideal like “justice” or “fairness” or “equity”.

While Sowell emphasizes that his visions transcend political labels, it's hard to dispute that the unconstrained vision is typically held by progressives and the constrained by conservatives.1 The archetypical Burkean conservative has a deep admiration for tradition and is skeptical of radical change. The archetypical progressive is pointing out some current injustice, demanding that the system be changed until the injustice is uprooted. Their belief that the system can be deliberately designed to eradicate injustice leads them to think that change is being avoided only because of the evil people in power (corporations, billionaires, and so on).

With this framing of political disagreement in mind, Donald Trump's rise to the helm of conservative politics is all the more surprising. Trump's administration represents the unconstrained vision in full force.2 They are quickly reversing relationships with longstanding allies, deliberately delivering some of the biggest shocks to the world financial system in modern history, radically dismantling longstanding institutions, and asserting control over universities and their funding.

Regardless of whether you think these policies are good or bad, they clearly embody the unconstrained vision. They show a willingness to redesign the system from scratch, believing that much national and international order can be intentionally and successfully manipulated. Policies are not justified by weighing up merits and demerits, instead they appeal to fairness and justice, framing the discussion in terms of victimization (either America as a whole, or subsets of American culture). There's no incremental rollout of these policies, no discussion of potential tradeoffs. Insofar as conservatives have the constrained vision of human nature, Trump is the most anti-conservative president to ever hold office.

Of the two American parties, the unconstrained vision used to be the purview of the democrats. If, fifteen years ago, you had been told that one of the two parties was going to radically change the structure of government overnight, which party would you have guessed? Republicans used to write op-eps criticizing the democrats for adopting the unconstrained vision, with titles like "Unconstrained, Un-conservative".3 Insane policies of the left, like defunding the police, used to be mocked by conservatives for being utopian. Then conservatives turned around and adopted their own utopian defunding scheme in the form of DOGE.

Specific policies aside, the careening of the American right towards the unconstrained vision is worrying precisely because the left is already prone to it. A healthy tension often emerges between the constrained and unconstrained visions: the latter pushing for drastic reform, the former looking to maintain what we’ve built. This dialectic leads to change, but change that is slow, manageable, realistic, and attuned to tradeoffs. This tension is, in my view, responsible for the remarkable progress that we've witnessed over the last several hundred years.

Two parties susceptible to utopian thinking is destabilizing for American—and global—politics. I'm hoping the constrained vision makes a comeback.

It's now a common complaint that many empirical sciences are an epistemic nightmare. Psychology, medicine, nutrition science, cancer research, economics, and other disciplines have dealt with embarrassing replication crises over the past decade. All have massive, conflicting literatures, suggesting that much of the published research is wrong.

One reason for the unreliability of this research is the unwieldy application of statistical tests. Journal reviewers require that empirical claims come with evidence, and this evidence is quantified using statistics. This, of course, incentivizes researchers to engage in shoddy practices (consciously or not) to pass the required tests and get their paper published.

Statistician Andrew Gelman thinks many researchers are doing this consciously:

I think that many or most researchers think of statistical tests as a kind of annoying paperwork, a set of forms they need to fill out in order to get their work published. That’s an impression I’ve had of researchers for a long time: they feel they already know the truth–they conducted the damn experiment already!–so then they find whatever p-values are necessary to satisfy the reviewers.

This has been a grievance for years, and statisticians have been standing on the sidelines watching in horror as their favorite statistical tools get hijacked. Gelman goes on to argue that that researchers aren't just manipulating the statistics to get what they want, they are manipulating their entire research agenda:

My new thought (or, new to me, at least) is that many researchers also think of research itself as a kind of annoying paperwork. They already know what they want to say about disparities, or climate change, or evolutionary psychology, or the 2020 election, and, yeah, they’ll do an experiment or a data analysis or whatever, but that’s just a means to an existing end. They’re not doing science. My thought in the above-linked post is that there maybe is so much of this hackwork that it overwhelms the system.

In other words, we're swimming in an ocean of junk science because scientists are biased. Researchers come to the table with preconceived notions of how the world does and ought to work. They set out to prove their ideas, not to critically assess the evidence and uncover the truth.

This suggests the question: Does science only work if scientists are unbiased? Does each scientist need to be engaged in a fearless pursuit of objectivity for the system to be reliable? Need we hope that each individual scientist is a beacon of hope and light, always ready to sacrifice their personal beliefs at the altar of truth?

It would of course be nice if scientists were such saintly figures. But scientists are humans, and given everything we know about human nature, this is an unreasonable expectation. We now have extensive evidence that humans are biased and stubborn, not easily swayed by evidence that contradicts our preexisting beliefs, especially when that evidence does not conform with our social group.

But surely we can't pin our hopes on scientific progress on the idea of unbiased scientists? No: the whole point of scientific institutions is to create a system whereby progress is made despite the flaws of the individual researcher.

In fact, the bias of individual scientists is often a good thing. The system works better when scientists are adamant, even dogmatic, about their hypotheses. It means that the scientific community as a whole is exposed to the strongest version of each idea. (You might call this the paradox of dogmatism.) If people gave up on their ideas at the first sign of trouble, many correct ideas would have been forgotten. For example:

• Barry Marshall and Robin Warren proposed that stomach ulcers were caused by bacteria rather than stress or spicy food. Doctors were in disbelief but they persisted. Marshall famously drank a broth containing H. pylori, the bacteria in question, to prove his point, which eventually resulted in a Nobel Prize. As The Lancet documents: Warren recalls:

Warren recalls: “Every time I spoke to a clinician they would say, ‘Robin, if these bacteria are causing it as you say, why hasn't it been described before?'.” Orthodox medical teaching at the time was that bacteria did not grow in a normal stomach. However, as Warren wrote in the 2002 book Helicobacter Pioneers, “I preferred to believe my eyes, not the medical textbooks or the medical fraternity.” Marshall also believed what he saw through Warren's microscope. “The first time I sat down with him he didn't really have any trouble convincing me there were these organisms in the stomach”, he told The Lancet. “As far as I was concerned he was right, and I thought this was a unique observation.”

• Lynn Margulis introduced symbiogenesis which is now the leading account of the origin of eukaryotic cells. After unanimous opposition for many years, her ideas finally gained traction. In 1995, Richard Dawkins wrote about her tenacity:

I greatly admire Lynn Margulis's sheer courage and stamina in sticking by the endosymbiosis theory, and carrying it through from being an unorthodoxy to an orthodoxy. ... This is one of the great achievements of twentieth-century evolutionary biology, and I greatly admire her for it.

• Ignaz Semmelweis observed that when doctors washed their hands before delivering babies, maternal mortality dropped dramatically. His findings contradicted miasma theory (the prevailing belief that "bad air" caused disease). He committed to an asylum because of his ideas, but was ultimately vindicated by the germ theory of disease.

• There are many others: Alfred Wegener's theories of continental drift, Barbara McClintock's "jumping genes" theory; Stanley Prusiner's prions and, of course, Galileo.

Even if it were always a bug and not a feature, the dogged commitment of scientists to their own theories is not going to stop. Our goal should be to design scientific institutions, and cultivate cultural norms, which constrain and channel the fallibility of individual scientists. That is, instead of asking the question "how do we make scientists less biased?" we should ask the question "what kinds of scientific practices, both formal and informal, result in scientific progress despite dogmatic researchers?"

There is, undoubtedly, a lot of progress to make on this front. Practices like peer-review, preregistration, and meta-analyses are good but could be improved. Other things to try are overlay journals, incentivizing replications, incentivizing the publication of null results, harshly penalizing scientific fraud, and using more sophisticated statistical tools that make p-hacking harder. We could require higher standards for publications, demand more adversarial collaborations, incentivize the sharing of all raw data and code. There is a lot of innovation to be had—considering how long homo sapiens have been around, we haven't been doing science for very long.

I agree with Gelman that it's a mess out there—many empirical sciences are in crisis, and much “research” of the past 20 years should be heavily discounted. The amount of junk science out there is horrifying, and the system feels broken in all sorts of ways. But our solution can't be to yell at individual scientists until they're robotic truth seekers. It has to be to design the system in a way that incrementally approaches truth despite the inevitable bias of the researchers.

In Statistical evidence: A likelihood paradigm, Richard Royall asks three questions that various schools of thought in the foundations of statistics have tried to answer in the past century:

1. What should I believe?

2. What should I do?

3. How much evidence does this observation provide for a hypothesis?

Royall focuses on the third question, arguing that any reasonable answer must rely on the “likelihood ratio” (a view called the law of the likelihood).

But Royall’s questions also serve as a useful frame on the debates at the foundations of statistics. They highlight the distinctions between various schools—Bayesian and frequentism, pragmatism versus puritanism, and disagreements over what the role of statistics really is and how probability should be used. Here I’ll try to show that much of the debate arises because various schools of statistical thought are trying to answer different questions.

A reader could be forgiven for staring in bewilderment at the phrase “debates at the foundations of statistics.” Are there such things? What is there to debate about tallying the number of car crashes in Missouri, or calculating the standard deviation in the heights of school children? Regrettably, this is a reasonable question because of the way statistics is taught … everywhere. The stale view of statistics taught from grade school to undergraduate haunts every grad student who, at the bar, is asked “what do you do?”

In fact, statistics has been the epicenter of more feuds and controversies than most fields. Throughout the 20th century, entire academic departments would often pick a side and staunchly refuse to hire anyone with the opposing perspective. Deborah Mayo calls these the statistics wars. Statisticians believed the very foundations of reason itself were at stake. Ronald Fisher, one of the more colorful characters in the discipline’s history, is said to have called one of his colleague’s views “horrifying [for] intellectual freedom in the west.”

Here I’ll give a brief tour of how I think about the statistics wars. I’ll try to be mostly impartial—I’ll explore where I sit personally in future posts.

A note before we begin: Statistics is written in the language of probability, and debates about one usually bleed into debates about the other. Views on the proper use of statistics often directly follow from your views on the nature of probability, and vice versa. Here I’ll touch on both the philosophy of probability and the philosophy of statistics, separating them only if appropriate.

Question 1: What should I believe?

The idea that probability can and should be used to model beliefs is held most firmly by the Bayesians, a school of thought named after the 18th century reverend Thomas Bayes.1 This is not to say that other schools think that statistics has no role to play in shaping our beliefs (it’s supposed to help us reason about the world, after all), but the Bayesians are unique in how formally they make the connection between probability and belief.

In the Bayesian conception, everything should be treated as a random quantity and everything is thus amenable to probabilistic analysis. Whereas most people would consider “the number of people who sneezed on December 21st, 1988, in Accra, Ghana” to be a fixed (deterministic) but unknown number, Bayesians treat this as a random variable drawn from some distribution. They begin with a guess for what this distribution is (a prior distribution), and then slowly change their guess as new information arrives (a friend reports they saw 12 people sneezing). The process of changing the distribution over time is known as Bayesian updating.

While one can use Bayesian methods for purely pragmatic reasons, if you push Bayesian practitioners hard enough they will typically justify the use of their methods by appealing to the Bayesian interpretation of probability, which claims that degrees of belief are represented by probabilities. This is the core claim behind the Bayesian philosophy and is the link between Bayesianism and Royall’s first question. Bayesians contend that belief is best handled using the tools of probability theory and that the role of statistics is to formally update these beliefs using data.

For Bayesians, this link between rationality and probability is more than mere analogy. They have developed frameworks to formally connect the two. The most famous of these is Cox’s theorem which posits that the beliefs of a rational agent must be treated as probabilities and obey certain arithmetical axioms. Cox was reacting to Kolmogorov’s axioms, which are the most widely used axioms of probability theory but have nothing to do with rationality or belief. Unfortunately for Cox, while his axioms are the subject of the occasional philosophy seminar, they are never used by practicing statisticians (not least because they do not handle countable additivity, a technical yet crucial property when dealing with all but the simplest sample spaces).

Most modern Bayesians appeal instead to Dutch book arguments to justify the connection between probability and belief. These say that if you assign numerical values to your beliefs and bet on them, then you are guaranteed to lose money unless your beliefs are consistent. That is, you should not bet at 3:2 odds that a coin will come up both heads and tails, otherwise a bookie can take both bets and be guaranteed to make money from you. Dutch book arguments are cleaner than Cox’s theorems, but significantly less powerful. They show only that if you are willing to assign numbers to your beliefs and want to bet on all outcomes, then your beliefs should be consistent. They don’t tell you why you should assign numbers to your beliefs in the first place, nor how to update your beliefs in the face of new evidence, a core part of Bayesianism.

A technical but important caveat: Because Bayesianism has entered the mainstream due of movements like effective altruism and Βay-area rationalism, there is naturally tremendous confusion over what exactly it means. This is not helped by people with names like Nate Gold and Liam Silver who write books confusing using Bayes’ theorem with Bayesianism itself. Everyone uses Bayes’ theorem, it’s simply a mathematical identity. The identifying feature of Bayesianism is treating unknown parameters as random quantities. Bayesians put distributions over things that other statisticians would rather die than treat as random.

Objective vs Subjective Bayesians

There are a couple of sub-schools of Bayesian thought, the main ones being subjective and objective Bayesianism. The split comes from a disagreement over how one should assign prior probabilities. Before you see any data, what should the distribution be over the number of people who sneezed in Accra, Ghana? Perhaps it should be uniform over the population, or perhaps it should be a Gaussian centered at 10?

Objective Bayesians (sometimes called logical Bayesians) are of the view that there is a single, correct prior that should be used. They typically appeal to symmetries and rational constraints to determine what it is. For instance, they would typically take the view that the correct prior to have when estimating the bias of a coin is 50/50 heads/tails because there is no reason to favor heads over tails and vice versa.

The father of objective Bayesianism is arguably Harold Jeffreys, though Keynes introduced many of the concepts first. But Keynes didn’t think that all beliefs were quantifiable, setting him apart from most Bayesians. Jeffreys introduced the Jeffreys’ prior which is often appealed to by objective Bayesians as the correct prior. Other notable objective Bayesians are Rudolf Carnap, ET Jaynes, and James Berger.

Subjective Bayesians, as you can probably guess, disagree that there is an objectively correct prior. They think of probability as something much more personal. Everyone will have a different prior to reflect their own thinking and assumptions. In the modern era, the two figures most associated with the subjectivist school are Frank Ramsey (see his 1926 paper truth and probability) and Bruno de Finetti (if you speak French, see his 1937 paper La prévision: ses lois logiques, ses sources subjectives). Other key figures in this tradition are LJ Savage and JL Doob.

Question 2: What should I do?

Battles in the history of statistics are often framed in terms of Bayesianism versus frequentism. Frequentist methods treat unknown parameters as fixed (the number of people who sneezed in Accra, Ghana isn’t random, it’s deterministic but unknown), and then perform inference on them. In other words, for the frequentist the only randomness comes from the data.

In terms of outcomes that practicing statisticians actually care about, frequentism has mostly won the day. Confidence intervals, minimax rates, type-I and type-II error guarantees are all frequentist notions of performance.2 And even when statisticians use Bayesian methods, they usually try to measure performance of those methods with frequentist notions (see, e.g., the Bernstein von-Mises theorem).

But just like Bayesianism, there are divides within frequentism as well. The most famous of these is the battle between Jerzy Neyman, Egon Pearson, and Ronald Fisher, which began in the 1920s and lasts until this day. The disagreement is usually framed in terms of hypothesis testing, but it’s really over what statistics is trying to do as a discipline. The Neyman-Pearson camp were focused on Royall’s second question, while the Fisherian camp was focused on the third.

Neyman and Pearson considered statistics to be about action; about what to do next. For Neyman in particular, statistics is a branch of decision-theory. He called mathematical statistics the search for “inductive rules for action”, meaning we’re making decisions informed by the data we’ve gathered.

The problem of testing a statistical hypothesis occurs when circumstances force us to make a choice between two courses of action: either take step A or take step B…

- Neyman, A first course in probability and statistics, 1950, pg 258.

The influence of the Neyman-Pearson approach to statistics can be felt every time universities offer classes on “decision-making under uncertainty,” which invariably use statistical decision theory to make “data-driven decisions”. In hypothesis testing in particular, we often test the null hypothesis against an alternative hypothesis, choosing to accept one and reject the other. This also comes from Neyman and Pearson, who thought it was nonsensical to simply test a single hypothesis (as opposed to Fisher, as we’ll see when studying the third question.) For them, one can only evaluate the relative merits of two hypotheses; you can’t say anything sensible about one hypothesis on its own.

You might think that the Neyman-Pearson paradigm, and Royall’s second question more generally, are still about belief. After all, won’t you only choose action A instead of action B if you believe A and not B? Isn’t talking about decision making just a sneaky way to talk about belief? Νο, for several reasons:

1. You might believe neither A nor B is true (replace true with closest to true if you’re an adamant fallibilist). Suppose you’re a doctor and a patient presents with symptoms of Malaria. Even though you’re quite sure it’s Malaria, you might order a test for yellow fever as well. If it turns out, based on the data, they’re more likely than not to have yellow fever, you might order extra screenings despite maintaining a belief that they really have malaria.

2. Utility versus belief. You work in public policy and are deciding whether to roll out a new congestion tax. The economists in your department have run a small study: implementing the tax reduces the average number of cars on the road by 10%, but also disproportionately affect the poor. While you might now believe that congestion taxes are effective, they are not effective enough to warrant implementation.

3. Lack of alternatives. Now you’re a physicist. You predict a new particle, particle X, based on the standard model of particle physics. We run some tests to look for particle X, but we don’t have sufficient evidence to reject the hypothesis that the particle doesn’t exist (the null hypothesis). Do you stop believing in the existence of the particle? Maybe, but maybe not. The standard model is our best theory. Until a coherent, alternative theory is proposed that does not have a particle X, it may not be sensible to entirely discard the idea of a new particle.

Neyman himself was adamant about the distinction between Royall’s first and second questions. He made sure to highlight the difference between action and belief:

The terms ‘accepting’ and ‘rejecting’ a statistical hypothesis are very convenient and are well established. It is important, however, to keep their exact meaning in mind and to discard various additional implications which may be suggested by intuition. Thus, to accept a hypothesis H means only to decide to take action A rather than action B. This does not mean that we necessarily believe that the hypothesis H is true. Also if the application… ‘rejects’ H, this means only that the rule prescribes action B and does not imply that we believe that H is false.

- Neyman, A first course in probability and statistics, 1950, p. 259.

Question 3: How much evidence does this provide?

Ronald Fisher—often called the father of modern statistics, seeing as he invented half the methods covered in a first year grad course—thought that statistics was in the business of quantifying evidence. This is a very natural idea and is in fact how statistics is often taught. If we have the hypothesis “smoking cigarettes causes lung cancer,” we collect data from smokers and non-smokers, see how many of each have lung cancer, and check to what extent the results refute our hypothesis.

Fisher developed the p-value—bane of students and social scientists alike—for precisely this purpose. The p-value is a number that is computed from a dataset. The smaller the p-value, the more inconsistent the data is with a given hypothesis. Legend has it that Fisher developed the p-value trying to test if his colleague Muriel could tell if milk was placed in tea before or after the hot water. (She could … probably.)

Because Neyman and Pearson were focused on action, they always thought statistical testing should involve a null and an alternative hypothesis. Accepting a hypothesis meant choosing one action over another. Fisher, on the other hand, thought it was perfectly sensible—and preferable—to examine a single hypothesis and try to quantify the evidence in its favor.

Fisher was concerned that the Neyman-Pearson paradigm confuses policy and science. He thought that statisticians should be in the business of helping people tell to what extent the data disqualifies a particular hypothesis. What to do with that information is up to others. For him, the statistician is not the decision-maker, he’s the lab tech.

Fisher, Neyman, and Pearson went back and forth, attacking each other in writing and trying to ensure that the statistics community adopted their preferred position. Fisher compared Neyman to

Russians [who] are made familiar with the ideal that research in pure science can and should be geared to technological performance, in the comprehensive organized effort of a five-year plan for the nation.

- Fisher, 1955, pg 70.

He goes on to suggest that Neyman and Pearson are sacrificing truth for economic value:

In the U.S. also the great importance of organized technology has I think made it easy to confuse the process appropriate for drawing correct conclusions, with those aimed rather at, let us say, speeding production, or saving money. There is therefore something to be gained by at least being able to think of our scientific problems in a language distinct from that of technological efficiency.

What happens in practice?

In terms of how scientists and social scientists use statistics in practice, very few are concerned with Royall’s first question. Seldom, if ever, do papers give explicit probabilities for the authors’ beliefs, nor claim a right to tell the reader what to believe. Using statistics to mathematize beliefs has mostly left industry and academia, insteading ending up in niche (or perhaps not so niche) corners of the internet.

However, statistical practice today—especially hypothesis testing—is an awkward blend of both the second and the third question. In Mindless Statistics, Gert Gigerenzer calls attention to the “null ritual”, which is the standardized hypothesis testing procedure taught in many textbooks and graduate programs to psychologists and social scientists.

The null ritual:

1. Set up a statistical null hypothesis of “no mean difference” or “zero correlation.” Don’t specify the predictions of your research hypothesis or of any alternative substantive hypotheses.

2. Use 0.05 as a convention for rejecting the null. If significant, accept your research hypothesis. Report the result as p < 0.05, p < 0.01, or p < 0.001.

3. Always perform this procedure.

There’s no alternative hypothesis specified in the null ritual, which makes it Fisherian. But we’re also making a binary decision about whether to accept or reject the null, which is firmly in the Neyman-Pearson camp. However, we also report the p-value, which is again Fisherian.

This may not seem like an issue—so what if we’re combining two perspectives? But it leads to trouble.

For one, we’re including irrelevant information. In the Neyman-Pearson paradigm, the p-value has no use beyond comparing it to the pre-chosen significance level (0.05 in the null ritual). If the p-value is less than 0.05, we reject the null hypothesis. If not, we don’t. This means that a p-value of 0.049 and 0.00001 have the exact same implication for decision-making. So what are we doing by reporting it?

Unfortunately, this extra information is not only irrelevant, it’s actively harmful. Reporting the p-value alongside the decision (accept/reject) confuses the p-value for the significance level, resulting in invalid statistical guarantees. You can read more about the issues here, but the non-technical overview is that the null ritual makes it dangerously easy to make false and misleading claims.

The upshot is that the statistics wars haven’t been fought and won, they’re still being fought.

Thanks to Rich for comments.

The Anxious Generation by Jonathan Haidt is not a good book. Sorry.

I’m tempted to end the review there but, alas, I cannot get away with that because I’m not Tyler Cowen. Instead, I’m going to treat you to 7,000 words of excruciating detail on what makes it a bad book.

But for those of you who will only read the introduction, here’s my argument in a nutshell: It’s a bad book because it radically oversimplifies what should be a nuanced conversation about certain mental health trends among adolescents.

Had the book done its job, a reader would come away with an informed picture of what the literature says on this topic. Instead, an unwary reader will emerge from the book with a wildly distorted view of the academic conversation about this question. You won’t know about the data sets that conflict with Haidt’s claims, you won’t know how many studies question his findings, you won’t know how difficult it is to even study causation in this setting, or of the many issues plaguing the studies that attempt to do so. You certainly won’t know about effect sizes, or correlation coefficients, or alternative hypotheses that explain the data. You will not be well-prepared to have informed conversations with your children about social media.

Am I saying that all books must be nuanced, careful, and academic? Must they be dry and technical, talk about statistical models, sample sizes, demand effects, and confounders? Yes, I suppose I am saying that. Or, at least, books claiming to weigh in on an important public health question (and suggest government intervention based on their conclusions!) should be held to a higher standard than regular popular science books. Nobody demanded that Haidt write this book. He chose to write it because he feels strongly about his thesis, but that doesn’t permit him to oversimplify the literature to suit his needs.

My annoyance is directed towards The Anxious Generation in particular, not Haidt in general. Outside of the book, he’s treating his thesis how we should want any social scientist to treat their favorite ideas: By soliciting feedback, being transparent with data, and engaging with critics. Haidt’s substack does this well, and he and his co-authors have open-sourced their literature reviews—a rare move in the social sciences.

I also respect Haidt for other reasons. He has been an admirably consistent voice in the fight for free speech on college campuses, sounding an early alarm about the rise of illiberalism along with Greg Lukianoff in the The Coddling of the American Mind. He founded Heterodox Academy with the goal of protecting open inquiry and increasing viewpoint diversity in what has become a left-wing silo. And he co-founded Let Grow, which encourages resilience and self-reliance in kids.

But I think Haidt’s conclusions in The Anxious Generation are premature, and that his book is likely to do more harm than good.

I. What’s the problem?

Haidt’s claim is that there is a severe decline in the mental health of adolescents—particularly adolescent girls—and that social media is a major cause of this decline. I think he’s correct about the decline (at least, in some parts of the world) but I’m skeptical that we know enough to attribute the decline to social media. But let’s first look at evidence for the decline itself.

The first thing Haidt looks at is self-reported rates of anxiety, depression, and suicidal ideation. You should pay attention to these rates, but not anchor on them too closely. Reported rates can increase for many reasons beside an actual increase in anxiety and depression. There might be less stigma around discussing mental-health, less stigma about seeking help, evolving diagnostic categories, an increased willingness to diagnose anxiety among therapists, or an increase in social status accompanying poor mental health.

The more important metric is self-harm and suicide rates. Are reported rates of anxiety translating to more girls and boys showing up in morgues and hospitals? If so, we have a real problem on our hands.

I.I. Self-harm rates

Sadly, self-harm rates do appear to be increasing in various countries, and the increases typically started around 2011-2012. As Haidt points out, this is roughly when we should expect to see social media start to have an effect on mental health if his thesis is true. The smartphone was introduced in 2007, Facebook in 2006 (for the public), Instagram in 2010, and Snapchat in 2011. But before the 2010s, it was rare for teenagers to have access to social media on their phone. In 2011, less than a quarter (23%) of teenagers (ages 12-17) had a smartphone. In 2015, 73% did. In 2016, 79% did. Haidt calls the period between 2011 and 2015 “the great rewiring,” because adolescents were spending more of their childhoods on their phones.

Here is the data for self-harm rates among adolescents girls in the US:

Haidt does not present this plot in the book. He presents only the data for girls aged 10-14 (this is where the often-quoted 189% increase comes from1). Now, clearly, something seems to be happening with girls aged 10-14 and 15-19, starting around 2010. But the increase is less among those aged 15-19 (62% instead of 189%). This is notable because 15-19 year olds are more likely to have phones than 10-14 year olds. This would seem to pose a problem for Haidt—if those adolescents more likely to own phones are seeing less of an increase in self-harm rates, are we sure it’s the phones? Unfortunately, Haidt does not discuss these data in the book, so we’re left unsure what his explanation is. He is aware of the data, however, since I got this plot from Section 2 in his collaborative review doc.

Meanwhile, boys in the US are not undergoing any significant increases in non-fatal self-harm rates:

Haidt acknowledges that the story for boys is more complicated. But he also doesn’t present this data in the book, presumably because it’s undermining his case. It’s important to keep this in the back of your head, especially when Haidt begins advocating severe, government-led restrictions on all adolescent screen time.

Let’s take a moment to put the scale of the problem into perspective. Among 10-14 year old girls, hospital admissions for non-fatal self-harm increased from an average of 100/100,000 in 2010 to roughly 300/100,000 in 2015. This is indeed a big increase percentage wise, but as Vaden points out in our episode, it’s important to keep the absolute scale in your mind as well. In a high-school class of 2,000, half of them girls, the increase represents one girl in the ER a year versus three girls a year. Obviously, we should be concerned about any teenage girls in the ER, but when deciding on public policy we need to be clear and honest about the size of the problem.

Still, a small increase in the number of girls who legitimately hurt themselves signals a larger increase in the number of girls who are anxious or depressed, but who won’t self-harm. And that’s concerning.

How are other countries doing? The internet and smartphones were both widespread across the western world by 2010, so we should expect to see similar increases across the anglosphere. And in some countries, we do.

Here’s the case in Canada (Ontario contains 40% of Canada’s population. The left figure is what Haidt presents in his substack, the right corroborates the general trend in the rest of Canada—excluding Quebec—from 2009-2014):

And here’s the UK and Australia. The data for the UK is not self-harm based on hospital admittances, but instead based on primary care records (I’m not going to pretend to know if this makes a big difference or not). Both of these plots are from Haidt’s substack; he only presents the UK data in the book.

So yes, there is a common trend across several countries. Namely, from 2010-2015, non-fatal self-harm rates increased among adolescent women (and in two of the four countries, also among adolescent men, but by significantly less).

But it’s also important to notice that while the rise from 2011-2015 is similar across these countries, the post-2015 trend varies. The basic logic of Haidt’s thesis, however, would suggest that self-harm rates should continue increasing post-2015. After all, the number of teens with smartphones kept increasing. In the US, in 2022, 95% of teens had smartphones, up from 79% in 2016. And social media didn’t stop developing. Tiktok was introduced in 2016, after which other platforms also introduced “shorts” (short-form vertical video) to compete. If anything, social media got more addictive post-2015.

On Haidt’s view, more social media is consistent with the continued rise of self-harm hospitalizations in the US. But what about Australia? The number of phones has continued to increase (91% of teens have their own smartphone as of 2023), but self-harm rates decreased from 2017-2020 among females 15-19 and 20-24 years old.

Canada’s post 2015 trend is also inconsistent with Haidt’s thesis. The only increase appears to be among 20-24 year old women—10-14 year olds and 15-19 year olds have consistent or decreasing rates of self-harm. (The vertical line is the onset of Covid, which is what the authors here were interested in.)

Ok, so we’ve seen increasing self-harm rates in several countries, especially in the 2011-2015 period. Are self-harm rates increasing in 2011-2015 everywhere? No. Here’s data from Sweden.

There is an uptick for girls aged 10-14 and 15-19, but it occurs well after 2015. If I had to guess, I would assume it was a spike from Covid-19. Here’s data from Denmark:

There are some subtleties here. In particular, one thesis for the decrease in Denmark is that they implemented policies to restrict the supply of over-the-counter analgesics. But Haidt does not discuss this in the book. It’s buried deep in his substack, where anxious parents with children are less likely to look.

I.II. Suicide rates

Do these increasing self-harm rates lead to increasing suicides? Not in OECD countries at large:

In fact, for many countries, suicides in 2015 among 15-19 year olds is less than it was in 2000. (Caveat: for some of these countries the latest year available is ridiculous long ago. See footnote (c) in the report for specifics.)

Here are suicide rates for US, France, Japan, UK, Australia, and Canada, from 2000-2022. The US, Japan, and the UK saw an overall increase in suicide rates among 15-19 year olds, while Australia, Canada, and France saw no change or a decrease. Moreover, in the US, Japan, and the UK, suicide rates rose among all age groups, not just adolescents.

In Sweden, suicide rates have decreased from 2006 to 2020 among 15-19 year olds, but increased for other age groups (Figure is in Swedish, sorry. But if you can’t read Swedish by now what are you doing? It’s the lingua Franca of the best looking people on the planet for God’s sake):

Finally, let’s return to the US. There is an increase in suicide rates among teens in the US. The rise is considerably more drastic for males than females:

Another important piece of context is that suicide rates have been increasing for almost all age groups in the US. This is tragic, but recall that Haidt’s thesis is that social media is uniquely bad for adolescents. If suicide is increasing among all age groups, it suggests that there might be a common cause.

I.II. Summarizing the morass

Where does all the data leave us? There does seem to be a precipitous rise in self-harm rates from 2011-2015, and to a lesser extent suicide rates, in several countries. The rise started roughly at the same time, and it seems too consistent to be random noise. There is something in need of explanation here.

However, there are clearly some data which contradict his view. For one, there are countries in which self-harm rates are decreasing. Second, the rise stops in many countries after 2015, which is possibly inconsistent with Haidt’s thesis (though we wouldn’t know, because we’re not told what his thesis predicts specifically). Third, the rise in suicide is only in select countries.

Disappointingly, the book does nothing to explore these complexities. It leaves readers with the impression that wherever data exists, we see this rise. Moreover, it deliberately sweeps various complexities under the rug. It uses data for 10-14 year olds when convenient, and data for 15-19 year olds otherwise. Later in the book Haidt will justify his conclusions with experiments done on college students, i.e., 18-25 year olds. This lack of nuance and attention to detail is frustrating, to say the least.

II. Haidt’s thesis

But let’s give Haidt the benefit of the doubt about these trends. What’s his explanation?

There are developmentally sensitive times of childhood—times when our brains are uniquely primed for socializing and learning. This has a well-known evolutionary logic: unlike other animals, our brains are not fully wired up before birth because we have much more learning to do during childhood.

We spend these formative years playing and socializing in order to learn about our environments and how to productively interact with adults and other children. Haidt claims that social media has hijacked this learning process. We’ve moved our developmentally sensitive years to our phones and this, in his view, is a disaster. He calls this “the great rewiring”.

To Haidt’s credit, how teenagers are spending their time does seem to be changing:

Of course, it’s unclear whether these trends are bad and it’s especially unclear whether they’re caused by social media.

Haidt gives a variety of explanations for why social media in particular is detrimental to our mental health. He discusses social deprivation, sleep deprivation, attention fragmentation, and addiction. He claims that social media disrupts our natural process of social attunement, whereby we learn how to navigate relationships and social norms. He claims that social media disrupts our ability to learn from those we need to (teachers, community members), and instead has us copying influencers. He claims social media keeps us in “defense mode” instead of “discover mode,” making us wary of the real world and afraid to take on challenges, navigate risk, and be agentic.

These explanations all sound good if you’re already convinced that social media is causing harm. But the more skeptical you are, the weaker they sound. Is it bad to have kids learning from people on the internet rather than people in their immediate communities? You can interact with many more theoretical physicists, chess grandmasters, professional dancers, and standup comedians online than you can in person.

In other words, it’s easy to come up with any post-hoc story you like to explain the rise in self-harm rates. Explanations are easy; the point of statistics is to stress test our cherished ideas so that we don’t fool ourselves. Haidt’s case rests on whether he can show a causal connection between social media and mental health.

And here we get to the major issue I have with The Anxious Generation. The section discussing evidence of correlation and causation, which are the data needed to substantiate his points, takes up all of five pages. Five pages! The book is over 400 pages long and waxes lyrical about the spiritual degradation we sustain as a result of social media, the four foundational harms, and how we as a society can solve this urgent collection action problem. I don’t know whether to be enraged or impressed with Haidt’s chutzpah here. I would not have the nerve to write a several hundred page book calling for significant government intervention while summoning only five pages of statistical evidence.

To make matters worse, the evidence is weak! The data quality is poor, the studies are flawed, and researchers are divided (in fact, I’d go so far as to say that most researchers in this area disagree with Haidt—see below). Would you know of any of this after reading the book? No, you’d be convinced that Haidt is standing on a rock-solid mountain of evidence.

II.I. Is anxiety/depression correlated with social media use?

First up: is there a relationship between social media and poor mental health? We’re not asking about causation yet. We’re simply looking at whether social media is associated with poor mental health outcomes. (An association, if found, could be due to causation in either direction, or to some third variable which causes an increase in both.)

A note to the reader: Haidt’s argument depends on establishing a causal link between social media and mental health. If that’s your primary concern, you might consider skipping this section. But correlation is still worth examining for at least two reasons. First, a robust correlation suggests there’s something potentially worth studying causally—it acts as a “signal” in the noise. Second, correlation coefficients (and particularly r^2) can tell us how much of the variance in mental health outcomes is associated with social media use. Even if we later establish a causal link, understanding this variance is useful for questions of public policy.2

Haidt’s claim is that the size of the correlation between social media use and “internalizing disorders” (i.e., poor mental health) is roughly r=0.2 for girls. Here “r” is “Pearson’s correlation coefficient” and runs from -1 (perfect anti-correlation) to 1 (perfect correlation).

He writes the following in footnote 7 of Chapter 6:

Jean Twenge and I have found [the correlation] to be around r=0.20 when you limit the analysis to girls and social media. Orben & Przybylski (2019) said that the correlation was equivalent to r < 0.04, which truly would be negligible, but again, that was for all digital activities and all teens. When Amy Orben (2020) reviewed many other studies that were limited to social media (rather than all digital media), she found that the associations with well-being range from r = 0.10 to r = 0.15, and that was for boys and girls combined. The effects are usually larger for girls, so that puts it up above r = 0.15 for the link between social media and poor mental health for girls, which is very close to what Twenge and I have found. … So the research community is closing in on a consensus that crude measures of social media use are correlated with crude measures of anxiety and depression, for girls, at a level around or above r = 0.15.

I think this is a mostly accurate summary of a general trend in the literature. There seem to be many studies that don’t find a significant association when you look at the effects for men and women combined. But when you focus in on adolescent women in particular, there is often an association of somewhere between r=0.05 to r=0.2 (though the highest I’ve ever seen is r=0.2; it’s usually more like r=0.1 to r=0.15. I find it annoying that Haidt keeps using 0.2 as some sort of standard. ) For example, this seems to be the case for the Valkenburg et al. study, which is a study you’ll often see thrown around by people who want to discredit Haidt’s thesis. (Haidt’s discussion of this study is on page 375 here.)

There are a few things to say about this finding.

First, this is the association between social media use and reported mental health measures. This is not self-harm or suicide. I don’t know where those data are, but I’d like to see them.

Second, if there’s no association when we look at both girls and boys together, then the effect is nowhere near as severe for boys alone. This is consistent with the data we saw above. To his credit, Haidt acknowledges in the book that the effect of social media is more complicated for boys. He writes “[m]y story is more speculative than the one I told about girls in the previous chapter because we just don’t know as much about what’s happening to boys.” (pg 189). However, he rarely acknowledges this caveat when discussing his thesis in public. And his policy proposals do not take into account that the effect for boys is often non-existent.

Third, and somewhat technically, we should note that these correlation coefficients assume a linear relationship between the two variables. If, say, the response is quadratic in the covariates (eg Y=X^2), then r will underestimate the strength of the relationship. So if mental health gets worse super-linearly (i.e., more than linearly) with social media use, r will not capture the strength of that correlation.

Fourth, it’s important to recognize that there is still disagreement here. We shouldn’t think that r=0.15 is the end of the story for girls. Plenty of studies find almost no correlation (e.g., here). Further, in his substack post (written before The Anxious Generation), he notes that analysis decisions can have a huge impact on the findings:

There was one other difference that turned out to make a large difference in our results. Orben and Przybylski had not only controlled for demographic variables (such as race and parents’ educational levels, which is universally done); they also controlled for some psychological variables that are potential mediators of a relationship between social media usage and poor mental health, such as negative attitudes about school and closeness with parents. We found that controlling for these psychological variables heavily suppressed the relationship between social media use and poor mental health.

Finally, let’s assume Haidt is right (Haidt makes right!). Is r=0.15 substantial? Depends on what we’re studying. In fairness to Haidt, it’s high enough to sometimes warrant public policy changes in other domains. In one of the collaborative review docs he writes:

For example, Gotz et al. note that the correlation of calcium intake and bone mass in pre-menopausal women is r = .08, which is enough to recommend that women take calcium supplements. The correlation between childhood lead exposure and adult IQ is r = .11, which is enough to justify a national campaign to remove lead from water supplies.

He continues:

So is a correlation coefficient of .15, or .20, between a behavior and a health outcome “small potatoes”? Is it so small that it has no implications for public policy? No. In the Millennium Cohort Study, the correlation of well-being with social media use, for girls, is around .19. This is about the same as the correlation of well being with heroin use for boys (.19), and it is larger than the correlations of well-being with many known health-related behaviors for girls, including heroin use (.10), and exercise (.06).

Okay, so where are we at? I’m ready to believe there’s an association of somewhere between 0.1 and 0.2 for girls. I’m not ready to believe we have consistent correlation for boys, despite Haidt claiming so in several places. But the data are extremely messy, and most of the studies are bad. In fact, researchers in this area have written papers talking about how bad the data are. For instance, Odgers and Jensen (2020) and Orben (2020) . We’ll revisit their criticisms later.

Let’s move onto causation.

II.II. Causation part one: Dose response-studies

There are several ways to study causation for this question: dose-response studies and natural experiments.

Dose-response studies look for effects at the individual level. These ask people to change the amount of time spent on social media (the dose) and then monitor their anxiety and depression levels (response). If the participants were randomly assigned, and if the group spending less time on social media is less anxious, we conclude that social media is a cause of increased anxiety.

Here’s the entire paragraph devoted to dose-response studies in Haidt’s book:

[O]ne study randomly assigned college students to greatly reduce the use of social media platforms (or not reduce, for the control group) and then measured their depressive symptoms three weeks later. The authors reported that “the limited use group showed significant reductions in loneliness and depression over three weeks compared to the control group.” Another study randomly assigned teen girls to be exposed to selfies taken from Instagram, either in their original state or after modification by the experimenters to be extra attractive. “Results showed that exposure to manipulated Instagram photos directly led to lower body image.” Taken as a whole, the dozens of experiments that Jean Twenge, Zach Rausch, and I have collected confirm and extend the patterns found in the correlational studies: Social media use is a cause of anxiety, depression, and other ailments, not just a correlate.

- The Anxious Generation, page 159.

Let’s look at the quality of these studies. The first study is No more FOMO: Limiting social media decreases loneliness and depression by Hunt et al. They took 143 undergraduates from University of Pennsylvania and asked them to restrict the time they spent on three apps—Facebook, Instagram, and Snapchat—on their phones to 10 minutes a day. They were still allowed to use social media on their computers. The study ran for three weeks, giving students a questionnaire to report how they felt each week.

The study examined seven outcomes (self-acceptance, self-esteem, depression, anxiety, loneliness, fear of missing out, and perceived social support). They found statistically significant results for two of the conditions: loneliness and depression. In other words, for five out of the seven outcomes they examined, there was no statistically significant link between time on social media and the outcome.

So one of the major pieces of evidence Haidt summons in defense of his case is a three week study run on 143 American undergraduates, which are not of the same age as the group he’s concerned about in the first place (which is either 10-14 year olds or 15-19 year olds depending on which graph he feels like using). The students could still use social media on their computers, they were incentivized to participate with course credit, and the only way to verify they were staying off of social media on their phones was with screenshots of their battery use screen (see page 757). Finally, as Peter Gray points out, the students knew what the researchers were studying. There are no controls for demand effects (participants are usually motivated to support the hypothesis they think is being studied) or placebo effects (a belief you are doing something that will make you feel better can make you feel better).

The second study Haidt references above is Picture perfect: The direct effect of manipulated Instagram photos on body image in adolescent girls by Kleemans et al. They showed 144 girls, ages 14-18 (at least the age range is relevant for Haidt’s thesis this time) instagram-style pictures of other girls, either photoshopped or not. Then they had them rate themselves on a nine-point body satisfaction scale, with higher scores indicating higher satisfaction. The girls shown the manipulated photos scored an average of 4.57; the girls shown the original photos scored 4.94 (page 101, also quoted here3). The difference was statistically significant at the 0.05 level (the p-value was 0.021), but a statistically significant difference is not the same thing as the difference being large. And the difference here is simply not that large.

Further, while Haidt applies the conclusions of this study to social media, they could be applied to any setting in which women are seeing curated photos of other women. In other words, the design had nothing to do with the social aspects of instagram, it simply tested how teenagers reacted to images with beauty enhancements. Why not apply the study to TV, magazines, or dance performances? Maybe makeup at large should be banned.

Haidt doesn’t discuss the specifics of these studies, or their drawbacks. Instead, he’s ready to confidently conclude that the “tidal wave” of poor teen mental health is due to social media.

What about the other studies he references in the previous quote? The “dozens of experiments that Jean Twenge, Zach Rausch, and I have collected” that support his thesis? As you might expect by now, these studies are a mess. Both Aaron Brown and Stuart Ritchie have written good criticisms of their quality.

In fact, even researchers in this area have published calls for higher quality studies. In 2021, academic psychologists Candice Odgers and Michaeline Jensen published an overview of research in this area. They write

The most recent and rigorous large-scale preregistered studies report small associations between the amount of daily digital technology usage and adolescents’ well-being that do not offer a way of distinguishing cause from effect and, as estimated, are unlikely to be of clinical or practical significance.

In a 2020 review article on teenagers, screens, and social media, Amy Orben, leader of the Digital Mental Health group at Cambridge, summarizes her results as follows:

When examining the reviews, it becomes evident that the research field is dominated by cross-sectional work that is generally of a low quality standard. While research has highlighted the importance of differentiating between different types of digital technology use many studies do not consider such necessary nuances. These limitations aside, the association between digital technology use, or social media use in particular, and psychological well-being is—on average—negative but very small. Furthermore, the direction of the link between digital technology use and well-being is still unclear: effects have been found to exist in both directions and there has been little work done to rule out potential confounders. (emphasis mine)

In her conclusion she notes that “it is evident that the research field needs to refocus on improving transparency, interpreting effect sizes and changing measurement.” It’s important to note that Odgers, Jensen, and Orben all work in this area. They are incentivized to agree with Haidt—it would bring more public attention and acclaim to their work. The fact that they diverge with Haidt so strongly is a good signal that his thesis is not as watertight as presented.

Despite the poor quality of the studies, there have been meta-analyses attempting to establish whether the literature overall establishes a causal relationship between social media and mental health. The main meta-analysis was done only recently by Christopher Ferguson: Do social media experiments prove a link with mental health: A methodological and meta-analytic review. The study was published in 2025 but available in 2024. He finds that “meta-analytic evidence for causal effects was statistically no different than zero.” (Incidentally, Ferguson was one of the first to debunk the link between violent video games and aggression. What a delicious twist if he does the same for social media.)

Haidt has criticized the Ferguson study (see part I and part II) and I think some of his criticisms are correct. (Others were corrected by Ferguson, and they did not affect the results.) Haidt then proceeds to do his own meta-analysis, and claims that he does find evidence of a causal effect.

But Haidt himself makes several mistakes. Instead of listing them here, I’ll point to Matthew Jané’s interrogation of his methods. Jané is a statistician whose expertise is in meta-analyses. Quoting from his conclusion,

The re-analysis of Ferguson (2024)’s meta-analysis by R&H [Rauch and Haidt] does not have adequate statistical rigor to build a “case for causality”. Post-hoc subgroup analyses conducted by R&H did not use a principled statistical model, they did not report any variability in their estimates, the estimates themselves were sub-optimal (unweighted averaging), and they misinterpreted the point-estimates that they calculated. They did not do a proper comparison of their point-estimates and instead they treated them as fixed quantities and simply claimed that these average effect sizes are different without consideration of variability in their estimation procedure.

In sum, I think the verdict on dose-response studies is still out. The existing studies are not of sufficient quality to conclude anything about the effect of social media on mental health, and Haidt is being irresponsible by claiming otherwise.

II.III. Causation part two: Natural experiments

Dose-response studies are not the only way to study causality.

As I mentioned, dose-response studies assume individual level effects. If Matilda spends more time on social media, then Matilda will be more depressed, regardless of anyone else’s behavior. But social media might manifest itself as a network level effect. If all of Matilda’s friends use social media, she may still be affected by it even after quitting herself. She risks being excluded from social activities and conversations, and might not be able to replace her social media time with in-person interactions.

Haidt says the same on page 160-161:

When [social media] was carried into schools in the early 2010s, on smartphones in students’ pockets, it quickly changed the culture for everyone. Students talked to each other less between classes, at recess, and at lunch, because they began to spend much of that time checking their phones, often getting caught up in microdramas throughout the day. This meant that they made eye contact less frequently, laughed together less, and lost practice making conversation. Social media therefore harmed the social lives even of students who stayed away from it. These group-level effects may be much larger than the individual-level effects, and they are likely to suppress the true size of the individual-level effects.

In fairness to Haidt, I think this is correct. That is, if social media does harm mental health, then I would imagine the effects manifest themselves most strongly at the group level. This could be why we’re seeing such weak results in the individual-level studies we looked at previously.

So how do we study network effects? We want to ask: Is a community better off with less social media? A common way to answer this question is with natural experiments: did people in a community (a high-school, say) become more depressed when social media was rolled out? Ideally we compare two high schools, one which got social media after the other, in order to control for other things happening in the culture at the time. If the school whose students adopted social media become more depressed and the other school didn’t, then this is evidence of causation.

The gold standard natural experiment for this question is the Facebook rollout study by Braghieri et al. The authors took advantage of the fact that Facebook was available at different colleges at different times from 2004-2006. I think this is the strongest study in Haidt’s favor. The authors are careful, the analysis looks well done, and they do find that Facebook had a detrimental effect on the mental health of teens.

Our index of poor mental health, which aggregates all the relevant mental health variables in the NCHA survey, increased by 0.085 standard deviation units as a result of the introduction of Facebook. As a point of comparison, this magnitude is around 22% of the effect of losing one’s job on mental health, as reported in a meta-analysis by Paul and Moser (2009). … The effect of the introduction of Facebook on our index of poor mental health is equivalent to a two-percentage-point increase in the share of students suffering from depression according to the PHQ-9 over a baseline of 25%.

- Braghieri, Levy, Makarin (2022), page 3.

So the best evidence for network-level effects that Haidt could find shows a two-percentage point increase over baseline depression levels. This is not nothing, and hopefully we can get our hands on more studies like this. But combined with the sketchy results of the dose-response studies, is this enough to go-to-green that social media is destroying a generation?

(Stuart Ritchie claims that after adjusting for multiple comparisons — that is, testing multiple things — that the effect disappears. This would be damning for the result if true, but I have been unable to corroborate his findings and he doesn’t cite anything. So I’m going to assume he’s wrong for now.)

Bizarrely, to bolster his point about network effects, Haidt reaches for studies which purport to show that the rollout of the internet had disastrous consequences for the mental health of girls. These studies have problems. But even if they didn’t, this claim doesn’t support Haidt’s argument. Is it the internet at large that’s the problem, or social media? If the former, why is Haidt going on a crusade against social media in particular? If the latter, then why cite studies of the former? Perhaps it’s some secret combination of the two, but we’ll never know because Haidt never tells us.

III. Conclusions

Candace Odgers wrote a review of The Anxious Generation, published in Nature. She accurately summarizes my feelings:

Two things need to be said after reading The Anxious Generation. First, this book is going to sell a lot of copies, because Jonathan Haidt is telling a scary story about children’s development that many parents are primed to believe. Second, the book’s repeated suggestion that digital technologies are rewiring our children’s brains and causing an epidemic of mental illness is not supported by science. Worse, the bold proposal that social media is to blame might distract us from effectively responding to the real causes of the current mental-health crisis in young people.

My goal here is not to prove that social media is net positive, or has no negative effects. I’m open to the possibility that social media has overall adverse effects for some age-groups, and a well-powered, rigorous study could come out tomorrow that would convince me. Some commentators even make some (semi-) compelling cases with arguments distinct from Haidts’ (e.g., Derek Thompson and Richard Hanania, though they also make some mistakes and rely on some bad data).

My goal is instead to argue that the claims in The Anxious Generation are misleading. Even if Haidt turns out to be right, it does not excuse the sloppy way in which the results are presented. Sometimes he draws a fine-grained distinction between social media and the rest of the internet, and sometimes he doesn’t. Sometimes youtube, and dating apps, and blogs, are considered social media, and sometimes they aren’t. He discusses different age groups at different times, and uses studies on college students to justify conclusions about 10 year olds. He reports data and studies inconsistently and selectively, leaving readers with more confidence in his assertions than is warranted. We should demand better of public intellectuals writing books that will reach millions of parents and likely influence public policy around the world.

I’m aware that I’m critiquing this book as a non-parent. I don’t doubt that many parents feel they’re in a bind with social media, sensing that it’s making their teenagers' lives worse but feeling powerless to do anything about it, lest they damage their relationship with their child. And I don’t doubt that Haidt’s book has an emotional appeal because of this dynamic.

My only advice is to not take Haidt’s conclusions at face value. Even if he is correct, we’re in the world of small, average, effects. Regardless of how many meta-analyses we run, we can never make claims about how any particular child is going to react to social media. You are far more of an expert on your child’s mental health than any social psychologist, simply by virtue of being their parent and spending time with them. And you can talk to them. Solving the problem together is far better than handing down verdicts rubber stamped by ivory tower academics.

Thanks to Sam, Kasra, Rich, and Vaden for feedback.