Sort of against "if you can't program it, you haven't understood it"

And definitely against its converse

I.

On page 154 of The Beginning of Infinity, David Deutsch makes the following remark when discussing theories of consciousness:

I have settled on a simple test for judging claims, including Dennett’s, to have explained the nature of consciousness (or any other computational task): if you can’t program it, you haven’t understood it.

Deutsch is, understandably, annoyed by all the vague hand-waving in this field and proposes this test as a criterion for sorting the good theories from the bad. If your theory isn’t precise enough to be implemented, do you really understand what you’re talking about?

As is the case with many popular thinkers, Deutsch’s ideas have come to be widely adopted and applied. Perhaps more widely than originally intended. The line if you can’t program it, you haven’t understood it is now being incorrectly applied to dismiss theories well outside the realm of consciousness research, especially regarding AI.

(In defense of people applying it widely, Deutsch does preface his criterion with “or any other computational task.” And if you’re familiar with Deutsch’s work, you’ll know that his worldview is built heavily around computation and computational universality. He’s the father of quantum computation, after all. Many tasks that an average person wouldn’t consider “computational” would qualify as such for Deutsch, so arguably his statement is meant to apply to most theories across most domains.)

Before we get to AI in particular, let’s just evaluate the statement if you can’t program it, you haven’t understood it at face value. Clearly, it can’t be literally true. For one, it would mean that someone who can’t program can’t understand anything. Second, there are computational tasks we understand perfectly well that provably can’t be programmed. Turing showed that no program can take an arbitrary program and decide whether it will ever halt (the halting problem). So either nobody understands the halting problem, or there are things we can understand but not program.

Ok fine, but this example is obviously unfair to Deutsch. He’s not asking you to sit down and write everything you know in C++. The spirit of his remark is something like:

If you claim to understand something, you should be able to specify it precisely enough that it could be programmed by someone, given unlimited resources, unless you can say exactly why it can’t be (e.g., you’ve proved it uncomputable).

But I think even this modified form of the idea has problems. There are lots of things we can’t write down a set of complete instructions for that I think we understand. Things like the horror of war, or the fear of standing on a cliff edge, or the feeling of falling in love, or the beauty of a sunset. Basically anything that emerges at a high enough level of abstraction.

Do we understand these things completely? No. Can we unerringly predict who we’ll fall in love with, or whether a marriage will work, or whether a friend will cry when they see a piece of art? Certainly not. But there are levels to understanding, whereas Deutsch’s criterion implies that it’s binary. There’s such a thing as wisdom in the realm of love, for instance, and ideally we get more of it as we get older. I don’t know what to call that if not partial understanding. (Maybe skeptics call it resignation.)

Being binary is a general feature of Deutschian epistemology. In this worldview, things are either computable or not, animals are either universal explainers or not, and theories are either good explanations or not. This clean categorization is part of what first drew me to Deutsch, but I admit that I’m becoming more skeptical of it with time. It holds for some things, like being computable or not, but applying it to scientific theories and knowledge creates more problems than it solves.

(As it happens, this is one place where Deutsch departs from Popper. Popper spent a long time emphasizing corroboration and degrees of falsifiability in scientific theories, I think for good reason.)

Anyway, this has all been self-indulgent warmup. Overall, I think Deutsch’s criterion is often quite useful, especially when people are making substantial claims about the fundamental nature of reality. But I think it’s overused in the AI conversation in particular, and I’ve brought you all here today to bear witness to my grievances.

II.

Deutschians are typically very skeptical of AI doom and the idea that we’re on the path to building superintelligence. We have this in common, which puts me in a somewhat strange position as I happen to think that most of their arguments are pretty bad. They are right for the wrong reasons.

One of those wrong reasons is applying if you can’t program it, you haven’t understood it to AI. The argument is roughly: Nobody has a detailed theory of creativity or general intelligence, so we don’t truly understand how these processes work. Therefore, we’re not about to give rise to a technology that can do these things.

To unfairly pick on Brett Hall, perhaps the world’s best expositor of Deutsch’s ideas, here is an example of him using this argument to criticize the idea that AGI is imminent. Starting around 3:10, he says:

David Deutsch has a crucial principle that unites epistemology and computer science in a fundamental way, and which undermines the entire “imminent” argument (...). And that principle is “if you can’t program it, you haven’t understood it”.

This is a profound and exciting claim. In just nine words, it captures what is wrong with thinking that ever faster and better hardware is the key to creating just the right software.

He goes on:

We humans can program computers to do all sorts of amazing things. We can simulate galaxies of stars colliding because we understand many-body gravity interactions. We can program computers to replicate the structure of buildings or how populations of rabbits might grow. We understand those things first and so construct an algorithm and write a program, in that order. (emph mine)

There are several things wrong with Hall’s arguments here. The biggest is that he’s inverting the logic of Deutsch’s original criterion. Even if you stand by if you can’t program it, you haven’t understood it, you can’t use it to support Hall’s conclusions here.

Hall’s argument is: We don’t understand AGI so we can’t program it. More generally, written as an if-then statement: If you don’t understand X, then you can’t program X. But the original claim is the converse of that: If you can’t program X, then you don’t understand X.

These are not the same! To make this obvious, consider a driving test. If you can’t drive, you haven’t passed your driving test is true, since the test checks whether you can drive. But if you haven’t passed your driving test, you can’t drive is false. Everyone who shows up to take the test can already drive. That’s why they’re there.

(There is an irony to this error, since it was precisely this mistake which motivated Popper’s criticism of verificationism in science. Imagine you have a theory T which makes a prediction P, which we write as T → P. Verificationism wants to observe P and conclude T. In symbols, that’s T → P, P ∴ T (where ∴ means “therefore”). This is called affirming the consequent, and it’s logically invalid. Popper instead said you should try to falsify your theory by looking for ¬P (not P), since T → P, ¬P ∴ ¬T is valid. That pattern is called modus tollens.)

Ok fine, so Hall’s argument doesn’t follow logically from Deutsch’s criterion. But this doesn’t mean that it’s wrong ...

III.

... But it is in fact wrong. To stop picking on Hall for a moment, there are many other people making the same argument. In fact, here’s Deutsch himself arguing that a philosophical theory of creativity and intelligence must precede our making it:

Expecting to create an AGI without first understanding in detail how it works is like expecting skyscrapers to learn to fly if we build them tall enough.

And here’s Moritz Wallawitsch fleshing out that idea:

Expecting to create a program that is creative without first understanding in detail how creativity works is like expecting skyscrapers to learn to fly if we build them tall enough. The inventors of the steam engine Savery and Newcomen, while relying on experimentation and mechanical intuition, had rudimentary and intuitive knowledge of thermodynamics. Similarly, the Wright brothers understood aerodynamics before inventing airplanes.

These criticisms miss the mark in several ways. First, the “AI is imminent” crowd do in fact think they have an explanation of what AGI and creativity are. Roughly speaking, they think it’s an emergent property of systems undergoing lots of optimization pressure. For humans, that optimization pressure was natural selection and for LLMs it is minimizing cross-entropy on huge datasets.

You may disagree with this view (I do disagree with this view), but you can’t just ignore it and assume they have no explanation. And you definitely can’t try and prove that they don’t have an explanation by asking them to program it, because they are programming it.

Second, the idea that you need to understand what you’re building before you build it is just not congruent with the history of discovery. Humans accidentally stumble onto useful things all the time without knowing how they work. Some examples:

• Aspirin was mass-produced starting in 1899 without anyone knowing the underlying mechanism. Before that, people chewed willow bark for pain for thousands of years (it contains salicin, which the body converts into salicylic acid, aspirin’s parent compound). The mechanism (COX inhibition) was only worked out in 1971.

• General anesthesia has been used since the 1840s. Even now, the mechanism isn’t fully understood.

• The smallpox vaccine came decades before the germ theory of disease was developed.

• We made bread, beer, and wine for thousands of years before we understood the fermentation process (we didn’t know microorganisms were doing the work until the 1800s).

• Selective breeding was popular long before we understood genetics. Maize was bred from wild grass about 9,000 years ago.

Before we discovered any of these things, it would be easy to say: “Expecting to create pain meds without developing a detailed understanding of pain receptors is like expecting skyscrapers to learn to fly if we build them tall enough.” Or, “Expecting to reshape species over generations is like expecting skyscrapers to learn to fly if we build them tall enough.” Alas, the history of discovery is messy and doesn’t always obey our favorite philosophical principles.

There are even more pertinent examples of developing dangerous technologies (or, at least, technologies with possibly substantive downsides) without understanding the mechanisms.

• TNT was first made in 1863 but it wasn’t noticed that it was an explosive until 1891.

• Chinese alchemists developed gunpowder centuries before anyone understood the chemistry of combustion.

• It’s believed that fission was produced in 1934 by Enrico Fermi’s group when they were bombarding uranium with neutrons. But they didn’t realize what they were looking at until 1938.

• Thomas Midgley developed CFCs in the 1920s as safe refrigerants. It wasn’t understood that they had a negative effect on the ozone layer until 1974.

Again, there are many good reasons to be skeptical of AI doom. Unfortunately, one actually has to engage with the arguments in order to refute it, not throw around nine-word slogans. As skeptics, let us be charitable, let us be precise, and let us, at least once, try to pass an intellectual Turing test.