6: A Few Questions on AGI

Popperian algorithms, evolvability, explanations, and more.

The fun of research is as much in the questions as the answers, so I figured I’d share some of my latest ones!

  1. What are the limits of biological evolution?

  2. What makes explanatory knowledge special?

  3. How can Popperian epistemology improve narrow AI algorithms?

  4. What are the different kinds of conflicts between ideas?

  5. Why is the brain a network of neurons?

  6. How does the low-level behavior of neurons give rise to high-level information processing?

1. What are the limits of biological evolution?

Could evolution ever produce a wheel-and-axle? Perhaps such inventions are “all or nothing”, and are therefore impossible to create in a sequence of tiny, trivial genetic mutations. It’s a counterintuitive fact that evolution can produce stupendous complexity (if you doubt this, then try 3-D printing a hippo…), but it cannot produce even a simple thing if it is too different from what already exists. Evolution is a bit like a mountain climber that can go high as the sky, but never jump a gap. (By the way, I suspect today’s neural networks have similar limitations.)

This question about limits is interesting because there is something special about humans - something we can do that evolution can’t. But what? While we know something of the answer, our knowledge is vague. Creating artificial general intelligence may depend on a more precise understanding of the boundary between the capabilities of human minds and biological evolution.

One could call this the question of evolvability - what can be evolved and what can’t be? Incidentally, this question - about what is possible and what is impossible - has the same form as statements in constructor theory, which is about what physical transformations are possible and impossible, and why. Perhaps the limits of evolution can be expressed in constructor theory, or illuminated by ideas from the constructor theories of information, life, and thermodynamics.

2. What makes explanatory knowledge special?

In his 2012 article, Creative Blocks, David Deutsch argues that “the ability to create new explanations is the unique, morally and intellectually significant functionality of people (humans and AGIs)…” He elaborates on this in The Beginning of Infinity and his TED talk on explanation, but we have a long way to go before our understanding is sufficient to create a computer program capable of explaining anything.

In a recent post, I asked:

Explanations are about what is objectively true rather than only what is useful. What are the consequences of this? What makes this sort of knowledge more powerful, and indeed more useful, than other kinds? What is different about its structure? What mechanisms of variation and selection are required to create this sort of knowledge? What is so difficult about creating it? (After all, it’s an extremely recent innovation in the history of life on earth.)

In my last post, True vs. Useful, I tried to explore one feature of explanations that makes them especially powerful:

In the end, the search for truth entails the pursuit of logical consistency among all our ideas, and thus takes advantage of all our knowledge - not just a single, fixed idea. It subjects our ideas to a powerful form of selection - logical contradiction - not found in biological or machine learning systems. Most importantly, it provides a combinatorial explosion of opportunities for conflict - and thus for progress.

On a different note, what makes explanations so hard to create? One idea, as Deutsch argues, is that good explanations are “hard to vary,” meaning that most modifications not only make them worse, but completely non-functional - unable to explain anything at all. This makes them hard to reach in a way much like “all or nothing” ideas like the wheel-and-axle.

To visualize this property, imagine a vast cube representing the space of all ideas, where the best - the most true and useful - ideas are bright points of light while the worst ideas are invisible. In this space, good explanations are solitary, bright points. They are rare, sparse, and disconnected from other bright regions. On the other hand, useful rules of thumb exist as fuzzier clouds of points, for similar rules of thumb are about as good as one another, and so the nearest points to a given rule of thumb are about equally bright. Similarly, genetic knowledge forms a fuzzy tree without any gaps - for any two points in the tree, you can get from one to the other by following a path of bright points.

Now imagine you’re trying to navigate this space to improve your ideas, but can only see a short distance. If you find yourself in a fuzzy cloud or tree, you can look around to nearby points to see if any are better and brighter, and move to it. By contrast, finding a good explanation is far more difficult, for it is hidden in a vast space, and you might pass within a short distance of it without ever seeing it. Stumbling and looking around isn’t enough. You need something more like high-powered telescopes and ultra-accurate, long-distance teleportation in the space of ideas. How do we do that?

3. How can Popperian epistemology improve narrow AI algorithms?

In his 2012 article, Creative Blocks, David Deutsch argues Popper’s work on epistemology is key to building artificial general intelligence. I think it may also inspire unique advances in narrow artificial intelligence algorithms (which, despite their lack of generality, can still be tremendously useful). After all, Popper’s work applies to the creation of knowledge in all its forms, from biological evolution to human minds - and narrow AI.

Also, the whole point of AGI is to write a program that is a mind, so the earlier one can apply and test one’s theoretical ideas (by programming them), the better. Such tests are bound to uncover all manner of subtle theoretical issues that would otherwise go unnoticed. That’s how programming usually goes - getting one’s ideas to work in practice is harder than anticipated, and leads to a far better understanding of things.

One idea is to apply the conclusions of True vs. Useful, and focus on solving constraints rather than maximizing performance. For a visual metaphor, it’s like trying to get puzzle pieces to fit together rather than walking to the top of a hill. While this approach has a long history (e.g. logic programming in Prolog), logic-based approaches to artificial intelligence have been mostly unsuccessful. They are brittle and full of precise statements, while human knowledge is flexible and full of fuzzy statements. So, there are unsolved problems here, and perhaps deep learning and Popperian ideas can help address them.

For one thing, I think a common mistake in the history (and present) of AGI research is to take a particular cognitive tool like logic, language, or analogy and suppose it is the core of intelligence. As Popper explained, variation and selection are at the core, and other things just provide specific (and often tremendously useful) mechanisms of variation and selection. Perhaps taking this seriously will help solve the problem of how to use logic in artificial intelligence - both narrow and general.

4. What are the different kinds of conflicts between ideas?

A key part of understanding minds is understanding how ideas interact within them. How do these interactions lead to the variation and selection required for the evolution of knowledge? How are they combined and altered to form new ideas? How do they exert selection pressure on each other? What kinds of interactions spark the search for new ideas?

As I argue in True vs. Useful, logical contradiction offers one example of how ideas can interact, but a subtler example might be when you are surprised by something. In this case, it’s unlikely there’s any explicit logical contradiction at work, but there is still a conflict of ideas. If you are surprised upon entering an elevator containing three goats and a glowing block of uranium, you are experiencing a conflict between what you expected to see and what you are actually seeing.

Something similar must be happening when you find something interesting. Here, the conflict is even more subtle, though, and I don’t quite understand it. Presumably an idea appears interesting if it seems both novel and relevant to problems that one cares about. Given the vagueness of such a statement, it can no doubt be improved upon by trying to program it, as I mentioned earlier.

At any rate, the interactions between ideas are fundamentally important, conflict is one key example, and it exists in many different forms which have evolved for different purposes, like finding things dangerous, desirable, interesting, and surprising.

5. Why is the brain a network of neurons?

There are many ways to build a computer, and the only fundamental requirement is that it be Turing-complete. While brains and modern processors both satisfy this requirement, one does it with a network of neurons and the other with a von Neumann architecture.

Why the difference? Presumably because brains had to be evolvable while modern computers could be designed (see question #1 above). A network of neurons can start small and grow larger in the course of evolution and be useful at each stage. In contrast, modern computers are like the wheel-and-axle. They’re all-or-nothing. If one part of the system breaks, or has yet to be created, then it is useless.

Setting aside the question of evolvability, though, should minds be made of networks of neurons (or simulations of them)? While the way a computer is built doesn’t affect what it can do in principle, it does affect what it can do in practice. After all, the integrated circuits in modern computers are millions of times faster than their vacuum tube predecessors. Moreover, there are different algorithms for doing the same thing, and they can be wildly different in their speed and memory usage. Perhaps the network structure of the brain indicates that minds depend for their efficiency on concurrent, distributed, networked computation.

For example, consider how efficiently the brain can search its memory in the course of a conversation. I’ve sometimes wondered how it was that I recalled a perfectly-apt anecdote despite having not thought of it for years. Evidently, it was stored in such a way that, under the right circumstances, it could quickly and easily be found and shared. That is not a trivial feat, given the vast collection of memories in a mind. For instance, it would be far too slow to go through all one’s memories one by one. By the time you’d found a good story, the conversation would be over! The network structure of the brain seems to handle the problem with ease, though. The general picture (as in deep learning) is that a situation “activates” some neurons, which in turn activates others which are connected, and this cascade of activity can eventually activate a region of the brain associated with some long-dormant anecdote that’s perfectly relevant to the conversation one is having.

So, if that’s one example of the efficiency and practical value of network-based computation, what are others?

6. How does the low-level behavior of neurons give rise to high-level information processing?

Human brains must be Turing-complete (after all, they came up with the idea of Turing-completeness!) but how does one build a universal Turing machine from neurons and their connections? This is an active area of research (here’s one potential explanation).

More generally, for any given computation, how can it be expressed in terms of the behavior of neurons? The same question exists for modern computers, too, but instead of expressing things in terms of neurons, one uses the low-level instructions which a computer processor offers. Historically, engineers hand-coded these low-level instructions, then developed a slightly higher-level language to make things easier. They could write code in this language, and it would be translated, or compiled, into the relevant low-level instructions. Later, other languages were built on top of that language. This process has continued, and now modern programmers can express high-level ideas easily and then compile them into the instructions which a processor can understand and execute. Perhaps a similar process happened historically with brains, and can be used in any network-based AGI computer we wish to build.

5: True vs. Useful

How logical consistency causes a (good) combinatorial explosion

Consider two contradictory statements like, “all swans are white” and “some swans are black.” They can’t both be true. One or both must be false. Surprisingly, the discovery of such a contradiction in our ideas is cause for celebration, not despair. Why? Because it offers an essential guide to progress. Without it, we would not know which of our ideas to improve, nor how. With it, we know what’s wrong and how to fix it: resolve the contradiction.

Logical consistency is therefore a powerful constraint on our ideas. It means that any idea can in principle contradict another, sparking the search for an improvement that’s free of the contradiction. This yields a combinatorial explosion, for if any pair of, say, 10,000 ideas can conflict, that yields (roughly) 10,000 × 10,000 = 100,000,000 potential opportunities to spark progress. This also means any one idea is potentially constrained by 10,000 others. In practice, we cannot compare every pair of our ideas to check their consistency, and doing so would be a waste of time even if we could. Most pairs of ideas are irrelevant to each other. Nevertheless, each of our ideas is tremendously constrained by other, related ideas (and we can search among all our ideas to find which are relevant). To borrow the language of Darwinian evolution, the need for logical consistency subjects each of our ideas to tremendous selection pressure from other ideas.

And, knowledge-creation is all about variation and selection. In the biosphere, genes are subjected to random variation and to natural selection (e.g. predation, mating, starvation). In human minds, ideas are subjected to intentional variation and many forms of selection, usually involving some form of conflict with other ideas. Logical contradiction is just one important example. It’s these sorts of differences in the mechanisms of variation and selection which are central to explaining why human minds are vastly more capable than blind evolution at creating knowledge.

It also helps explain why human minds are qualitatively superior to today’s narrow AI algorithms, which are useless outside the narrow domain they’ve been trained for. A human idea is subject to selection by many other ideas within the mind, which are themselves subject to variation and selection (and therefore improvement). In a machine learning system, a model is subjected to only one form of selection: how well does it perform on the given task? In other words, how useful is it? First of all, this mechanism of selection is not itself open to improvement from within the system. It is subject to no variation and selection - except from without, by human minds. Secondly, even supposing different “ideas” in a model could be made to constrain one another, they are not allowed to. There is only a single, fixed form of selection: usefulness.

In a human mind, this would be like changing your ideas only in response to external reward and punishment, as David Deutsch explains in his essay, Beyond Reward and Punishment. Such an approach makes very poor use of the available knowledge in a system, quite like the way a totalitarian state suppresses all ideas but the dictator’s.

More generally, though, there is a fundamental difference between seeking ideas which are true versus seeking those which are useful. While two contradictory ideas cannot both be true, they may both be useful. This is the case for quantum theory and general relativity, both of which are famously useful and mutually incompatible. At present, many seek a unifying theory that will supersede them both. Why? Because of some practical problem? No. Their incompatibility presents a dramatic logical problem, and therefore an opportunity for improvement - one that we would be utterly blind to if we did not seek truth and thus logical consistency. Indeed, if we cared only about usefulness, the irony is that we’d be denied our most useful theories, for many were sought in response to theoretical problems rather than practical ones.

Incidentally, this is why it is a mistake to pursue only things that are known to have good consequences - altruistic or otherwise. After all, that would put an end to all research, the results of which are - by definition - unknown, and therefore the consequences of which are unknown.

In the end, the search for truth entails the pursuit of logical consistency among all our ideas, and thus takes advantage of all our knowledge - not just a single, fixed idea. It subjects our ideas to a powerful form of selection - logical contradiction - not found in biological or machine learning systems. Most importantly, it provides a combinatorial explosion of opportunities for conflict - and thus for progress.

4: Thinking - The Universal Superpower

The prime requirement for progress is creativity, which is available to everyone.

Ironically, I discovered the power and joy of thinking while on a meditation retreat, where one’s usually meant to think less, not more. But, since meditation only took up half the day, and the rules said not to read, write, talk, or use electronics, the only available entertainments were those provided by grey matter alone - cogitation, contemplation, reflection, speculation, and even cerebration. So, over the course of ten days, and in chunks of one to three hours, I must’ve racked up nearly a hundred hours of focused thinking.

If that seems odd to you, then you’re in good company! It seemed odd to me too. While I’d often focused on particular tasks for extended periods, I’d never just lied in bed and thought for hours at a time. Now, I was doing it constantly, and in isolation. While I’m told some people are distressed or even destabilized when alone with their own thoughts like this, I found it delightful and exciting.

Wait. It was exciting? Shouldn’t it have been supremely boring?

Just the opposite. By setting aside all my normal responsibilities, pursuits, and entertainments, I’d created the space for an entirely different activity: open-ended exploration. That’s right - despite my sedentary state and the scarcity of Arctic sled dogs, I was exploring. Not physical environments of course, but mental, virtual ones. After all, as the physicist David Deutsch points out in The Fabric of Reality, “All reasoning, all thinking and all external experience are forms of virtual reality.”

At the close of each hour-long meditation, I’d return with eagerness to my room and those bustling, dynamic, abstract worlds. When I left them, called by the next meditation bell, it was always with some reluctance. Thinking was addictive. Compelling. Engaging. Unpredictable. The sudden appearance of an unforeseen thought might set me on a new and intriguing path to who-knows-where for who-knows-how-long. Confronted by some obstacle, I might patiently try every available tool, technique, and route until I’d overcome it. I had time.

I also had opportunity. Novel ideas are not accessible only to some privileged class of intellectuals who’ve mastered all prior thought on some subject. Every mind has the capacity to create knowledge - new knowledge. We could not learn to read, speak, or walk otherwise. Despite appearances, there is no fundamental difference between learning to say “mama” and “papa” and discovering fire, photons, and fusion. All such discoveries, however trivial or significant, are made by trial and error.

And they are ready for the taking. Our infinite ignorance dwarfs our knowledge, and one can come face to face with the unknown by simply asking “why” or “how” a few times (and rejecting the many inadequate answers on offer). Isaac Newton was acutely aware of our ignorance. Perhaps this contributed to his great scientific success, for if one feels that vast riches of understanding are available, one seeks them. He said,

I do not know what I may appear to the world; but to myself I seem to have been only like a boy playing on the seashore, and diverting myself in now and then finding a smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all undiscovered before me.

In a way, none of this is surprising. It’s obvious there are things we don’t know, and that we can try to figure them out by thinking. Though simple and true, such a statement fails to capture the full scope of our situation. We know almost nothing. Ignorance, and therefore opportunities for every sort of improvement, are everywhere. The content of future discoveries, and how they will be made, are known to no one. The prime requirement for progress is creativity, which is available to everyone.

We are just beginning.

3: Minds on my Mind

How I came to be interested in studying minds

Minds are the most complex, interesting, and important things in the universe. They offer both scientific mysteries and practical, existential problems. For example, how is it that we recall just the right information when we need it, bypassing thousands of unrelated things in the process? And how do we avoid the damage and destruction of minds by aging, disease, and accident?

These and other questions about minds first began to develop in my mind in high school, while (nerdily enough) I was attending a math and physics competition held at a local university. Between events, I browsed the campus bookstore, wide-eyed at my first exposure to technical genres like popular science.

The book that won my pocket money that day was V. S. Ramachandran’s A Brief Tour of Human Consciousness, which had one idea at its core: damaged brains are a window into healthy ones. Their weird errors, like seeing numbers as having colors, offer an opportunity to study mechanisms which are also silently at work in healthy brains. Steven Pinker, come to think of it, also took a similar approach when he studied the grammatical errors of children as a window into the mind. He noted that, when children incorrectly conjugate irregular verbs by saying things like, “I heared a sound” instead of “I heard a sound”, they had to be generating new words according to some general rule - not just recalling things they’d heard before. After all, “heared” wasn’t a word they could’ve picked up from adults, since adults always say “heard”.

At any rate, brain damage and irregular verbs aren’t the only games in town when it comes to studying the mind. The philosophy of knowledge - epistemology - is another. In his 2012 article, Creative Blocks, David Deutsch argues that the work of the philosopher Karl Popper is indispensable in understanding minds well enough to create them, for “What is needed is nothing less than a breakthrough in philosophy, a new epistemological theory that explains how brains create explanatory knowledge.”

Popper viewed knowledge-creation as an evolutionary process consisting of variation and selection - conjecture and criticism. Deutsch echoes this point and adds, “the ability to create new explanations is the unique, morally and intellectually significant functionality of people (humans and AGIs)…” So, while knowledge-creation in general is always about variation and selection, it matters what is being varied and selected, and what mechanisms of variation and selection are at work. For instance, genes are changed only by undirected, random mutations while human ideas are often varied on purpose by recombining existing ideas. Though these are both forms of variation, their speed, efficiency, and range of practical application could not be more different.

Within minds, too, there is another important distinction - this time between explanations and other kinds of ideas. Explanations are about what is objectively true rather than only what is useful. What are the consequences of this? What makes this sort of knowledge more powerful, and indeed more useful, than other kinds? What is different about its structure? What mechanisms of variation and selection are required to create this sort of knowledge? What is so difficult about creating it? (After all, it’s an extremely recent innovation in the history of life on earth.)

These open questions are fascinating in their own right, for they are about knowledge and knowledge-creation, which are fundamental. They are essential to understanding biological evolution, human minds (and minds in general), political systems, markets, and much else besides. Although these domains all have their own unique problems and controversies, they also share a unifying logic. They all require conjecture and criticism - the creation of new ideas and the elimination of bad ones. In the course of jumping rapidly from field to field while applying Popper’s ideas, I’ve often thought of epistemology as the intellectual equivalent of a wormhole - it lets you jump across vast stretches of the universe at speeds that should be impossible. One moment, you may be thinking about error-correction in political systems, and in the next, you may be thinking about biological selection pressures, software testing, or scientific peer review.

So, I guess what I’m saying is… epistemology and artificial general intelligence would be worth studying even if they didn’t hold the promise of everlasting life in the form of highly-reliable, backed-up digital minds.

2: The Buffet and the Supermarket

A surprising path to depression, and my escape from it.

Inspired by a post from Derek Sivers, I once decided to be much more critical of what I spent time doing. He argues that people too often fill their time with things that are only mildly exciting, leaving less time for the really interesting things - the ones that make you say “Hell yeah!” As I applied his rule - to say “no” to things which weren’t a “Hell yeah!” - I found myself dismissing more and more. I started saying “no” to most articles I came across and project ideas I had. Eventually, I found myself saying “no” to everything. Ironically, my attempt at having more fun led to having no fun, and spending all day in bed.

Just as there is no way of speaking such that you cannot be misunderstood, there is no way of offering advice such that it cannot potentially hurt those who take it. A missing or misunderstood idea could spell trouble. More generally, there is no attempt at improvement which is guaranteed to succeed, whether in personal life, public policy, or anything else. Subtle-but-crucial misconceptions may always be lurking beneath the surface. So, we must always be on the lookout for mistakes.

In this case, I’d made two.

First, I’d been dismissing anything that wasn’t immediately super-interesting, based on a snap judgment. But, it takes creativity (and therefore a little time and effort) to understand or imagine why something might be interesting. It’s not always immediately obvious. So, I was effectively saying, “I’m going to ignore any idea that doesn’t seem ultra-valuable within five seconds of first hearing it.” That is a recipe for ignoring many worthwhile ideas.

Second, I gave up exploring new options, since they all met with rejection anyway. But, if you dismiss ideas without creating new ones, you’ll soon run out and be left empty-handed and bored to bits.

So, to correct these two mistakes and escape my depression-by-dismissiveness, I first needed to take ideas much more seriously when I came across them. I needed to put some real creativity into understanding what might be interesting about them. Second, I needed to actively seek out and create new ideas - new opportunities for exciting things to do.

In culinary terms, my mistake was treating life like a buffet rather than a supermarket.

At a buffet, a limited number of dishes are on offer, and a customer can only accept or reject them. They are at the mercy of the cook. If a buffet-goer dismisses every dish as unappetizing, they go hungry. By trying to apply the “Hell yeah or No” rule, I’d dismissed every dish at the buffet, and was left starving for options.

A supermarket is entirely different. It mainly contains ingredients, not dishes. From that vast collection, a customer can create an infinite variety of dishes. Only an insane shopper would expect to enjoy an ingredient by eating it raw. Imagine someone complaining, “I bought this flour and ate a bowl of it. It was terrible!” Come on, you’re supposed to make things with it! In my case, I should’ve been looking at every idea I came across as an ingredient rather than a dish - an opportunity to apply my creativity rather than something to accept or reject as-is. If one goes hungry in a supermarket, there is no chef to blame - just a failure of imagination.

So, a buffet only allows selection from existing options while a supermarket allows for the creation of new options.

Though my experience may seem like an isolated incident, the buffet view of reality dominates many areas of life. It is present whenever you are told to choose a class, course, or career rather than create one. To choose a friend or romantic partner rather than create a relationship. On this static view of the world, there are no new subjects to study, no new jobs, no new kinds of relationships - nothing new at all. If all the existing options are unsatisfactory, then you must pick the least worst, for you cannot hope for anything better. If there are evils, they are here to stay. If your job is horrible, then you may as well settle in, because “that’s life.” This is the attitude that would have boots stamping on human faces forever and people describing their relationship to their job with the motto “eat shit, cash checks.”

Thankfully, that attitude is false. It is always possible to create new options - to choose the “new option” option. There is no guarantee you’ll find one worth taking, but better options are always in principle available. The screen you’re reading from, the words I’m writing, and the freedom to share them - these once did not exist. Now they do, and they’re a testament to the possibility (and necessity) of creating new ideas, new technologies, and new ways of life. To the epic history and future of progress, I think only one response is rational: “Hell yeah!”

Tell me what you think by replying to this email, reaching me at carlosd@substack.com, or commenting on this Twitter thread:

Loading more posts…