Progress always requires four things
Novelty, Taste, Control, Memory
Sustained progress requires four things: novelty, taste, control, and memory.
This is true for any system.
a giraffe, evolving to be taller or stronger
an AI, training to be better at math problems
an animal, playing and learning to fight
a human, pondering new recipes or new physics
Novelty
If you try new things, there's no guarantee they'll be better than what you started with. But, if you never try anything new, you ARE guaranteed to never improve yourself. (Though things outside your control may make your situation better or worse, like a war or random cash giveaway.)
Novelty just means doing something new. That said, there are different measures of novelty. A box of bouncing molecules is always doing something new - you'll never see the molecules in the same configuration twice. But, if you ask what it looks like, it always looks random. In that sense, it never does anything new. There are infinitely many ways to look at the box and what it's doing. Infinitely many measures of novelty.
What is a measure of novelty? It takes a thing - a box of molecules, say - and asks a question about it which has some number of different answers. Here's one question: Does the box contain life? That has two answers, yes or no. If the box doesn't contain life, and then it does, then it the life question gets a new answer. By that measure, the box has done something novel.
A different question might be: what is the temperature of the box? To achieve novelty in this sense requires the box to reach a new temperature - something higher or lower than it has before.
A musician never really creates new notes, but they can make new melodies, or inspire new feelings.
Taste
Like measures of novelty, a system needs measures of quality. Which animal is fastest or strongest. Which machine learning model is best at face recognition. Which recipe tastes best. Which scientific theory make the most sense. Without such measures, a system is exploring the possibilities at random, and unlikely to find good stuff, and especially unlikely to recognize good stuff and focus on it.
Control
It's one thing to have a sense of quality - what's good and what's bad, and in what way. But, you have to be able to *do something* with that information. If you couldn't, then you'd be like a consultant at a company that nobody listened to. You might know what to do, but that will have zero impact on what happens. If the company is making a mistake, then they'll go on making it.
When a lion catches a gazelle, two things happen. First, it has established the gazelle is slow (either at running, or at recognizing danger), and then... it eats the gazelle. That removes its genes from the gene pool. It is a biological consultant that is being listened to. The alternative would be that the lion catches the gazelle, then lets it go. That gazelle has been found to be slow, but that information gets ignored by evolution, and the fast gazelles and slow ones reproduce equally, and the species never evolves to become faster.
Memory
To make continued progress, you need not only to *make* improvements, but *keep* them, so you can build on them. If you have amnesia and your library burns down every day, then you have to start from scratch, and can never discover or create anything that requires more than a day of work. If you want to invest centuries into a cathedral, or a billion years into ever-more-complex living organisms, you'll need to save your work along the way. The bricks of the cathedral need to stay where you put them. The genes that specify an organism need to persist - either in one instance, or in copies. Otherwise, evolution would have to start from scratch - and giraffes can't be built in a day.
Without novelty, things remain the same. Ask: what are the sources of novelty in the system? What kinds of novelty?
Without taste, the good and the bad are indistinguishable. Ask: how does the system distinguish the good from the bad? What measures does it use? Can these change over time?
Without control, taste has no impact. Ask: if a system has taste, how does that impact what happens? Once something is determined to be good or bad, what is done with that information?
Without memory, all prior work is lost. Ask: when something happens in the system, how does affect the distant future of the system? How are the results saved? Where is the information stored?
Lose any of these four things, and sustained progress is impossible. You need all four. That said, they can each be achieved in innumerable ways. Evolution did them one way. Animals like giraffes and chimps do them in other ways. LLMs yet another. And humans achieve them in a uniquely powerful way - and that's why a human is writing this, a human is reading this, and humans invented the hardware and culture involved in making all that happen.