i was wrong about ai
In a Nutshell
The video argues that modern AI, especially transformers, works by taking simple mechanisms—like edge detection in vision or next-token prediction—and scaling them massively with compute, rather than relying on handcrafted human-like understanding. This aligns with the "bitter lesson": over time, brute-force scaling outperforms clever, biologically inspired designs. It questions whether human intelligence and consciousness are fundamentally different or just more instances of simple processes scaled up by evolution.
These notes were generated by AI and may contain inaccuracies.
In the spring of 1958, in a basement at Johns Hopkins, researchers had a cat strapped into a chair with its eyelids propped open and a wire going directly into the back of its skull into the part of the brain that handles vision. That wire ran to a speaker. When a brain cell fired, the speaker went off. For weeks, they flashed dots of light at this cat using little black spots on glass slides, and nothing happened. Then by chance, one of them went to swap the glass slide and as he slid it into the projector, the edge of the slide swiped across the screen and the cat's brain exploded. Not literally, but the speaker went off like a machine gun.
The neuron did not care about some dot. It cared about the edge. Not just any edge. It had to be at a very specific angle. If you tilted it away, it went quiet. If you tilted it back, it lit up again. They found a single brain cell whose entire reason for existing was basically detecting that there is a line and it leans this specific way.
What they found is that vision is not like a camera. The brain does not take a picture. Instead, the brain has a layer of cells that each give attention to one tiny thing, like a line at this angle or an edge moving that way. They called these things simple cells. Those simple cells feed into complex cells that combine the simple things into slightly less simple things. So edges become corners, corners become shapes, and shapes become a face.
They won the Nobel Prize in 1981 for their discoveries in information processing in visual systems. This was the basis of convolutional neural networks which Yann LeCun pioneered in 1989. It is basically a bunch of small filters, each one responsible for spotting something very simple in a little patch of an image like an edge. Then when you layer and stack these together, they start being able to detect bigger things like cats and dogs in a photo.
Convolutional neural networks turned out to be incredibly good at vision tasks. It is the same tech behind the cameras in self-driving cars. But they were less suited to language. The word convolutional basically refers to sliding across an image and checking one small segment at a time, which means proximity matters. Things that are close together are what get compared.
In language, the things that matter to each other can be very far apart. You can say the word "it" in a sentence and it could point back to something all the way back at the beginning of the paragraph. These networks struggled with these kinds of long-distance connections.
In 2017, a big leap came with the paper "Attention Is All You Need," which introduced transformers, which is what the T in GPT stands for. The idea is that each word, or really each token because words can get chopped into smaller pieces, can pay attention to all the other words around it. Think of every word as sending out a little signal that says this is what I am about while also listening to the signals coming from everyone else. Leaning harder towards the ones that are most relevant to it and barely paying any attention to the ones that are not. This all-at-once way of relating words is essentially what made today's large language models possible.
Simple things scaled up. Like the visual cortex, which builds vision out of a stack of simple edge detectors, AI is a story of simple things scaled up. It is as much a hardware story as it is a software one. Many of the core ideas that power our AI today go back decades. The first artificial neuron models appeared in the 1940s and 1950s. The way we train these networks was worked out in the 1980s. While the architectures behind today's large language models are somewhat newer, what was missing from most of that history was not the ideas, it was the raw computing power to bring them to life.
Now we have that. The current AI surge is in large part a story about hardware chips that run thousands of calculations in parallel, reaching trillions of operations every second. In essence, LLMs are simple and naive. So naive you would think they would never work. Take a dumb algorithm and just blast it with enough computing power to light up a small city. And out come these seemingly magical results.
At first the idea was that this was a hack. The fact that we are taking something so simple, so naive, and just scaling it up could not be real science. But this is what Richard Sutton called the bitter lesson. The bitter lesson is a short essay that basically said over the long run, trying to be clever about AI tends to lose out to methods that just scale up with raw computing power. You do not need to handcraft human-style understanding into a machine. You just have it do ungodly amounts of statistical computation.
It is somewhat similar to animation. Think Mickey Mouse. People have known how to draw for thousands of years. But a stack of drawings just sit there. The magic only appears when you have a machine that can show those drawings in rapid succession, flashing about 24 pictures every second, and your brain blends those separate frames together. And suddenly, still drawings seem to move and come to life.
AI is a bit like that. Usually the thought was that because this is a projection, because it is all a hack, a trick, it is nothing to get concerned about. But recently the question has arisen whether maybe that assessment was wrong. Because in fact, a lot of what the brain does might be simple things scaled up, too. It is the bitter lesson again. The idea that simple methods plus massive computation tend to win. It is doing small things millions and billions of times over, shaped across hundreds of millions of years in a lifetime of learning and reinforcement.
Are LLMs and human brains exactly the same? Obviously not. But in a sense, the human brain fits a kind of curve to the real world, approximating whatever helped our ancestors and us survive. And it turns out machines can fit a curve, too.
The question arises: is it not just autocomplete, though? The naive assumption is that during training, the model simply predicts the next word based on how often words appear in the training data. But that is not quite right. The model is not scored on raw frequency, but on whether it predicts the right next word for this particular context.
So, if you show the model a paragraph and ask it to fill in the word after the word "the," in the data, maybe the single most common word after "the" is something like "man." But if the actual answer in this paragraph is something else, then guessing "man" counts as a mistake. And when the model guesses wrong, it gets penalized. And that penalty is used to nudge its internal settings just a little so that next time when a similar context comes up, it is more likely to get it right. Now, repeat that across millions and millions of examples. With each wrong guess, the model adjusts and slowly it settles into a kind of fit, an incredibly complex one.
The only way to actually win this game, to keep guessing the right next word, is to stop leaning on raw frequency and start picking up on what the previous sentences and paragraphs actually mean. So, it ends up capturing a sort of essence. This alludes to what is essentially called a world model. There is a question of whether LLMs have a world model, whether they have fitted a curve not to the training data itself but to the process that produced the data. So essentially one level prior, which is big if true, and there is argument on both sides here.
It is remarkable how well a simple thing scaled up works in the world of computers. And it is also remarkable how we find the same thing in nature. The question arises of what if there truly is no limit to how well AIs can approximate reality or form a line of best fit that could be even more functionally useful than ours. These are thoughts that have no present basis in reality.
At this point, the usual pushback is that AI cannot do love, feeling, quality, sensation, experience. But recently, the haunting question has been that simple phrase: simple things scaled up. Is that what everything is? If that is how the eye works, if that is how vision works, why stop there?
This conversation is not really just idle curiosity or aimless philosophizing. It is really a question of what it means to be human. Because first AI came for our jobs and now it is sort of coming for our spirituality. What remaining centrality humans ascribe to themselves to make reality and suffering more tolerable, to give meaning to the chaos. We hold out on intelligence and maybe consciousness. We say intelligence is special. It is not a simple thing scaled up. It is a different category altogether. And maybe that is right. That is what we like to believe. But there is a reason it is called the bitter lesson.
Keep Mo Bitar in your library
Save the videos and channels worth coming back to, and find them again in one place.
