The Rise of Deep Learning: How AI’s Biggest Idea Finally Won

Deep learning is the closest thing AI has to an overnight success that took sixty years. The core idea — stacked layers of simple neuron-like units learning from data — was described in the 1960s. For decades it was mocked, defunded, and dismissed as a dead end. Then, in roughly 2012, it started winning everything, and it hasn’t stopped since.

I find this story useful because it corrects a common misunderstanding: deep learning didn’t win because someone had a brilliant new theory in 2012. It won because three separate things — data, compute, and a few clever training tricks — finally arrived at the same time. Understanding that history makes you much better at judging what’s hype and what’s real in AI today. For the full sweep of AI’s past, see my overview of the history of artificial intelligence.

In This Guide

• The Idea Is Older Than You Think
• The First Neural Winter
• Backpropagation: The Algorithm That Changed Everything
• The Second Winter: Why Neural Nets Were Abandoned Again
• The Three Ingredients That Finally Aligned
• 2012: The ImageNet Moment
• From Vision to Language: The Transformer Leap
• What the History Teaches Us
• Frequently Asked Questions

The Idea Is Older Than You Think

The perceptron, the ancestor of every modern neural network, was built by Frank Rosenblatt in 1958. It was a machine that could learn to distinguish simple visual patterns — punched cards marked left or right — by adjusting weights. The press went wild. The New York Times reported it as the embryo of a machine that would walk, talk, and be conscious.

Rosenblatt’s perceptron was genuinely important: it introduced the idea that a machine could learn from examples rather than being explicitly programmed. But it had a fatal limitation, and two critics were about to expose it in the most damaging way possible.

The First Neural Winter

In 1969, Marvin Minsky and Seymour Papert published Perceptrons, a book proving mathematically that single-layer perceptrons couldn’t learn simple functions like XOR — exclusive or. The math was correct but narrow: it applied to single-layer networks, and the authors knew multi-layer networks might overcome it. That nuance got lost.

The effect was devastating. Funding for neural network research dried up almost overnight. The field entered its first “winter” — a period where an entire research direction is considered dead and unfundable. It would last over a decade. Talented researchers left the field; those who stayed worked in obscurity.

Here’s what I want you to notice: the idea wasn’t wrong. The timing was wrong — no data, no compute, and the training methods weren’t there yet. This pattern repeats throughout AI history.

Backpropagation: The Algorithm That Changed Everything

The breakthrough that thawed the first winter was backpropagation — an efficient way to train multi-layer networks by propagating errors backward through the layers, adjusting each connection’s weight. The core mathematics had been floating around since the 1960s and 70s, but in 1986, Rumelhart, Hinton, and Williams published the paper that made it famous and practical.

Suddenly, multi-layer networks could learn the things single-layer perceptrons couldn’t — including XOR, the very function that had supposedly killed the field. Neural networks were back. The late 1980s saw genuine excitement again, and Yann LeCun’s work on convolutional networks for reading handwritten digits (used by banks to read checks through the 1990s) proved neural nets could do real work.

But the revival had a ceiling, and the field was about to hit it hard.

The Second Winter: Why Neural Nets Were Abandoned Again

Through the 1990s, deep networks — ones with many layers — simply wouldn’t train well. Gradients vanished as they propagated backward through layer after layer, so early layers barely learned anything. Training was slow, datasets were tiny by modern standards, and computers were thousands of times weaker than today.

Meanwhile, competing methods like support vector machines offered better results with cleaner mathematics and less fuss. By the late 1990s, mainstream researchers had left neural networks again. The second winter was arguably colder than the first: neural nets weren’t just unfashionable, they were considered embarrassing — the kind of thing serious researchers didn’t admit to working on.

Geoffrey Hinton, one of the backpropagation pioneers, kept going anyway, with a tiny group of believers. He later joked that the field survived because a handful of people refused to take the hint. That stubbornness turned out to be one of the most consequential acts of persistence in science.

If you’re curious about the mechanics that make modern training work, my explainer on how AI models are trained covers the process step by step.

The Three Ingredients That Finally Aligned

Deep learning’s rise wasn’t one breakthrough — it was three independent trends converging:

Data. The internet produced datasets of unprecedented size. ImageNet, assembled by Fei-Fei Li’s team, contained over 14 million labeled images — orders of magnitude more than anything before it. Deep networks are data-hungry; finally, there was enough food.

Compute. GPUs — graphics processing units designed for video games — turned out to be spectacularly good at the parallel matrix math neural networks need. A GPU could train in days what took CPUs months. Later, specialized AI chips pushed this further.

Algorithmic tricks. A series of practical innovations made deep networks trainable: ReLU activation functions (which eased the vanishing gradient problem), better weight initialization, dropout (a regularization trick from Hinton’s lab), and batch normalization. None was a grand theory — each was a clever hack that made training actually work.

Any two of these without the third wouldn’t have been enough. That’s the real lesson: revolutions often look like single breakthroughs but are really alignments.

2012: The ImageNet Moment

The moment everything changed has a date: September 30, 2012. At the ImageNet competition — the annual contest to classify images — a deep convolutional network called AlexNet, built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, crushed the competition. Its error rate was about 26%; the runner-up was around 16 percentage points worse. In a field where progress was measured in fractions of a percent, this was a demolition.

Within two years, every serious computer vision team had switched to deep learning. Within five, deep learning had conquered speech recognition, machine translation, and game playing — culminating in AlphaGo’s 2016 victory over Go champion Lee Sedol, a moment many compare to the moon landing for AI.

The funding followed the results. Tech giants built enormous AI labs. Startups raised billions. “Deep learning” went from an embarrassing phrase to the hottest words in technology.

From Vision to Language: The Transformer Leap

The final chapter of the rise — so far — belongs to language. In 2017, Google researchers published “Attention Is All You Need,” introducing the transformer architecture. Transformers handled sequences far better than previous approaches, and they parallelized beautifully on GPUs.

Then came scale. Researchers discovered that bigger transformers trained on more text kept getting better in predictable ways — “scaling laws.” GPT-3 in 2020 showed that a large enough model could write, reason, and code with startling fluency. ChatGPT in late 2022 put that capability in everyone’s hands and triggered the current AI boom.

Today’s large language models are deep learning’s most visible children. If you want to understand what they actually are under the hood, read my piece on what a large language model is.

What the History Teaches Us

Three lessons I take from this story:

Dismissed ideas can be right too early. Neural networks were “dead” twice. Both times the dismissal confused currently impractical with fundamentally wrong. When you hear that some AI approach is a dead end, ask whether it’s wrong or just early.

Progress comes from alignment, not just genius. Backpropagation existed for years before it mattered. The transformer paper was brilliant, but it needed GPUs and internet-scale data to change the world. Watch the infrastructure, not just the papers.

Winters end. AI has had multiple winters and multiple springs. The current boom will eventually cool — that’s normal and healthy. The underlying trajectory, though, has been upward for sixty years. The perceptron’s grandchildren are writing poetry and folding proteins. Rosenblatt would be stunned, and then he’d probably say he told us so.

The rise of deep learning is my favorite case study in why you should never confuse “not working yet” with “never going to work.” The researchers who kept the faith through two winters didn’t have better theories than the skeptics — they had patience, and eventually, the world caught up with the math.

Frequently Asked Questions