Trains, planes, and automobiles

The recent disagreements (for example the face-off between Gary Marcus and Yann LeCun, or Gary’s recent NY Times article and the ensuing Twitter conversations, or Judea Pearl’s discussion of his new book) about whether we’re on the “right track” in AI research seem a little strange to me.

As I’m a fan of metaphors, I’ll liken the goal of general AI to the goal of inventing a car.

Logic. All the cool kids are doing it.

Decades ago, researchers had a big effort in knowledge engineering and logical reasoning. The path towards human-level AI seemed obvious; we must encode enough world knowledge into a system, and give it the correct algorithms to draw on that world knowledge.

In my metaphor, they were designing steering wheels, speedometers, and other devices that were intended for human understanding and communication while driving.

We made great progress, but that work was largely unusable for practical applications. I mean seriously… who wants a steering wheel without anything to drive?! Many critics concluded that the efforts were wasted, and that the approaches were fundamentally flawed.

Just kidding… Neural networks are the answer.

Also decades ago, we saw the invention of the core mathematics and inventions driving advances in neural networks. Backpropagation, feed-forward neural nets, convolutional layers, and Long-Short-Term-Memory (LSTM) are examples of things that existed a long time ago, but have only hit their stride in the last few years.

In my metaphor, we developed the wheels and chassis, along with the required physics understanding of torque and friction.

However, despite high expectations, this also disappointed initially, since what good is a cart without a horse to pull it? This disappointment led to an AI winter, until…

I need a new gaming PC… to do research (I promise!)

In the early part of this decade, a group of researchers discovered that with enough compute power, old techniques using neural networks could perform very well in sensory-motor domains such as image recognition and speech-to-text. They had found a horse to pull their dusty old cart!

Rather, they now had an engine! Combining the “engine” with the “wheels” allowed for numerous applications, with seemingly limitless possibilities. However, all that power is limited to very narrow domains: asking Echo to play a song, or getting calendar entries from Siri, or making a haircut appointment with Duplex.

Without the ability to interface seamlessly with humans, the technology must stay “on the rails”, like a train.

Now, no one can deny that railways were both useful and transformative. They paced the way for efficient trade and globalization of economies. But while railways were important for industry, they had less impact on the day to day lives of people. It was, quite simply, not as transformative as the automobile.

Back on topic: “What is the right track for AI?”

We’ve separately invented the steering wheel and controls (world knowledge and logical reasoning), the tires and the chassis (statistical models aka Deep Learning), and a powerful engine (CPUs, GPUs, and TPUs… oy my!). But we don’t have a car (AI).

I think Gary, Judea, and others simply feel this: no one has invented the car, and we won’t get there by improving the tires. We shouldn’t choose between logic or statistics… we should build a system that uses both! It works for humans, after all.