There has been some recent discussion on X about the origin of Transformers—specifically the attention mechanism—sparked by a post from Carlos Perez, where he claims the initial insight was stimulated by the 2016 film, Arrival.

Andrej Karpathy, the famous AI researcher, felt this was contrary to his recollection, so he did us all a favor and published an email interaction he had with the author some 2 years prior. This is a seminal moment in modern AI, so I transcribed the email and provided some follow-up commentary below.

Dzmitry Bahdanau response to Andrej

Hi Andrej,

Happy to tell you the story as it happened 8 years ago!

I came to Yoshua’s lab as an intern, after having done my first year of MSc at Jacobs University with Herbert Jaeger.

I told Yoshua I’m happy to work on anything. Yoshua put me on the machine translation project to work with Kyunghyun Cho and the team. I was super skeptical about the idea of cramming a sequence of words in a vector. But I also really wanted a PhD offer. So I rolled up my sleeves and started doing what I was good at – writing code, fixing bugs and so on. At some point I showed enough understanding of what’s going on that Yoshua invited me to do a PhD (2014 was a good time when that was enough – good old times!). I was very happy and I thought it’s time to have fun and be creative.

So I started thinking about how to avoid the bottleneck between encoder and decoder RNN. My first idea was to have a model with two “cursors”, one moving through the source sequence (encoded by a BiRNN) and another one moving through the target sequence. The cursor trajectories would be marginalized out using dynamic programming. Kyunghyun Cho recognized this as an equivalent to Alex Graves’ RNN Transducer model. Following that, I may have also read Graves’ handwriting recognition paper. The approach looked inappropriate for machine translation though.

The above approach with cursors would be too hard to implement in the remaining 5 weeks of my internship. So I tried instead something simpler – two cursors moving at the same time synchronously (effectively hard-coded diagonal attention). That sort of worked, but the approach lacked elegance.

So one day I had this thought that it would be nice to enable the decoder RNN to learn to search where to put the cursor in the source sequence. This was sort of inspired by translation exercises that learning English in my middle school involved. Your gaze shifts back and forth between source and target sequence as you translate. I expressed the soft search as softmax and then weighted averaging of BiRNN states. It worked great from the very first try to my great excitement. I called the architecture RNNSearch, and we rushed to publish an ArXiv paper as we knew that Ilya and co at Google are somewhat ahead of us with their giant 8 GPU LSTM model (RNN Search still ran on 1 GPU).

As I later turned out, the name was not great. The better name (attention) was only added by Yoshua to the conclusion in one of the final passes.

We saw Alex Graves’ NMT paper 1.5 months later. It was indeed exactly the same idea, though he arrived at it with a completely different motivation. In our case, necessity was the mother of invention. In his case it was the ambition to bridge neural and symbolic AI, I guess? Jason Weston’s and co Memory Networks paper also featured a similar mechanism.

I did not have the foresight to think that attention can be used at a lower level, as the core operation in representation learning. But when I saw the Transformer paper, I immediately declared to labmates that RNNs are dead.

To go back to your original question: the invention of “differentiable and data-dependent weighted average” in Yoshua’s lab in Montreal was independent from Neural Turing Machines, Memory Networks, as well as some relevant cog-sci papers from the 90s (or even 70s; can give you any links though). It was the result of Yoshua’s leadership in pushing the lab to be ambitious, Kyunghyun Cho’s great skills at running a big machine translation project staffed with junior PhD students and interns, and lastly, my own creativity and coding skills that had been honed in years of competitive programming. But I don’t think that this idea would wait for any more time before being discovered. Even if myself, Alex Graves and other characters in this story did not do deep learning at that time, attention is just the natural way to do flexible spatial connectivity in deep learning. It is a nearly obvious idea that was waiting for GPUs to be fast enough to make people motivated and take deep learning research seriously. Ever since I realized this, my big AI ambition is to start amazing applied projects like that machine translation project. Good R&D endeavors can do more for progress in fundamental technologies than all the fancy theorizing that we consider the “real” AI research.

That’s all! Very curious to hear more about your educational AI projects (I heard some rumors from Harm de Vries :)).

Cheers,

Dima

The following is ChatGPT's interpretation and summary of the email:

Bahdanau’s email is valuable because it cuts through the mythology that often forms around major breakthroughs in AI. Attention, in his telling, was not born from a cinematic epiphany or a grand theoretical theory. It emerged from a practical problem in machine translation: how do you get a computer to translate a sentence without making it forget important parts of the original?

Early neural translation systems worked a bit like this: first, the model would read the entire sentence in one language and compress it into a single internal representation. Then it would use that compressed representation to generate the sentence in another language. That sounds reasonable, but it created a serious bottleneck. A sentence can contain many words, relationships, and details. Trying to squeeze all of that into one fixed bundle of information is like asking someone to read a paragraph, close the book, and translate it from memory without being allowed to glance back.

Attention solved that problem by letting the model “look back” at the original sentence while producing the translation. Instead of relying on one compressed memory of the whole input, the model could decide which parts of the source sentence mattered most at each step. When generating one word, it might focus heavily on the noun. When generating the next, it might focus on the verb. The model was no longer translating from memory alone. It had a learned way of shifting its focus.

Bahdanau’s first attempts were more literal. He imagined something like cursors moving through the source and target sentences, tracking where the model should look as it translated. But those designs were either too complicated or too rigid. The breakthrough came when he made the idea softer and more flexible. Instead of forcing the model to choose one exact word to focus on, he let it assign different levels of importance to all the words in the sentence. Some words received high weight, others low weight, and the model used that weighted mixture to decide what to generate next.

This is the basic idea of attention: not all information matters equally at every moment. A model should be able to decide what to focus on, depending on what it is currently trying to do.

That simple idea turned out to be enormous. In translation, attention helped solve the problem of alignment: figuring out which parts of an original sentence correspond to which parts of the translated sentence. But the idea was much bigger than translation. Once researchers realized that models could learn what to pay attention to, attention became a general tool for processing information. It gave AI systems a way to connect relevant pieces of data, even when those pieces were far apart.

This is why attention became central to Transformers, the architecture behind modern large language models. In older systems, information often had to move step by step through a sequence, like a message being passed down a line. Transformers changed that. Using attention, every word can directly relate to every other word. The model can compare, connect, and weigh relationships across the whole input at once. That is one reason Transformers became so powerful.

What makes Bahdanau’s account especially interesting is how ordinary the ingredients were. He was a motivated intern trying to prove himself. He was working in Yoshua Bengio’s lab, alongside Kyunghyun Cho and others, on a real machine translation project with competitive pressure from other teams. He had limited time, strong coding skills, and a concrete problem in front of him. His inspiration partly came from the human act of translation itself: when people translate, their eyes naturally move back and forth between the original sentence and the sentence they are writing. Attention turned that everyday behavior into a machine-learning mechanism.

The email also shows that attention was an idea whose time had come. Similar mechanisms were appearing around the same period in other research groups. Bahdanau does not present the invention as a lone genius moment. He presents it as the result of the right problem, the right tools, the right lab, and the right moment in the field. Neural networks had become powerful enough, GPUs had become useful enough, and machine translation had exposed a clear enough weakness that someone was going to find this path.

That is the deeper lesson. Attention was not magic. It was a simple answer to a hard question: how can a model decide what matters right now? From that question came one of the most important ideas in modern AI.

The origin of attention was not Arrival. It was arrival in another sense: the arrival of the right problem, the right hardware, the right research culture, and the right person with enough taste to turn a messy cursor idea into something elegant. Attention began as a way to help machines translate. It became one of the foundations of modern artificial intelligence.