History of LLMs

The Era of Mechanical Translation and How It Crashed — Fascinating birth of AI, first chatbots and the power of US Department of Defense

Why should I know about it?

“To fundamentally push the deep learning research frontier forward, one needs to thoroughly understand what has been attempted in the history and why current models exist in present forms”

Haohan Wang and Bhiksha Raj from On the Origin of Deep Learning

Large Language Models (LLMs) have a fascinating history that dates back to the early 1930s when the first ideas of computational linguistics were born. You may argue that this tracing is excessive and that LLMs have nothing in common with the old-fashioned prehistoric computer systems. You may also argue that LLMs are based on real, hard-core deep learning. However, deep learning itself originated in 1943, when the first ancestor of the artificial neural model was proposed by McCulloch and Pitts. Exactly 60 years ago! What took us so long to get to modern LLMs?

This series isn’t about drowning you in technical details. While we provide an extensive list of references for those who want to delve deeper, our main goal is to captivate your attention and share the influential developments that have shaped LLMs. Consider it a springboard for further exploration, a chance to find something in history that can inspire you for a new ML discovery. It’s an invitation to immerse yourself in the story of LLMs, which made such a splash last year.

In this episode, we’ll take you on a time-travel adventure from 1933 to 1966. Ready? Let’s dive into The Era of Mechanical Translation and How It Crashed!

The first steps in the US, USSR, and UK

The concept of mechanical translation (MT) has always been a distant dream that tickled the imagination of many inventors, but it wasn’t until the early 20th century that engineers and mathematicians began to develop the first concrete ideas about how to make it a reality.

In 1933, significant progress was made when two individuals, George Artsrouni, a French-Armenian, and Petr Smirnov-Troyanskii, a Russian, independently patented…