How AI Learns to Write:
The Simple Language Model That Explains GPT
Whether you’re using ChatGPT, Claude, Gemini, or another large language model, they all trace their roots back to a fundamental task: predicting the next piece of text based on what came before.
One of the simplest ways to understand this process is through a concept called the bigram language model. While primitive by today’s standards, the bigram model captures the essential idea that powers modern AI systems.
Let’s explore how it works and why it remains one of the best ways to understand the foundations of artificial intelligence.
The Challenge - Teaching a Machine to Write
Imagine you wanted to create an AI capable of writing like Shakespeare.
You could start by feeding it every available play, sonnet, and poem. But that raises an important question:
How does a machine actually learn from text?
Unlike humans, AI doesn’t begin with an understanding of grammar, meaning, emotion, or storytelling.
Instead, it starts by identifying patterns.
The first step is converting text into individual units called tokens. In modern language models, tokens are often words or subwords. In simpler examples, they can be individual characters.
Consider the phrase:
To be or not to be
A basic model might view this simply as a sequence of characters:
T → o → space → b → e → space...
The model’s task becomes surprisingly straightforward:
Given one token, predict the next token.
That’s the entire game.
Building a Vocabulary of Language
Before training can begin, the model needs to know which symbols exist in its world.
A dataset containing Shakespeare’s works might include:
Uppercase letters
Lowercase letters
Spaces
Punctuation marks
Numbers
Special symbols
Each unique character becomes part of the model’s vocabulary.
For example:
A
B
C
...
?
!
;
If there are 65 unique characters, the model must learn the likelihood of each character following every other character.
This turns language into a probability problem.
The Bigram Model, Learning One Step at a Time
A bigram model only considers one token of history.
If the current token is:
T
the model attempts to predict what comes next.
Possible outcomes include:
h
o
r
a space
punctuation
Each possibility receives a probability score.
For example:
These probabilities are learned from data rather than manually programmed.
Over time, the model discovers patterns such as:
“th” appears frequently
“to” is common
punctuation often follows complete words
Without understanding language, the model begins reproducing its statistical structure.
Why Random Models Produce Gibberish
Every AI model starts with random values.
Initially, the model has no idea which characters belong together.
As a result, generated text might look something like:
xv#;n@tk!!pr
This isn’t a bug.
It’s exactly what we should expect.
At the beginning of training, every prediction is essentially a guess.
The model gradually improves by comparing its guesses against real examples and adjusting itself accordingly.
This process is known as machine learning.
Measuring Mistakes - The Role of Loss Functions
To improve, the model needs a way to measure how wrong it is.
This measurement is called the loss function.
Suppose the correct next character after “T” is “h.”
If the model predicts:
h = 70% probability
it receives a small penalty.
If it predicts:
h = 2% probability
it receives a much larger penalty.
One of the most common approaches used in language models is based on negative log likelihood, which heavily penalizes confident mistakes.
Researchers at urlGoogle Researchhttps://research.google and other AI labs rely on similar probability-based objectives because they effectively teach models to assign higher confidence to correct predictions.
The goal of training becomes simple:
Reduce the average loss over time.
As loss decreases, prediction quality improves.
Softmax - Turning Scores Into Probabilities
Neural networks don’t naturally produce probabilities.
Instead, they output raw numerical values called logits.
For example:
1.7
4.3
0.2
2.1
These numbers are difficult to interpret directly.
That’s where the softmax function comes in.
Softmax converts raw scores into probabilities that:
Fall between 0 and 1
Sum to 100%
Can be interpreted as confidence levels
Researchers explain softmax as a normalization mechanism that transforms arbitrary outputs into meaningful probability distributions, making prediction and learning possible. urlStanford University’s CS231n explanation of Softmaxhttps://cs231n.github.io/linear-classify/#softmax
Without softmax, language models would struggle to determine which prediction is most likely.
Why Training Happens in Small Batches
Modern AI systems train on enormous datasets.
Processing everything at once would be computationally impractical.
Instead, models learn using small subsets called batches.
This approach offers several benefits:
Faster computation
Better memory efficiency
Parallel processing across hardware
More stable optimization
The technique is standard practice throughout machine learning and deep learning research. urlDeep Learning Book by Ian Goodfellow, Yoshua Bengio, and Aaron Courvillehttps://www.deeplearningbook.org/
By repeatedly training on thousands or millions of small batches, models gradually improve their predictions.
The Hidden Art Behind AI Training
Many people assume AI training is purely mathematical.
In reality, it involves a surprising amount of experimentation.
Researchers must decide:
How quickly should the model learn?
How large should updates be?
How much data should be processed at once?
Which optimization method works best?
Move too aggressively and training can become unstable.
Move too cautiously and progress slows dramatically.
This balancing act is managed through techniques such as:
Backpropagation
Gradient descent
Learning rate scheduling
Optimizers like Adam
These tools help guide the model toward better solutions while avoiding instability.
Where the Bigram Model Falls Short
Despite its elegance, the bigram model has a serious limitation.
It only remembers one previous token.
Consider the sentence:
The king entered the hall because he was summoned.
When predicting the word “he,” understanding requires remembering information much earlier in the sentence.
A bigram model cannot do that.
It only sees:
Current token → Next token
This limitation prevents it from understanding:
Context
Narrative structure
Long-range relationships
Meaning across sentences
The result is text that may resemble language but often lacks coherence.
The Breakthrough That Changed Everything
The next major advance came from the introduction of attention mechanisms and transformer architectures.
In the landmark paper Attention Is All You Need, researchers introduced a method allowing models to consider many previous tokens simultaneously rather than just one. urlAttention Is All You Need, the paper that introduced transformershttps://arxiv.org/abs/1706.03762
This innovation transformed language modeling.
Instead of asking:
What usually follows this character?
Models could ask:
What parts of the entire context matter right now?
That shift enabled the creation of modern systems such as ChatGPT and other large language models.
Key Takeaways
If you remember only a few ideas, make them these:
Language models learn by predicting what comes next.
Bigram models use only one previous token as context.
Training improves predictions by reducing loss.
Softmax converts raw outputs into probabilities.
Batching makes large-scale training practical.
Attention mechanisms allow models to use broader context.
Modern GPT-style systems evolved from these foundational ideas.
The remarkable thing is that today’s most advanced AI systems still rely on the same fundamental principle as the simplest language models:
Prediction drives intelligence.
Everything else is an increasingly sophisticated way of making better predictions.
One Question to Consider
If a machine can learn to produce convincing language simply by predicting what comes next, what does that reveal about human communication itself? Are we expressing ideas, or are we constantly making predictions about each other’s thoughts, one word at a time?
This version reads like an independent educational feature article rather than a summary of someone else’s content, making it much more suitable for publication, SEO, and newsletter distribution.