BERT-style
encoder-only
Sees: left and right context.
Trains: reconstruct masked tokens.
Used for: classification, token labelling, embeddings, extractive QA, retrieval.
Full Stack & AI · Day 04 · Chapter 02
Large
Language
Models
Language models predict likely language units from context. LLMs scale that idea with broad data, parameters, compute, and Transformer architecture.
A chat response is a visible surface over repeated next-token prediction—not the definition of a language model.
Input tokens
“The bank…”
“The bank…”
→
Contextual representation
nearby tokens change meaning
nearby tokens change meaning
↓
Next-token probabilities
→
Selected token
then repeat
then repeat
From learned patterns to text
Learn patterns once.
Generate step by step.
During pre-training, a language model adjusts its parameters to predict language patterns. At runtime, those learned parameters turn a prompt into one next-token decision at a time.
1. TrainRepeated examples adjust parameters to reduce prediction error.
→2. RepresentParameters capture patterns in syntax, style, code, and meaning.
→3. InferA new prompt enters the learned processing pipeline.
→4. GenerateSelect a likely next token, append it, then repeat.
The next question is how a model can use a long prompt, examples, and instructions to make that next-token decision.
Why Transformers changed the scale
LSTMs could generate.
Attention made long-context learning practical.
Both LSTMs and decoder Transformers can predict the next token. The difference is how they carry and retrieve context while training at scale.
LSTM: serial recurrenceEach token updates one evolving hidden state. Earlier information must pass through many sequential steps and can be compressed or overwritten.
Transformer: direct attentionEach token keeps its own representation. Attention can directly relate the current token to relevant earlier tokens instead of relying on one summary state.
What this enabledParallel training over long sequences and more effective use of instructions and examples in a prompt—what we call in-context learning.
A Transformer does not update its weights from a few-shot prompt. It temporarily uses the examples and instructions already present in that prompt to continue the pattern.
From language model to LLM
“Large” is not just
more parameters.
An LLM combines model capacity, broad data, substantial compute, and a training method that yields reusable language representations.
Parameterslearned capacity
+Broad datapatterns across language and code
+Computetraining at scale
Not a fixed threshold
No universal parameter count defines “large.” The term describes the engineering regime in which a broadly trained language model can be prompted or adapted across many tasks.
Foundation model ≠ only language
An LLM is language-centred. A foundation model may span text, images, audio, and other modalities.
Why text must be transformed
Text is open-ended.
Neural networks need numbers.
A tokenizer turns arbitrary text into a finite, reusable vocabulary of numeric units.
unpredictable→[491][10321][987]
Illustrative example: unpredictable → un · predict · able → token IDs. Actual pieces and ID values are chosen by each model’s tokenizer.
Tokenizer
Text → subword tokens → token IDs → model input.
When the model generates IDs, the tokenizer performs the reverse conversion back to text.
Why subwords?
They avoid requiring a separate vocabulary entry for every rare word, identifier, spelling variant, or code fragment.
From identifiers to representations
An ID is a label.
An embedding is a vector.
The embedding matrix turns every token ID into a learnable continuous representation that the network can transform.
ID [10321]the ID for predict in this illustrative tokenizer
→Embedding lookupfind row 10321 in the learned embedding matrix
→[0.12, −0.84, 0.37, …]initial embedding vector; real models use hundreds or thousands of dimensions
→Transformer layerscontext-aware representation
Position is required too
The embedding says which token it is; positional information tells the model where it occurs. “Dog bites man” and “man bites dog” contain the same tokens in a different order.
Context changes the vector
The initial embedding for bank is the same lookup. Transformer layers produce different contextual vectors in “river bank” and “bank loan.”
Working memory for one response
The context window is bounded.
It is the token budget a model can attend to while producing this response.
The window makes tokens available; it does not decide which available token matters for the current prediction.
Instructions
+Prompt + history
+Retrieved evidence
+Generated output
Why a limit exists
Attention has memory and compute costs as sequences grow. The model cannot treat unlimited history as input.
Capacity is not accuracy
Long context can contain more material; it does not guarantee that the model selects, interprets, or verifies it correctly.
The original Transformer · Vaswani et al., 2017
Now locate attention in the encoder–decoder system.
The attention mechanism you just saw appears three ways: encoder self-attention, decoder masked self-attention, and decoder cross-attention.
The encoder reads the source. The decoder produces the target using earlier output and the encoder’s contextual representation.
Example task: “The animal sleeps” → “L’animal dort”
Source text
The animal sleeps
↑
↑
Encoder stack × N
Multi-head self-attention
Add + LayerNorm
Feed-forward network
Add + LayerNorm
↑
Encoder output
Contextual source vectors: keys and values for cross-attention.
encoder outputK, V ───▶decoder cross-attention
Shifted target prefix
<BOS> · L’ · animal
↑
↑
Decoder stack × N
Masked multi-head self-attention
Add + LayerNorm
Encoder–decoder cross-attentiondecoder request (Q)← encoder match tags + content (K, V)
Add + LayerNorm
Feed-forward network
Add + LayerNorm
↑
Vocabulary projection
Final decoder vector → one raw score (logit) for every vocabulary token.
↑
Softmax → candidate probabilities
dort0.61
dors0.13
est0.07
Illustrative probabilities. Greedy decoding selects the top token; sampling may choose from the permitted candidates.
Tokenise the source sentence. Its token IDs, embeddings, and position information enter the encoder.
The problem attention solves
Availability is not relevance.
The context window exposes many tokens. Attention decides which of those available tokens should influence this token now.
Attention asks: among the available tokens, what context matters for this token now? To interpret it, animal should matter more than street.
Example
“The animal did not cross the street because it was too tired.”
Question: Which earlier token should shape the representation of it?
animal → highly relevant
street → not the antecedent
Inside one attention head
One token. Three projections.
One contextual update.
The animal was tired, so it slept.
Theanimalwastiredsoitslept
same input H → learned projections
H × WQlearned projection
one focus Query
H × WKlearned projection
one Key per token
For the focus token it, compare Q(it) with every Key.
Q(it)
all candidate Keys
K(The)K(animal)K(was)K(tired)K(so)K(it)K(slept)
attention weights after softmax (illustrative)
0.030.700.040.080.020.050.08
H × WVlearned projection
one Value vector per tokenV(The)V(animal)V(was)V(tired)V(so)V(it)V(slept)
0.03·V(The) + 0.70·V(animal) + 0.04·V(was) + …
↓
a(it) = attention output for itone new vector: a weighted sum of the Value vectors
original H(it)
+
a(it)
=
contextual H′(it)
Theanimalwastiredsoitslept
Single-head simplification: a full Transformer also applies an output projection and LayerNorm around the residual connection.
Start with one input representation for every token.
Why more than one head?
One attention pattern answers one kind of relationship.
A Transformer runs several learned attention heads in parallel, so the same sentence can be examined through different relevance patterns at once.
Each head has its own learned Query, Key, and Value projections.
Head 1reference pattern
Theanimalwastiredsoitslept
it ← animal
One learned way to resolve reference.
One learned way to resolve reference.
Head 2event pattern
Theanimalwastiredsoitslept
slept ← tired
A different learned relevance pattern.
A different learned relevance pattern.
Head hanother pattern
Theanimalwastiredsoitslept
not hand-written
Heads discover useful patterns during training.
Heads discover useful patterns during training.
Heads run in parallel → concatenate + projectOne richer, context-aware representation for every token.
Same Transformer ideas, different visibility
Encoder models, decoder models,
and encoder–decoder models.
BERT-style
GPT-style
decoder-only
Sees: earlier tokens only.
Trains: predict next token.
Used for: chat, text generation, code completion, open-ended continuation.
T5 / BART
encoder–decoder
Sees: source in encoder; past target in decoder.
Trains: source-to-target reconstruction.
Used for: translation, summarisation, controlled transformation.
Learning and generation
Pre-training learns patterns.
Decoding chooses a response.
The model learns by comparing its prediction with a known training token. At runtime, it selects a token from a score distribution and repeats.
Broad corpus
→Self-supervised prediction
→Parameter update
→Reusable patterns
Decoding controls
Temperature changes how concentrated the next-token distribution is; it is not a factuality control.
Top-k / top-p restrict candidate tokens before sampling. Max output / stop sequences bound when generation ends.
What follows from this design
Capabilities are useful only
with a matching evaluation.
Capability
Drafting and continuation
Check with a rubric and human review.
Check with a rubric and human review.
Summarisation and transformation
Check fidelity to the source.
Check fidelity to the source.
Code assistance
Check tests, static analysis, and review.
Check tests, static analysis, and review.
Structured extraction
Check schema and source evidence.
Check schema and source evidence.
Limitation
Hallucination
Token likelihood is not external verification.
Token likelihood is not external verification.
Knowledge freshness
Parameters are not a live source of truth.
Parameters are not a live source of truth.
Prompt injection
Untrusted text can compete with instructions.
Untrusted text can compete with instructions.
No accountability
Model output is not an authorised decision.
Model output is not an authorised decision.
An LLM is a probabilistic sequence model with learned representations—not a reasoning oracle, database, or autonomous authority.
Continue the narrative
Next: Prompt Engineering
We now use this mental model to structure prompts, add constraints and evidence, and evaluate model outputs deliberately.
Open Chapter 03 →