Full Stack & AI · Day 04 · Chapter 01

What is
Generative
AI?

A family of models that learns patterns and structure from data, then produces new synthetic content in response to an input or condition.

Modern systems commonly use foundation models: broadly pre-trained neural networks that provide this generative capability.

Training data
patterns and structure
→
Learned model
captures a distribution
↓
Input or condition
→
New output
text · code · image · audio · JSON
Three questions, not one

What it does. How it is built.
How it is trained for reuse.

Objective

Generative AI creates a new sample; discriminative AI predicts a label, score, or value. An LSTM can do either job, depending on its training objective.

Architecture

CNN, RNN/LSTM, and Transformer describe how a model is built. Architecture alone does not make a model generative or discriminative.

Foundation-model paradigm

A foundation model is broadly self-supervised pre-trained, then adapted across tasks. LSTMs and GANs could be generative long before foundation models.
Broad data becomes a reusable base

Foundation models
are the engine underneath.

Broad datatext · code · images · audio
→
Self-supervised learningthe data creates its own target
→
Foundation modelreusable learned patterns
Foundation
model
→
Promptingspecify this request
Retrievaladd approved evidence
Fine-tuningadapt learned behaviour
→
Task-specific
application

They are called foundation models because one broadly pre-trained base can support many downstream applications through adaptation. It is a base capability—not a complete application.

How training works at scale

Self-supervised learning:
the data teaches the model.

The original text supplies both the input and the answer. No human needs to label every sentence.

GPT’s causal objective supports left-to-right text generation; BERT’s masked objective learns bidirectional representations for classification, search, and extraction.

GPT

Causal / next-token prediction

Give only the prefix:

Thecompilerreportsan
↓

Predict the next token:

error

Sees: earlier tokens only.

BERT

Masked-token prediction

Hide one token:

Thecompilerreportsan[MASK]today
↓

Reconstruct the hidden token:

error

Sees: tokens on both sides.

In both methods, the original text contains the correct answer—so it supplies its own training target.
Evolution of AI

From explicit rules
to models that create.

Generative AI did not appear overnight. It is the latest change in where intelligence is encoded: rules → data → learned representations → broadly pre-trained models.

Each era solved a real problem, then exposed a limitation that drove the next technical shift.

Evolution of AI · 01 / 06 · 1950s–1970s

Symbolic AI:
intelligence as explicit rules.

The premise: describe reasoning with symbols, logic, search, and hand-written knowledge.

Representative systemsLogic Theorist (1956) and General Problem Solver (1957).
What it made possibleTransparent reasoning over explicitly represented facts and rules.
Why the field moved onIt became brittle when knowledge was incomplete, ambiguous, or expensive to encode.
Evolution of AI · 02 / 06 · 1980s

Expert systems:
capture a domain expert.

The system paired an if–then rule base with domain knowledge to make narrow decisions.

Representative systemsMYCIN for medical diagnosis and XCON/R1 for computer configuration.
What it made possiblePractical decision support in constrained, high-value domains.
Why the field moved onKnowledge acquisition and maintenance did not scale as conditions changed.
Evolution of AI · 03 / 06 · 1990s–2000s

Statistical ML:
learn from labelled data.

Instead of writing every rule, engineers trained a model to map features to a target.

Representative modelsHidden Markov Models, decision trees, logistic regression, and SVMs.
What it made possibleData-driven classification, ranking, and prediction.
Why the field moved onEach task still needed labelled data, engineered features, and a separate model.
Evolution of AI · 04 / 06 · 2010s

Deep learning:
learn the representations too.

Multi-layer neural networks began learning useful features directly from large datasets.

Representative modelsAlexNet for vision; LSTM encoder–decoder models for sequence-to-sequence learning.
What it made possibleMajor gains in image, speech, and language tasks without manual feature design.
Why the field moved onTraining demanded large data and compute; recurrent sequence processing remained serial.
Evolution of AI · 05 / 06 · 2017–2020

Transformers:
attention and pre-training at scale.

The Transformer removed recurrence, used attention, and made training much more parallelizable.

Representative modelsTransformer (2017), BERT (2018), and GPT-3 (2020).
What it made possibleBroad self-supervised pre-training created reusable language capabilities and few-shot prompting.
What remained difficultTraining cost, factual reliability, and aligning outputs with a user’s intent.
Evolution of AI · 06 / 06 · 2021–now

Generative foundation models:
create across tasks and modalities.

Broad-data models are instruction-tuned and increasingly work across language, images, audio, and code.

Representative modelsFLAN, InstructGPT, CLIP, and modern multimodal foundation models.
What it made possibleOne broadly trained base can support many generative tasks through adaptation and prompting.
Open issuesHallucination, bias, safety, privacy, grounding, and accountable use remain system-level concerns.
Choose bounded, measurable work

What tasks are
suitable?

A strong use case has a clear input, a useful output contract, and a way to judge quality.

GenerationSpecification → draft text, code, or design. Check with a rubric or human acceptance.
Transformation / extractionSource document → summary, rewrite, translation, or typed fields. Check fidelity, schema, and evidence.
Grounded Q&AQuestion + approved documents → cited answer. Check retrieval and faithfulness to sources.
The value filter

Where does GenAI
add value?

Use it when several outputs can be acceptable and creating or transforming content is the bottleneck.

Generate: drafts, code, tests, or design alternatives.
Transform: summaries, translations, reformatted documents.
Compose: a response from retrieved, approved evidence.

Use deterministic software instead

For exact calculations, fixed policy enforcement, stable rules, and authoritative lookups.

Beyond text

Multimodal Generative AI
connects modalities.

A multimodal model can accept, relate, or generate across text, images, audio, video, and structured data.

Text
Image
Audio
Video
→
Multimodal
foundation model
→
Text · code
Image · audio
Structured data
InspectionImage + text → structured inspection report.
MeetingsAudio → transcript → extracted actions.
Control boundaryEvery modality needs permission, validation, and audit controls.
Probabilistic input needs explicit controls

GenAI changes the
application architecture.

Natural-language / multimodal input
→
Probabilistic interpretation
→
Retrieval · rules · tools · validation
→
Audited result

It does not remove domain data models, permissions, business rules, testing, monitoring, or accountability.

Trust requires control boundaries

Fluent output is not
evidence of correctness.

Risk
Why it happens
Control
Hallucination
Plausible generation is not external truth.
Retrieval, citations, verification
Prompt injection
Untrusted text tries to alter instructions.
Separate data from instructions; allowlists
Data leakage
Sensitive information enters uncontrolled paths.
Classification, minimisation, access control
Unsafe action
Model proposes an action outside authority.
Server-side validation, approval gates
Quality drift
Model, prompt, or index changes.
Versioning and regression evaluation
Who retains authority?

Human-in-the-loop is an
architecture decision.

Candidate
output
→
Consequence
level
→
Low: validate + respond

Medium: review queue

High: domain authority decides

Define who reviews, what evidence they see, what they may override, and what gets logged for audit.

Continue the narrative

Next: Large
Language Models

We now zoom into the most widely used generative foundation models: how they represent, attend to, and generate language.

Open Chapter 02 →