Task quality
Does the answer meet the task’s own standard? Measure accuracy, source fidelity, rubric score, or executable tests. Fluent language is not evidence of correctness.
Full Stack & AI · Day 04 · Chapter 04
Using
LLMs
A provider is not a single model. Choose a versioned model family for a specific task, budget, and risk level.
Chat interfaces are only one surface over model APIs, tools, and serving infrastructure.
Task requirementsquality · latency · cost · risk
↓
Efficiencyhigh-volume, simple work
Balancedgeneral-purpose work
Reasoninghard, multi-step work
↓
Evaluate and selectwith your own representative tasks
Provider ecosystems
Recognise the family
before
choosing a model.
OpenAIGPT flagship / mini / nano, reasoning and coding
variants, media and embedding models.
ClaudeOpus for highest capability, Sonnet for balance, Haiku
for speed and efficiency.
GeminiPro, Flash, Flash-Lite; plus Live, media, embedding,
and agent paths.
Open modelsLlama, Qwen, Gemma, Mistral, DeepSeek, gpt-oss and
task-specific checkpoints.
Product names and aliases move. A deployment must record the exact model ID or pinned snapshot.
Generative and reasoning tiers
More reasoning is not
always a
better choice.
All LLMs generate tokens. Reasoning-capable models usually spend more inference-time effort on difficult tasks—trading latency and cost for potential task improvement.
Efficiency
mini · nano · Haiku · Flash-Lite
High-volume routing, tagging, simple extraction, well-defined transforms.
→
Balanced
general-purpose GPT · Sonnet · Flash
Document work, structured drafting, ordinary coding, general chat.
→
Flagship / reasoning
Pro · Opus · reasoning / thinking variants
Difficult debugging, planning, constraint-heavy and multi-step tasks.
Model selection
Start with the least expensive model that passes the full threshold.
Use task evidence—not fluent demos or a generic leaderboard—to decide whether escalation is justified.
Efficiency tier
Does it meet source fidelity, schema, latency, and cost targets?
Does it meet source fidelity, schema, latency, and cost targets?
Balanced tier
Escalate if the smaller model fails a measurable task criterion.
Escalate if the smaller model fails a measurable task criterion.
Reasoning tier
Use when the additional inference effort improves hard cases.
Use when the additional inference effort improves hard cases.
High volume: begin with an efficiency tier and validate its
accepted-result rate.
Compare candidates fairly
One task harness.
Many measured runs.
Fixed test setnormal · edge · adversarial
→
Same promptsame schema · same tools
→
Candidate modelsspecific version IDs
→
Blind evaluationquality · reliability · cost · latency
Record model ID, date, prompt version, tool state, temperature, tokens, latency, cost, and rubric result. A benchmark score is not a workflow evaluation.
Hosted provider APIs
Same request.
Different SDK shapes.
Use pinned, currently available model IDs in code. Keep keys in environment variables or a secret manager.
OpenAI
from openai import OpenAI client = OpenAI() r = client.responses.create( model="<MODEL_ID>", input="Extract outcomes as JSON." ) print(r.output_text)
Claude
from anthropic import Anthropic
client = Anthropic()
r = client.messages.create(
model="<MODEL_ID>",
max_tokens=1024,
messages=[{"role":"user",
"content":"Extract outcomes as JSON."}]
)
print(r.content[0].text)
Gemini
from google import genai client = genai.Client() r = client.models.generate_content( model="<MODEL_ID>", contents="Extract outcomes as JSON." ) print(r.text)
What remains constant
Prompt contract, structured-output requirement, validation, retry policy, logging, and evaluation.
The API call is the smallest part of a reliable system.
Open-model stack
Weights, runtime, and serving
are
different layers.
Hugging FaceDiscover weights, inspect model cards and
licences, pin revisions, or use managed inference.
→
OllamaRun selected model tags locally and expose a local API
for experiments and prototypes.
→
vLLMEfficient server-side inference; often exposes an
OpenAI-compatible API for applications.
Hugging Face
from huggingface_hub import InferenceClient
client = InferenceClient(api_key="<HF_TOKEN>")
r = client.chat.completions.create(
model="<MODEL_ID>",
messages=[{"role":"user",
"content":"Extract outcomes as JSON."}]
)
Ollama local API
ollama run <MODEL_TAG>
curl http://localhost:11434/api/chat \
-d '{"model":"<MODEL_TAG>",
"messages":[{"role":"user",
"content":"Extract outcomes as JSON."}],
"stream":false}'
vLLM server
vllm serve <HUGGING_FACE_MODEL_ID> # client base URL: http://localhost:8000/v1
Check before use
Licence · model card · exact revision · quantisation · local/cloud data path · authentication · runtime version.
Request configuration
A prompt is only one part
of model
behaviour.
The model, request parameters, tools, and output contract together determine what the system can do.
Sampling: which plausible token is chosen?
These settings alter the candidate distribution at each generation step. Tune deliberately; many providers recommend changing one sampling control at a time.
TemperatureRescales candidate scores before sampling. Lower concentrates probability on the leading candidates; higher spreads probability across more plausible continuations.
Top-p (nucleus)Keeps the smallest candidate set whose combined probability reaches p. It removes the low-probability tail while adapting the set size at each step.
Top-kKeeps only the k highest-probability candidates. It is a fixed-size cap, unlike top-p’s probability-mass threshold.
SeedInitialises the sampler when the provider supports it. It helps controlled experiments, but is not a model-version or platform-stability guarantee.
Request controls: what work and output are allowed?
Max output tokensSets a ceiling on generated length and cost. Too small can truncate an otherwise correct answer.
Stop sequenceEnds generation when a chosen marker appears. It can delimit a completion, but does not validate that the result is complete or well formed.
Reasoning effortAllocates more inference-time work on models that expose it. Use it for difficult tasks only after measuring accuracy, latency, and token cost.
Schema and tool choiceDefine the response shape and permitted actions. The application must still validate JSON and authorise every tool action.
Illustrative configuration
config = {
"max_output_tokens": 800,
"temperature": 0.2,
"seed": 42,
"response_schema": OUTCOME_SCHEMA,
}
Evaluation scorecard
A usable model must satisfy
more than
quality.
Reliability
Does the workflow behave predictably across ordinary and edge cases? Check valid-schema rate, variance across runs, and whether it abstains or routes uncertainty correctly.
Operational fit
Can the system sustain real use? Measure p50/p95 latency, cost per accepted result, and throughput against the application’s budget—not an ideal demo.
Safety and governance
Does it stay within the data boundary and resist untrusted instructions? Test injection resilience, auditability, and the application’s controls around consequential actions.
User experience
Can intended users understand and use the result? Evaluate language quality, multilingual performance, accessibility, and the human-review burden the system creates.
Continue the narrative
Next: LLM Integration
We now turn a selected model into a production component: APIs, SDKs, prompt assembly, structured validation, retrieval, tools, and fallbacks.
Open Chapter 05 →