Foundations

What is a model, really?

Strip away the product names and a large language model is a single, enormous mathematical function. During training it read a vast slice of human text and adjusted billions of internal numbers, called weights, until it became very good at one narrow task: given some text, predict which token comes next. That is the whole trick. There is no database of facts inside, no rulebook, no understanding in the human sense. There are patterns, compressed so densely that following them reproduces grammar, style, reasoning steps, and a great deal of knowledge, along with confident errors where the patterns run out.

Two consequences follow for decision-makers. First, a model is frozen at the moment training ends; it learns nothing from your conversations unless someone retrains it. Second, its answers are reconstructions from patterns, not lookups, which is why the same question can draw different answers and why fluency is no guide to accuracy.

One scope note: this course teaches the text case. Multimodal models apply the same prediction loop to images, audio, and video, and every architectural principle here carries over unchanged: the window, the costs, the checks, and the governance.

Text description of this diagram

The diagram shows a vertical flow with a loop. Your prompt is split into tokens, billable text fragments. The tokens enter the model, billions of frozen weights. The model produces probabilities for the next token, scoring every token it knows at once. One token is chosen with controlled randomness and appended, and the flow loops back into the model, repeating until the reply ends. The finished reply streams to your screen token by token. No lookup happens anywhere in the cycle; everything comes from the weights and the window.

What is a token, and why do you pay by it?

Models do not read words; they read tokens. A token is a short fragment of text, a common word, part of a longer word, or a piece of punctuation. English averages roughly three quarters of a word per token. Tokens are the unit of everything: models are priced per token in and per token out, context limits are measured in tokens, and speed is tokens per second. When a supplier's invoice or rate card looks opaque, translating it into "how much text, in and out" usually makes it legible.

What is inference?

Training and inference are two different events, and only one of them is your problem. Training is where the weights came from: one enormous computation, run once, in a data centre you will never see. Inference is the cycle in the diagram above, run from the beginning every time somebody asks a question.

Training was somebody else's capital cost. Inference is yours, charged again on every question, and it scales with the tokens involved:

• A long document pasted into a prompt costs more than a short one.
• A model that answers at length costs more than one that answers briefly.
• A wordy system instruction is paid for on every request, not once.
• Every query consumes computation, the thousandth as much as the first.

This is not how a bought machine behaves, where the money is settled up front and use afterwards is free. It is why later modules keep asking what a model actually needs to read before it answers, which is a cost question as much as a quality one.

What is a context window?

The context window is the model's working memory: the maximum number of tokens it can consider at once, covering your instructions, the conversation so far, any documents supplied, and the answer being written. Nothing outside the window exists for the model. It cannot remember last week's chat, and once a conversation grows past the limit, the earliest material falls away.

Context windows have grown enormously, and cost still keeps this interesting: the model processes everything in the window on every request, so pasting the same two hundred pages into each question multiplies spend and can bury the detail you care about. The craft, covered in Module 2, is putting the right material in front of the model rather than all of it.

Text description of this diagram

The diagram shows a fixed-size box labelled "the context window, measured in tokens". Inside it sit four stacked blocks: system instructions (standing rules set by the supplier and your team), the conversation so far, documents supplied with the question, and the answer as it grows, which occupies the same budget. Below the box, marked "outside the window, invisible to the model", sit two greyed blocks: last week's conversation, and everything else your organisation knows.

Prompting or fine-tuning: which lever, when?

There are two ways to change what a model does. Prompting changes what the model reads: instructions, examples, and supporting material placed in the context window. Fine-tuning changes what the model is: an additional round of training on your own examples that adjusts the weights themselves.

Prompting is cheap, instant, reversible, and auditable; it should be the default, and it resolves the large majority of business needs, especially combined with retrieval. Fine-tuning should be considered when the requirement is a deep habit rather than a fact: a house style applied across thousands of outputs, a specialised vocabulary, a consistent structured format.

Its costs are easy to underestimate:

• Curated training examples, in enough volume to teach the habit.
• Evaluation to prove the tuned model is better and has not lost general ability.
• A maintenance obligation, because the work must be repeated when the underlying model is upgraded. How much work depends on how the tuning was done.

Teach a model facts by showing it documents; change its behaviour, if prompting has been genuinely exhausted, by tuning it.

Rabbit hole What tuning actually changes, and why the repeat bill varies

A model's habits live in its weights, the billions of numbers set during training. There are two ways to change them, and they differ in what they touch.

Full fine-tuning adjusts those numbers directly. You finish holding a second copy of the entire model, several gigabytes of it, which is expensive to produce and expensive to keep. The other way leaves every original number untouched and trains a small separate set alongside them, which leans on the model's behaviour as it runs. That small set is called an adapter, and the usual method for producing one is called LoRA. The model is gigabytes; the adapter is megabytes.

That size difference is the whole of the practical story:

• An adapter is cheaper to produce, taking hours on modest hardware.
• Several can sit against one base model, one per task, and be swapped between. One model, several habits, rather than several models.
• An adapter is a file you hold, so no supplier can withdraw it.
• Moving onto a newer base model means retraining the adapter rather than repeating everything, which is why the maintenance bill above varies so much.

One question sits underneath all of this: whose model are you tuning? A habit trained onto a model you reach through a supplier's service lives inside that supplier's product lifecycle, and ends when they retire the model. A habit trained onto weights you hold cannot be withdrawn from you, though keeping pace with the frontier becomes your job rather than theirs. Module 6 covers where a model runs and what holding the weights involves.

Text description of this diagram

A decision flowchart. Start: you want to change what a model does. First question: is it facts the model lacks? If yes, show it the facts through prompting or retrieval. If no: is it tone, format, or a specialised habit? If not that either, neither lever fits; calculations and judgement need other tools. If it is a habit: has careful prompting truly been exhausted? If no, keep prompting, which is cheap, instant, and reversible. Only if yes, fine-tune, budgeting for training data, evaluation, and re-tuning when the underlying model is upgraded.

Try it: the journey of a prompt

The step-through below follows one short request from keyboard to reply. The token boundaries and probabilities are illustrative rather than taken from a live model, but the sequence is faithful: this is the loop running behind every AI answer you have ever read.

Written walkthrough of all six steps (text equivalent of the step-through)

1. You write a prompt. "Please summarise the attached report for Friday's board meeting." Behind your words, the system adds its own standing instructions and any supplied documents. All of it enters the context window together.
2. The text becomes tokens. The prompt is split into fragments: "Please", " summar", "ise", " the", " attached", " report", and so on. This is the unit you are billed by, and the unit the context window is measured in.
3. Tokens become numbers. Each token maps to an identifier the model was trained with. From here on, the model works entirely in numbers; it never sees letters.
4. The model predicts one next token. Given everything in the window, it produces a probability for every token it knows. For the first word of the reply, "The" might carry 41 per cent, "This" 23 per cent, "Here" 12 per cent. One is chosen, with controlled randomness, a setting called temperature, which is why identical questions can produce different answers.
5. Repeat, one token at a time. The chosen token joins the window and the model predicts again: "The" then " report" then " covers", each prediction seeing all the text so far. Long answers cost more because this loop runs once per token, and replies appear word by word because that is genuinely how they are made.
6. Tokens become text again. The identifiers are mapped back to characters and streamed to your screen. Every fact in the reply came from patterns in training or from what was placed in the window; nowhere in the loop did the model look anything up.

When is a bigger model the wrong answer?

Model capability comes in tiers, and the largest tier is the default choice far more often than it should be. Smaller models are faster, cheaper by an order of magnitude or more, and entirely adequate for classification, extraction, routing, and summarising at volume. The disciplined pattern is to define the task, build an evaluation for it, and choose the least capable model that passes, reserving frontier capability for work that demonstrably needs it. Done once per task, that exercise is what keeps frontier prices attached to judgement rather than to reformatting addresses, and it holds its value every time a supplier releases a new tier.

Next: Module 2, Retrieval-Augmented Generation: how systems put your documents in front of the model, and why that beats hoping the model remembers.