Governance

Why is governance an architecture question?

Ask where AI risk lives and the honest answer is: in specific components, doing specific things, owned by specific people. Hallucination is a property of the model loop Module 1 described. Confidently citing a withdrawn policy is a failure of the retrieval pattern in Module 2. Fluent nonsense about enrolment numbers traces to the pipelines of Module 3. An email that should never have been sent is an agent boundary from Module 4, drawn wrongly or not at all. Governance that starts from the architecture produces controls that fit; governance that starts from a policy template produces a document nobody consults after signing.

This is also why governance cannot be delegated to the technology team alone. Every control in the map below has a technical half and an organisational half, and the organisational half, naming owners, agreeing definitions, deciding what a person must approve, is the half that determines whether the technical half means anything.

What does a complete AI architecture look like?

Drawn the way a solution architect would draw it: grouped boundaries, one box per component, and numbered flows you can walk in order. Every box below was built in an earlier module, and every box appears again in the risk exercise that follows. For the layered version, with risk, ownership, cost, and evaluation as toggleable overlays, see the anatomy of an AI system.

1. Pipelines copy and reshape data from source systems into the data platform (Module 3).
2. Documents are chunked, embedded, and loaded into the vector store, once per document and on every change (Module 2).
3. A person asks for something: a question, or an outcome.
4. The agent retrieves the passages relevant to the request (Module 2).
5. The language model reasons over what was retrieved and drafts the response or the proposed action (Module 1).
6. Anything consequential leaves the guardrails only through a human approver (Module 4).

Text description of this diagram

The diagram shows three grouped zones. "Your estate" holds source systems (records and documents) feeding a data platform (lake and warehouse). "AI platform", marked as sitting inside guardrails of permissions, budgets, and audit logs, holds a vector store of chunk embeddings, a language model for reasoning and drafting, and an agent runtime running plan, act, observe. "People" holds staff and customers, whose questions come in, and a human approver, who signs off actions. Six numbered flows connect them, walked in order in the dataflow list above.

Exercise: map the risks across the stack

Below is the architecture this course has built, component by component. Before you open each one, pause and name the first risk you would look for; then compare. The exercise is the point: this is the same walk you can do through your own organisation's AI systems, and it takes an afternoon, not a programme.

1. The model from Module 1

   What can go wrong. Hallucination presented as fact; bias inherited from training data; behaviour shifting when the supplier upgrades the model underneath you.

   What proportionate control looks like. A named owner for every AI-powered service. Evaluation runs before launch and after every model change, so a decline is measurable rather than anecdotal. Model choice reviewed per task, not defaulted to the largest.

   Revisit Module 1

2. Prompts and context from Module 1

   What can go wrong. Prompt injection: instructions hidden in the content a system reads, a document, an email, a web page, that hijack its behaviour. Also over-broad standing instructions, and sensitive material pasted into windows it should never enter.

   What proportionate control looks like. Treat everything entering the context window as untrusted input, screened at input and output gates (Module 4 teaches the checking pipeline). Injection testing before launch. System prompts under version control, reviewed like policy documents, because that is what they are.

   Revisit Module 1

3. Knowledge base and retrieval from Module 2

   What can go wrong. Stale or wrong documents cited with perfect confidence; retrieval surfacing material the asker was never entitled to see; coverage gaps where the system improvises instead of declining.

   What proportionate control looks like. The knowledge base is a product with an owner and review dates, not a folder. Retrieval respects the asker’s permissions on every query. Retrieval quality measured on real questions, with "not in the knowledge base" as a first-class answer.

   Revisit Module 2

4. Data pipelines from Module 3

   What can go wrong. Silent pipeline failure feeding stale data to fluent systems; definitions drifting between systems; numbers nobody can trace to a source when a regulator or auditor asks.

   What proportionate control looks like. Pipeline monitoring that alerts people, lineage that answers "where did this number come from", and a named owner per pipeline, because "nobody" is the most common root cause in incident reviews.

   Revisit Module 3

5. Agents and tools from Module 4

   What can go wrong. Actions beyond intent: wrong emails sent, records changed, money committed. Runaway loops consuming budget. Approval fatigue turning oversight into ceremony. No answer to "why did it do that?".

   What proportionate control looks like. Permissions denied by default. Approval gates by action class, tightest on the irreversible. Hard budgets on spend and loop count. Complete audit logs, kept as carefully as financial records.

   Revisit Module 4

6. People and process from every module

   What can go wrong. The accountability vacuum: "the AI did it" offered as an explanation. Shadow use of unapproved tools with organisational data. Skills gaps that turn oversight into rubber-stamping. Incidents with no route to report them.

   What proportionate control looks like. One named accountable person per AI system, senior enough to stop it. An acceptable-use policy staff can find and understand. Training that builds judgement, not compliance theatre. AI incidents handled through the same discipline as any other operational incident, ITIL-style if that is the house method.

What do ISO 42001 and the EU AI Act ask of you?

Two frameworks dominate the governance conversation, and both are more approachable at concept level than their reputations suggest.

ISO 42001 is a management system standard for AI, structurally the sibling of ISO 27001 for information security. It does not certify that your AI is safe or accurate. It certifies that you run AI deliberately: risks assessed and owned, decisions documented, impacts considered, performance reviewed, improvement continuous. If the risk map above felt tractable, you have already seen most of what the standard formalises. Certification is a signal to customers and regulators; the operating discipline is available to any organisation without an auditor in the building.

The EU AI Act is law rather than standard, and its organising idea is proportionality: obligations scale with the risk of the use case, not with the sophistication of the technology. A handful of uses are prohibited outright. High-risk uses, in hiring, credit, essential services, and similar territory, carry duties around documentation, human oversight, and accuracy. Most business uses land in the lighter tiers, where transparency is the main obligation. The practical consequence: classify your use cases before worrying about the technology, because the same model can sit in different tiers depending on what you point it at. UK organisations face a principles-based domestic approach through existing regulators, and the Act still reaches any system whose outputs touch people in the EU. None of this paragraph is legal advice; classification of real systems belongs with counsel.

Text description of this diagram

The diagram shows a pyramid of four tiers, narrowest at the top. Prohibited: a small set of uses banned outright. High risk: uses in territory such as hiring, credit, and essential services, carrying duties around oversight and documentation. Limited risk: transparency duties. Minimal risk, the widest tier at the base: where most business uses sit. A caption notes that duties attach to the use case, not the technology.

What about bias, and can AI decisions be explained?

Two words the risk map names and a governance conversation cannot leave undefined. Bias arrives twice: once from the model's training data, carrying the patterns of everything it read, and again from your own history, because a model trained on yesterday's decisions learns yesterday's habits. Neither arrival announces itself. The control is measurement: run the evaluation across the groups the decision affects and compare outcomes, because bias is found by looking, never by assurance. The EU AI Act's high-risk tier, hiring, credit, essential services, is where this stops being good practice and becomes duty.

Explainability needs honest expectation-setting. A large language model cannot explain itself mechanistically; the billions of weights from Module 1 do not resolve into reasons a human can audit. What a well-built system offers instead is practical explainability: provenance. Which passages the answer drew on (Module 2's citations), which tools were called and approved (Module 4's audit log), which prompt version was live, and who signed off the action. Regulation asks for documentation and oversight of exactly this shape, not for a neuron-level account. And where a decision demands reasons the architecture cannot give, the honest conclusion is the one this module keeps returning to: use deterministic rules for that decision, and let AI draft, retrieve, and summarise around it.

Text description of this diagram

The diagram shows a chain running downward from an answer or action to the evidence behind it: the passages it drew on (retrieval citations, Module 2), the tools called and approved (the audit log, Module 4), the prompt version that was live (under version control, reviewed like policy), and the person who signed off (named accountability, human approval). The closing note: regulation asks for a chain of this shape, not a neuron-level account.

Who is accountable?

The people, process, technology triad is worn smooth with use, and it survives because it keeps being true. Technology controls fail without process: an audit log nobody reviews is a storage cost. Process fails without people: an approval gate staffed by someone afraid to say no is decoration. The organising principle that holds the triad together is single named accountability, one person per AI system, senior enough to pause it, close enough to understand it. If naming that person is difficult, that difficulty is the finding: the system has authority without ownership, which is precisely the condition every incident report describes.

How do you know your AI is working?

You measure it, on your own questions, continuously. Evaluation is the governance discipline that turns every opinion in this course into evidence: a set of real questions with agreed good answers, run against the system on a schedule and after every change. Without it, "the assistant is doing well" means "nobody has complained yet", which is how systems degrade politely for months.

The discipline is the same everywhere in the architecture; only the questions change. For retrieval (Module 2), sample real user questions and check the right passages come back, because retrieval quality is the ceiling on answer quality. For models (Module 1), run the evaluation before launch and again when the supplier upgrades, so a change in behaviour is detected by you rather than reported by a customer. For agents (Module 4), evaluate end to end, because small errors compound across steps, and the evaluation results are precisely the evidence that justifies each step up the autonomy ladder. None of this needs exotic tooling to start: fifty representative questions, an owner, and a diary date beat a sophisticated framework nobody runs. ISO 42001's performance evaluation clause asks for exactly this, so the same question set doubles as audit evidence.

When is "no AI" the right architecture?

The final trade-off in this course is the largest one. Declining to deploy is a legitimate architectural decision, and mature organisations make it regularly: when the error cost of a fluent mistake exceeds the value of a thousand fluent successes, when the data foundations of Module 3 cannot yet support the ambition, when no one can be found to own the system, or when a deterministic process already does the job reliably for less. An organisation that can say no with reasons is also the only kind whose yes means anything. That judgement, applied case by case across the architecture you now understand, is what this course was built to support.

Next: Module 6, Buying AI: the whole argument applied to the decision most organisations face first, choosing and contracting a supplier with your eyes open.