The Air-Gapped Domain Model
Capstone CAP1 · Course 3 — LLM Fine-Tuning Masterclass
120 minutes · 7 phases: choose base → data prep → QLoRA → eval → GGUF → serve → reproducibility
A model you fine-tuned for a sensitive domain, served entirely locally, from a pipeline anyone can reproduce.
Capstones · Tests P00–P06
The deliverable
A reproducible pipeline: open-data base → HIPAA-safe data prep → QLoRA fine-tune → eval → GGUF export → local serve.
Domain is your choice — medical, legal, or security. The constraints are fixed:
- Open-data base — auditable training corpus (MiniCPM, OLMo, Tulu, SmolLM3)
- HIPAA-safe data — no real PHI; open datasets + synthetic (FT05/FT06)
- Single consumer GPU or Apple Silicon — runs end to end, zero telemetry
If you cannot run it locally, you have not completed the capstone.
What this proves
The integration of Pillars P00 through P06 into one property:
Reproducibility under air-gap. A model you fine-tuned on your hardware and serve through Ollama, from a pipeline you can hand to someone else, is a model you own.
The air-gap is not a feature you add. It is a property of the whole pipeline — every phase local, no outbound calls.
The three success criteria
| Criterion | Measurement | Pass bar |
|---|
| Domain lift | held-out eval: fine_tuned − base | non-trivial positive (+3 to +15 pts) |
| Local serve | GGUF via Ollama/llama.cpp, no telemetry | loads, generates fine-tuned behavior, usable tok/s |
| Reproducibility | clone → one command → your numbers | pinned deps, seeds, re-run within tolerance |
Each criterion is falsifiable. A capstone without measurable criteria is a hackathon project.
Lift vs forgetting — why two evals
Lift (the gain)
Domain held-out: fine_tuned − base.
Proves steering worked. The number that justifies the fine-tune.
Forgetting (the cost)
General benchmark (MMLU/GSM8K): base − fine_tuned.
Proves the fine-tune was disciplined, not desperate.
Reporting only the lift is incomplete. A +15 lift with −10 on GSM8K is a fine-tune that broke reasoning to get domain style. The forgetting number is the discipline check.
The pipeline — 7 phases
1. CHOOSE DOMAIN + OPEN-DATA BASE · MiniCPM / OLMo / Tulu (FT02, FT03)
↑
2. DATA PREP (HIPAA-safe) · open + synthetic, dedup, decontaminate (FT05/FT06)
↑
3. QLoRA FINE-TUNE + MERGE · 4-bit base, LoRA, merge_and_unload (FT08/FT11)
↑
4. EVAL · domain held-out + general benchmark
↑
5. GGUF EXPORT · convert + quantize (FT19)
↑
6. LOCAL SERVE · Ollama / llama.cpp, no telemetry (FT20)
↑
7. REPRODUCIBILITY PACKAGE · README + pinned deps + one-command run
The air-gap is a property, not a feature
The air-gap is a property of the whole pipeline. If data prep, training, or serve touches the network, the claim is false.
Design for it from Phase 1:
- Data prep: local open datasets + local synthetic generation
- Training: local GPU/MPS, model cached locally
- Eval: local held-out + local benchmark runner
- Export: local GGUF conversion
- Serve: Ollama/llama.cpp, confirm no outbound calls
Proof: run with the network off, or tcpdump on a clean run. The air-gap is verified, not assumed.
The evaluation rubric
| Dimension | 3 (excellent) | 2 (adequate) | 1 (needs work) |
|---|
| Domain lift | >+5, defensible held-out | positive, weak methodology | no lift / contaminated |
| Forgetting control | reported, <2 pts or mitigated | reported, undiscussed | not measured |
| Local serve | loads, fine-tuned behavior, no telemetry | loads, no telemetry check | needs network |
| Reproducibility | one-command, pinned, numbers in README | mostly documented, manual steps | notebook-only |
| HIPAA-safety (GATE) | open/de-id, provenance documented | open, provenance unclear | real PHI |
Passing: 12/15+ with no dimension below 2. HIPAA-safety is a gate — a 1 fails the submission regardless of total.
What a passing submission looks like
A GitHub repo with a README that reads like a short technical report:
- Domain rationale + base choice, justified in one paragraph
- Results table: base / fine-tuned / lift / forgetting / latency
- One-command run:
make pipeline chains all 7 phases
- Data-safety note: provenance + de-identification, publishable recipe
Example numbers: Base MMLU-medical 42.1 → Fine-tuned 51.3 (lift +9.2). GSM8K 38.5 → 37.9 (forgetting −0.6). Q4_K_M, 14 tok/s on M2 Pro. A reviewer who runs your pipeline and gets these within tolerance passes you.
What fails
Notebook only. No pipeline, no pinned deps, no README with numbers. A result, not a deliverable.
Lift without forgetting. +12 domain but no general benchmark. Cannot tell if reasoning was traded away. Incomplete.
Serve that needs the network. GGUF loads but Ollama phones home. Air-gap claim is false. Fail the HIPAA gate.
Unjustified closed base. Llama 3.x "because it's better" without the auditability argument. Undermines the sensitive-domain claim.
Why this capstone matters
The proof that the course's thesis works end to end:
- Domain-lift proves steering works on a real domain
- Local serve proves the export layer — the model runs on your hardware
- Reproducibility proves you understood the whole pipeline, not one phase
A model fine-tuned on a cloud GPU and served via a vendor API is a model the vendor controls. A model fine-tuned on your hardware, served through Ollama, from a pipeline you can hand to a clinician — that is a model that belongs to its domain.
What you can now do
- Assemble a reproducible end-to-end fine-tuning pipeline that runs on consumer hardware.
- Defend the three success criteria — lift, local serve, reproducibility — with numbers.
- Distinguish steering gains from memorization using a held-out set.
- Ship a GitHub README asset that a reviewer can clone, run, and reproduce.
Next: CAP2 — The Calibrated Uncensored Agent · the synthesis capstone, the bridge to Course 1