Capstone CAP1 — The Air-Gapped Domain Model

Course: Course 3 — LLM Fine-Tuning Masterclass Module: CAP1 — The Air-Gapped Domain Model Duration: 120 minutes Level: Senior Engineer and above Prerequisites: Pillars P00–P06 complete (FT00 through FT20). You have run QLoRA, evaluated on a held-out set, and served at least one GGUF through Ollama or llama.cpp. This capstone assembles the full pipeline end to end.

This is a capstone, not a lecture. The teaching is design guidance — the decisions you make at each phase of the pipeline determine whether the deliverable is reproducible, whether it generalizes, and whether it is genuinely air-gapped. The lab (artifact 07) IS the capstone.

Learning Objectives

Assemble a reproducible end-to-end pipeline — open-data base, HIPAA-safe data prep, QLoRA fine-tune, domain eval, GGUF export, local serve — that runs on a single consumer GPU or Apple Silicon with zero telemetry.
Defend the three success criteria: domain-lift versus the base, successful local serve, and reproducibility from a clean clone.
Apply the steering thesis to distinguish where fine-tuning earned its gains (behavior) from where it cannot (knowledge), and design the eval accordingly.
Produce a GitHub README asset: a pipeline someone else can clone, run, and reproduce your reported numbers — the portfolio artifact.

CAP1.1 — The Capstone Goal

The deliverable

A model you fine-tuned for a sensitive domain, served entirely locally, from a pipeline anyone can reproduce. The domain is yours to choose — medical, legal, or security — but the constraints are fixed:

Open-data base. You start from a model whose training corpus you can audit (MiniCPM, OLMo, Tulu, SmolLM3). This is non-negotiable for sensitive domains because you must be able to prove what the model saw. An open-weights-only base (Llama 3.x) is a fallback only when you can justify it; the default is auditable.
HIPAA-safe data prep. No real PHI. You use open datasets (FT05/FT06 — synthetic generation, dedup, decontamination) or de-identified public corpora. The data prep phase produces a dataset that is safe to train on and safe to publish the recipe for.
QLoRA fine-tune. The steer (FT08/FT11). You train an adapter, merge it, and document the hyperparameters that produced it.
Domain eval. A held-out domain set measuring lift over the base, plus a general benchmark (MMLU, GSM8K, or equivalent) measuring forgetting. You report both.
GGUF export. Quantize the merged model (FT19) to a deployable format.
Local serve. Run it through Ollama or llama.cpp (FT20), fully local, no network calls, no telemetry.

The pipeline runs end to end on a single consumer GPU (RTX 4090, 16–24GB) or Apple Silicon (M-series, via Unsloth or MLX). If you cannot run it locally, you have not completed the capstone.

What this proves

This capstone is the integration of Pillars P00 through P06. P00 gave you the base and the mental model. P01 gave you the data discipline (the steering wheel). P02 gave you QLoRA (the cheap, swappable steer). P03 gave you the eval mindset. P06 gave you quantization and serving. Here you chain them into one pipeline that produces a usable, local, auditable model.

The property this proves — and that no individual module could prove alone — is reproducibility under air-gap. A model you fine-tuned on a cloud GPU and serve through an API is a model someone else controls. A model you fine-tuned on your hardware and serve through Ollama on your hardware, with a pipeline you can hand to someone else, is a model you own. That is the thesis of P07 (sensitive domains) made operational: the air-gap is not a feature you add; it is a property of the whole pipeline.

CAP1.2 — The Success Criteria

A capstone without measurable success criteria is a hackathon project. Three criteria, each falsifiable.

Criterion 1 — Domain lift versus the base

Your fine-tuned model must measurably outperform the base on a held-out domain evaluation. "Measurably" means a number, with a methodology someone else can run. Concretely:

Domain held-out set. 100–500 examples the model never saw during training, representative of the target task. For medical: clinical Q&A or summarization. For legal: statute interpretation or contract clause extraction. For security: advisory summarization or exploit-chain reasoning.
Metric. Accuracy for Q&A; ROUGE/BERTScore for summarization; pass@1 for code-generation-adjacent tasks. Pick one and defend it.
Baseline. Run the same eval on the un-fine-tuned base, same prompts, same decoding. The lift is fine_tuned_score - base_score.
Pass bar. A positive, non-trivial lift (typically +3 to +15 points depending on task and metric). A lift of +0.3 points is noise; a lift of +0.0 is a failed fine-tune.

The lift must come from steering, not memorization. This is why the held-out set is held out — if your fine-tune only memorized the training set, the held-out score will not move. This is the steering thesis (FT00) made testable: if fine-tuning were injecting knowledge, the held-out lift would be large and the forgetting would be small. If it is steering behavior, the lift is real but bounded, and you measure the cost.

Criterion 2 — Successful local serve

The GGUF you exported must run through Ollama or llama.cpp on your local hardware at a usable latency. "Usable" is task-dependent but concretely:

It loads. ollama run your-model or llama-server -m your-model.gguf succeeds without OOM.
It generates. A prompt produces a coherent response at a token rate that is acceptable for the use case (typically >5 tok/s for interactive, >20 tok/s for batch).
It is the fine-tuned behavior. The served model exhibits the steering you trained — the same format, the same domain style — not the base's default behavior. (A common failure: serving the base GGUF instead of the merged-and-quantized one. Verify the served model is yours.)
No telemetry. Confirm the serve stack makes no outbound network calls. tcpdump or a firewall log on a clean run. The air-gap is a property, not a hope.

Criterion 3 — Reproducibility

Someone else (or you, six months from now, on a fresh machine) can clone your repo, run one command, and reproduce your reported numbers within a tolerance. This is the criterion that separates a capstone from a notebook. Concretely:

Pinned dependencies. A requirements.txt or pyproject.toml with exact versions, plus the CUDA/torch wheel version and the OS.
One-command run. A make pipeline or python run_pipeline.py that executes all phases in order, or a documented phase-by-phase sequence with exact commands.
Deterministic-ish training. Set seeds. Document that re-running produces lift within ±2 points of your reported number. Full determinism across GPU architectures is not achievable, but wild variance (±15 points on re-run) means your fine-tune is unstable — a finding worth reporting, not hiding.
The numbers are in the README. Base score, fine-tuned score, lift, forgetting score, serve latency. A README without numbers is a README without proof.

CAP1.3 — The Evaluation Rubric

The rubric is how a reviewer (or you, self-assessing) judges the submission. Five dimensions, each scored.

Dimension	Excellent (3)	Adequate (2)	Needs work (1)
Domain lift	Clear, defensible lift (>+5 pts) on a well-constructed held-out set; methodology documented	Positive lift but weak held-out set or unclear methodology	No measurable lift, or lift on a contaminated held-out set
Forgetting control	General benchmark reported; forgetting is small (<2 pts) or, if large, explicitly discussed with a mitigation	General benchmark reported but not discussed	No general benchmark; forgetting unknown
Local serve	GGUF loads, generates the fine-tuned behavior, no telemetry confirmed	GGUF loads and generates but no telemetry check, or base-vs-fine-tune not verified	Does not serve locally, or requires network
Reproducibility	One-command run, pinned deps, seeds set, numbers in README, re-runnable	Most phases documented but manual steps or unpinned deps	Not reproducible; notebook-only, no pinned versions
HIPAA-safety of data	Open/de-identified data only, data provenance documented, recipe publishable	Open data but provenance undocumented	Uses or approximates real PHI

A passing submission is 12/15 or higher with no dimension below 2. A submission with a 1 in HIPAA-safety fails regardless of the total — that dimension is a gate, not a trade-off.

The rubric is intentionally weighted toward reproducibility and HIPAA-safety. A model with a spectacular lift that is not reproducible or whose data provenance is unclear is not a capstone deliverable — it is a result. The capstone deliverable is the pipeline that produced it.

CAP1.4 — The Pipeline Architecture

Seven phases. Each phase has a module anchor (where you learned it) and a deliverable (what you produce).

+---------------------------------------------------------------+
|  PHASE 7 — REPRODUCIBILITY PACKAGE                             |  README + pinned deps + one-command run
+---------------------------------------------------------------+
|  PHASE 6 — LOCAL SERVE (Ollama / llama.cpp)                    |  running model, latency, telemetry-off proof
+---------------------------------------------------------------+
|  PHASE 5 — GGUF EXPORT (quantize)                              |  .gguf file at target precision
+---------------------------------------------------------------+
|  PHASE 4 — EVAL (domain held-out + general benchmark)          |  lift + forgetting numbers
+---------------------------------------------------------------+
|  PHASE 3 — QLoRA FINE-TUNE + MERGE                             |  merged model (adapter folded in)
+---------------------------------------------------------------+
|  PHASE 2 — DATA PREP (de-identified, from FT05/FT06)           |  train + held-out datasets
+---------------------------------------------------------------+
|  PHASE 1 — CHOOSE DOMAIN + OPEN-DATA BASE                      |  base model + domain rationale
+---------------------------------------------------------------+

Read bottom-up: you choose the base, prepare safe data, fine-tune, evaluate, export, serve, package. Each phase depends on the one below it — a mistake in data prep propagates to a failed lift in eval, which no amount of export polish can fix.

Phase 1 — Choose domain and base

The domain is where the steering thesis bites. Pick a domain where the base model already has the capability (it saw medical/legal/security text during pretraining) but not the behavior (it doesn't reliably format clinical differentials, or cite statutes, or structure advisory summaries). This is the three-outcome test from FT00, outcome 1: the behavior is achievable but unreliable. If you pick a domain the base genuinely does not know, no amount of fine-tuning will help — you need RAG or a different base.

The base must be open-data (FT02): MiniCPM5-1B or 3B for the fastest iteration; OLMo 2 or Tulu 3 for larger. Open-data means you can document what the model saw, which is the HIPAA-safety argument's foundation. Justify your choice in one paragraph: why this base, why this size, what its training corpus covers.

Phase 2 — HIPAA-safe data prep

No real PHI. Two acceptable sources:

Open datasets. PubMedQA, MedMCQA, LegalBench, or security advisory corpora — all public, all de-identified by construction.
Synthetic data (FT05). Generate domain examples with a teacher model (Distilabel, Magpie), then dedup and decontaminate (FT06). The synthetic path is how you scale a small open dataset into a fine-tuning set without touching real records.

The output is two datasets: train (what you fine-tune on) and held_out (what you evaluate on, never seen during training). Document the provenance of every example: source, transformations, de-identification steps. The recipe must be publishable — if you cannot publish the data prep, you cannot prove it was HIPAA-safe.

Phase 3 — QLoRA fine-tune and merge

The steer (FT08, FT11). Train a QLoRA adapter: 4-bit quantized base, LoRA on the attention projections, TRL's SFTTrainer. Document the hyperparameters: rank, alpha, target modules, learning rate, epochs, batch size, warmup. These are the knobs that determined your result; pinning them is part of reproducibility.

After training, merge the adapter into the base (merge_and_unload) to produce a single model you can quantize. The merge is the step that converts a swappable adapter (Layer 2) into a standalone model (Layers 1+2 fused) ready for export (Layer 4). A common mistake: skipping the merge and trying to quantize the adapter separately — GGUF export expects a merged model.

Phase 4 — Eval (domain held-out + general benchmark)

Two evaluations, both required:

Domain held-out. Run the fine-tuned model and the base on the held_out set from Phase 2. Compute the metric. Report lift.
General benchmark. Run both on a general benchmark (MMLU subset, GSM8K, or HellaSwag) to measure forgetting. The fine-tuned model will score slightly lower than the base on general tasks — this is expected (steering toward the domain steers slightly away from generality). The question is how much. A forgetting of <2 points is negligible; >5 points means your fine-tune was too aggressive (too many epochs, too high a learning rate) and you have traded away capability you should have kept.

Report both numbers side by side. A submission with only the domain lift is incomplete — the forgetting number is what tells the reviewer the fine-tune was disciplined, not desperate.

Phase 5 — GGUF export

Quantize the merged model to GGUF (FT19). The target precision is a trade-off: Q4_K_M is the default for local serve (good quality, small footprint); Q8_0 if you have the RAM and want minimal loss; Q3 if you are severely constrained and accept the quality hit. Document your choice and why.

The conversion uses llama.cpp's convert_hf_to_gguf.py then llama-quantize. The output is a single .gguf file. Verify it loads before declaring Phase 5 done — a GGUF that fails to load is a common failure mode from a chat-template mismatch (FT07).

Phase 6 — Local serve

Serve the GGUF through Ollama or llama.cpp (FT20). For Ollama: create a Modelfile pointing at your GGUF, ollama create, ollama run. For llama.cpp: llama-server -m your.gguf --port 8080. Verify:

The model responds to a domain prompt with the fine-tuned behavior (not the base's default).
The latency is usable (measure tok/s).
No outbound network calls (run with the network off, or tcpdump, and confirm it still serves). This is the air-gap proof.

Phase 7 — Reproducibility package

The README is the deliverable. It contains: the domain rationale, the base choice and justification, the data prep recipe (publishable), the training command and hyperparameters, the eval methodology and the numbers (base, fine-tuned, lift, forgetting), the serve instructions, and a one-command entrypoint (make pipeline or equivalent) that chains the phases. Pin every dependency. Set seeds. This is what makes the capstone a portfolio asset rather than a result.

CAP1.5 — The Solution Key (What a Passing Submission Looks Like)

A passing submission is a GitHub repo with a README that reads like a short technical report and a pipeline that runs. Concretely:

README structure. Title, one-paragraph domain rationale, the pipeline diagram (the seven phases), the results table (base / fine-tuned / lift / forgetting / latency), the reproducibility section (how to run), and the data-safety note (provenance and de-identification).
Results table. Something like: Base MMLU-medical 42.1, Fine-tuned 51.3, lift +9.2. Base GSM8K 38.5, Fine-tuned 37.9, forgetting -0.6. Serve latency 14 tok/s on M2 Pro, Q4_K_M, 4GB. These are illustrative — your numbers will differ — but the shape is what a reviewer expects.
One-command run. make pipeline clones the base, runs data prep, trains, evaluates, exports, and serves — or fails loudly at the first broken phase. A reviewer who runs this and gets your numbers within tolerance passes you.
Data safety. A section documenting that the training data is PubMedQA (open) plus 2000 synthetic examples generated via Distilabel from a teacher, deduped with MinHash, decontaminated against the held-out set. No PHI. Recipe publishable.

What fails

A notebook only. No pipeline, no pinned deps, no README with numbers. This is a result, not a deliverable.
Lift without forgetting. The submission reports +12 points domain lift but no general benchmark. The reviewer cannot tell if the fine-tune traded away half the model's reasoning to get there. Incomplete.
Serve that needs the network. The GGUF loads but Ollama phones home for something. The air-gap claim is false. Fail the HIPAA gate.
Unjustified closed base. The student used Llama 3.x "because it's better" without addressing the auditability requirement. The base choice is the foundation of the sensitive-domain argument; an unjustified closed base undermines it.

CAP1.6 — Why This Capstone Matters

This capstone is the proof that the course's thesis works end to end. Fine-tuning steers behavior; it does not teach knowledge — and here you demonstrate that steering on a real domain, measure the lift and the cost, and ship the result to a place where it runs without anyone else's infrastructure. The three success criteria are the three claims of the course operationalized: domain-lift proves steering works; local serve proves the export layer; reproducibility proves you understood the whole pipeline, not just one phase of it.

The air-gap is the point. A model you fine-tuned on a cloud GPU and serve through a vendor API is a model the vendor can observe, throttle, or revoke. A model you fine-tuned on your hardware and serve through Ollama, from a pipeline you can hand to a clinician, a lawyer, or an analyst, is a model that belongs to its domain. That is what P07 is about, and this capstone is where it stops being theory and becomes a thing you built.

The next capstone (CAP2) is the synthesis: the same pipeline skills, applied to an uncensored model wrapped in an eval'd harness — the bridge to the Harness Engineering courses. The skills transfer directly; the domain shifts from "air-gapped and domain-steered" to "uncensored and harness-bounded."

Anti-Patterns

Fine-tuning to inject knowledge

Training on a domain corpus expecting the model to "learn" the domain, then evaluating on memorized examples. The held-out set exists to catch this. If your lift vanishes on held-out, you memorized, you did not steer. Use RAG for knowledge; fine-tune for behavior.

Serving the base instead of the fine-tune

A frequent failure: the GGUF that loads is the base model's GGUF, not your merged-and-quantized one. Always verify the served behavior matches the fine-tuned behavior — run a domain prompt through the served model and through the HF model and compare.

Skipping the forgetting measurement

Reporting only the domain lift. A +15 lift with a -10 on GSM8K is a fine-tune that broke the model's reasoning to get domain style. The forgetting number is the discipline check; without it, the lift is uninterpretable.

Treating the air-gap as a deployment feature

Believing you can "add air-gap later." The air-gap is a property of the pipeline — if your data prep, training, or serve touches the network, the claim is false. Design for it from Phase 1.

Key Terms

Term	Definition
Air-gapped	A pipeline whose data prep, training, eval, and serve make no outbound network calls; the model never leaves the owner's hardware
Domain lift	The fine-tuned model's score minus the base's score on a held-out domain evaluation
Forgetting	The fine-tuned model's degradation on a general benchmark (MMLU, GSM8K) relative to the base — the cost of steering toward the domain
Open-data base	A base released with its training corpus (MiniCPM, OLMo, Tulu, SmolLM3) — auditable, the foundation of the HIPAA-safety argument
Held-out set	Domain examples never seen during training; the test of whether the fine-tune generalized or memorized
Reproducibility package	Pinned deps, set seeds, one-command run, numbers in the README — what converts a result into a deliverable
GGUF	The quantized model format for local serve via Ollama / llama.cpp
QLoRA	Quantized Low-Rank Adaptation — the fine-tuning method that fits a 7B model on a consumer GPU

Lab Exercise

See 07-lab-spec.md. The capstone build itself: seven phases, each with a deliverable and a verification step, culminating in a reproducible GitHub README asset.

References

Dettmers et al. (2023) — QLoRA. arXiv:2305.14314. The fine-tuning method this capstone uses.
Aghajanyan et al. (2020) — Intrinsic Dimensionality. arXiv:2007.07784. Why the lift you measure is steering, not knowledge.
NTIA (2024) — AI Open-Model Weights Report. The government basis for open-data models in sensitive domains.
Course 3, FT00–FT20 — The module stack this capstone integrates.
Course 3, FT21/FT22 — HIPAA and government air-gap, the "why" behind this capstone.

# Capstone CAP1 — The Air-Gapped Domain Model **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: CAP1 — The Air-Gapped Domain Model **Duration**: 120 minutes **Level**: Senior Engineer and above **Prerequisites**: Pillars P00–P06 complete (FT00 through FT20). You have run QLoRA, evaluated on a held-out set, and served at least one GGUF through Ollama or llama.cpp. This capstone assembles the full pipeline end to end. > *This is a capstone, not a lecture. The teaching is design guidance — the decisions you make at each phase of the pipeline determine whether the deliverable is reproducible, whether it generalizes, and whether it is genuinely air-gapped. The lab (artifact 07) IS the capstone.* --- ## Learning Objectives 1. Assemble a reproducible end-to-end pipeline — open-data base, HIPAA-safe data prep, QLoRA fine-tune, domain eval, GGUF export, local serve — that runs on a single consumer GPU or Apple Silicon with zero telemetry. 2. Defend the three success criteria: domain-lift versus the base, successful local serve, and reproducibility from a clean clone. 3. Apply the steering thesis to distinguish where fine-tuning earned its gains (behavior) from where it cannot (knowledge), and design the eval accordingly. 4. Produce a GitHub README asset: a pipeline someone else can clone, run, and reproduce your reported numbers — the portfolio artifact. --- # CAP1.1 — The Capstone Goal ## The deliverable A model you fine-tuned for a sensitive domain, served entirely locally, from a pipeline anyone can reproduce. The domain is yours to choose — medical, legal, or security — but the constraints are fixed: - **Open-data base.** You start from a model whose training corpus you can audit (MiniCPM, OLMo, Tulu, SmolLM3). This is non-negotiable for sensitive domains because you must be able to prove what the model saw. An open-weights-only base (Llama 3.x) is a fallback only when you can justify it; the default is auditable. - **HIPAA-safe data prep.** No real PHI. You use open datasets (FT05/FT06 — synthetic generation, dedup, decontamination) or de-identified public corpora. The data prep phase produces a dataset that is safe to train on and safe to publish the recipe for. - **QLoRA fine-tune.** The steer (FT08/FT11). You train an adapter, merge it, and document the hyperparameters that produced it. - **Domain eval.** A held-out domain set measuring lift over the base, plus a general benchmark (MMLU, GSM8K, or equivalent) measuring forgetting. You report both. - **GGUF export.** Quantize the merged model (FT19) to a deployable format. - **Local serve.** Run it through Ollama or llama.cpp (FT20), fully local, no network calls, no telemetry. The pipeline runs end to end on a single consumer GPU (RTX 4090, 16–24GB) or Apple Silicon (M-series, via Unsloth or MLX). If you cannot run it locally, you have not completed the capstone. ## What this proves This capstone is the integration of Pillars P00 through P06. P00 gave you the base and the mental model. P01 gave you the data discipline (the steering wheel). P02 gave you QLoRA (the cheap, swappable steer). P03 gave you the eval mindset. P06 gave you quantization and serving. Here you chain them into one pipeline that produces a usable, local, auditable model. The property this proves — and that no individual module could prove alone — is **reproducibility under air-gap.** A model you fine-tuned on a cloud GPU and serve through an API is a model someone else controls. A model you fine-tuned on your hardware and serve through Ollama on your hardware, with a pipeline you can hand to someone else, is a model you own. That is the thesis of P07 (sensitive domains) made operational: the air-gap is not a feature you add; it is a property of the whole pipeline. --- # CAP1.2 — The Success Criteria A capstone without measurable success criteria is a hackathon project. Three criteria, each falsifiable. ## Criterion 1 — Domain lift versus the base Your fine-tuned model must measurably outperform the base on a held-out domain evaluation. "Measurably" means a number, with a methodology someone else can run. Concretely: - **Domain held-out set.** 100–500 examples the model never saw during training, representative of the target task. For medical: clinical Q&A or summarization. For legal: statute interpretation or contract clause extraction. For security: advisory summarization or exploit-chain reasoning. - **Metric.** Accuracy for Q&A; ROUGE/BERTScore for summarization; pass@1 for code-generation-adjacent tasks. Pick one and defend it. - **Baseline.** Run the same eval on the un-fine-tuned base, same prompts, same decoding. The lift is `fine_tuned_score - base_score`. - **Pass bar.** A positive, non-trivial lift (typically +3 to +15 points depending on task and metric). A lift of +0.3 points is noise; a lift of +0.0 is a failed fine-tune. The lift must come from steering, not memorization. This is why the held-out set is held out — if your fine-tune only memorized the training set, the held-out score will not move. This is the steering thesis (FT00) made testable: if fine-tuning were injecting knowledge, the held-out lift would be large and the forgetting would be small. If it is steering behavior, the lift is real but bounded, and you measure the cost. ## Criterion 2 — Successful local serve The GGUF you exported must run through Ollama or llama.cpp on your local hardware at a usable latency. "Usable" is task-dependent but concretely: - **It loads.** `ollama run your-model` or `llama-server -m your-model.gguf` succeeds without OOM. - **It generates.** A prompt produces a coherent response at a token rate that is acceptable for the use case (typically >5 tok/s for interactive, >20 tok/s for batch). - **It is the fine-tuned behavior.** The served model exhibits the steering you trained — the same format, the same domain style — not the base's default behavior. (A common failure: serving the base GGUF instead of the merged-and-quantized one. Verify the served model is yours.) - **No telemetry.** Confirm the serve stack makes no outbound network calls. `tcpdump` or a firewall log on a clean run. The air-gap is a property, not a hope. ## Criterion 3 — Reproducibility Someone else (or you, six months from now, on a fresh machine) can clone your repo, run one command, and reproduce your reported numbers within a tolerance. This is the criterion that separates a capstone from a notebook. Concretely: - **Pinned dependencies.** A `requirements.txt` or `pyproject.toml` with exact versions, plus the CUDA/torch wheel version and the OS. - **One-command run.** A `make pipeline` or `python run_pipeline.py` that executes all phases in order, or a documented phase-by-phase sequence with exact commands. - **Deterministic-ish training.** Set seeds. Document that re-running produces lift within ±2 points of your reported number. Full determinism across GPU architectures is not achievable, but wild variance (±15 points on re-run) means your fine-tune is unstable — a finding worth reporting, not hiding. - **The numbers are in the README.** Base score, fine-tuned score, lift, forgetting score, serve latency. A README without numbers is a README without proof. --- # CAP1.3 — The Evaluation Rubric The rubric is how a reviewer (or you, self-assessing) judges the submission. Five dimensions, each scored. | Dimension | Excellent (3) | Adequate (2) | Needs work (1) | | --- | --- | --- | --- | | **Domain lift** | Clear, defensible lift (>+5 pts) on a well-constructed held-out set; methodology documented | Positive lift but weak held-out set or unclear methodology | No measurable lift, or lift on a contaminated held-out set | | **Forgetting control** | General benchmark reported; forgetting is small (<2 pts) or, if large, explicitly discussed with a mitigation | General benchmark reported but not discussed | No general benchmark; forgetting unknown | | **Local serve** | GGUF loads, generates the fine-tuned behavior, no telemetry confirmed | GGUF loads and generates but no telemetry check, or base-vs-fine-tune not verified | Does not serve locally, or requires network | | **Reproducibility** | One-command run, pinned deps, seeds set, numbers in README, re-runnable | Most phases documented but manual steps or unpinned deps | Not reproducible; notebook-only, no pinned versions | | **HIPAA-safety of data** | Open/de-identified data only, data provenance documented, recipe publishable | Open data but provenance undocumented | Uses or approximates real PHI | A passing submission is **12/15 or higher with no dimension below 2.** A submission with a 1 in HIPAA-safety fails regardless of the total — that dimension is a gate, not a trade-off. The rubric is intentionally weighted toward reproducibility and HIPAA-safety. A model with a spectacular lift that is not reproducible or whose data provenance is unclear is not a capstone deliverable — it is a result. The capstone deliverable is the pipeline that produced it. --- # CAP1.4 — The Pipeline Architecture Seven phases. Each phase has a module anchor (where you learned it) and a deliverable (what you produce). ``` +---------------------------------------------------------------+ | PHASE 7 — REPRODUCIBILITY PACKAGE | README + pinned deps + one-command run +---------------------------------------------------------------+ | PHASE 6 — LOCAL SERVE (Ollama / llama.cpp) | running model, latency, telemetry-off proof +---------------------------------------------------------------+ | PHASE 5 — GGUF EXPORT (quantize) | .gguf file at target precision +---------------------------------------------------------------+ | PHASE 4 — EVAL (domain held-out + general benchmark) | lift + forgetting numbers +---------------------------------------------------------------+ | PHASE 3 — QLoRA FINE-TUNE + MERGE | merged model (adapter folded in) +---------------------------------------------------------------+ | PHASE 2 — DATA PREP (de-identified, from FT05/FT06) | train + held-out datasets +---------------------------------------------------------------+ | PHASE 1 — CHOOSE DOMAIN + OPEN-DATA BASE | base model + domain rationale +---------------------------------------------------------------+ ``` Read bottom-up: you choose the base, prepare safe data, fine-tune, evaluate, export, serve, package. Each phase depends on the one below it — a mistake in data prep propagates to a failed lift in eval, which no amount of export polish can fix. ## Phase 1 — Choose domain and base The domain is where the steering thesis bites. Pick a domain where the base model already has the *capability* (it saw medical/legal/security text during pretraining) but not the *behavior* (it doesn't reliably format clinical differentials, or cite statutes, or structure advisory summaries). This is the three-outcome test from FT00, outcome 1: the behavior is achievable but unreliable. If you pick a domain the base genuinely does not know, no amount of fine-tuning will help — you need RAG or a different base. The base must be open-data (FT02): MiniCPM5-1B or 3B for the fastest iteration; OLMo 2 or Tulu 3 for larger. Open-data means you can document what the model saw, which is the HIPAA-safety argument's foundation. Justify your choice in one paragraph: why this base, why this size, what its training corpus covers. ## Phase 2 — HIPAA-safe data prep No real PHI. Two acceptable sources: - **Open datasets.** PubMedQA, MedMCQA, LegalBench, or security advisory corpora — all public, all de-identified by construction. - **Synthetic data (FT05).** Generate domain examples with a teacher model (Distilabel, Magpie), then dedup and decontaminate (FT06). The synthetic path is how you scale a small open dataset into a fine-tuning set without touching real records. The output is two datasets: `train` (what you fine-tune on) and `held_out` (what you evaluate on, never seen during training). Document the provenance of every example: source, transformations, de-identification steps. The recipe must be publishable — if you cannot publish the data prep, you cannot prove it was HIPAA-safe. ## Phase 3 — QLoRA fine-tune and merge The steer (FT08, FT11). Train a QLoRA adapter: 4-bit quantized base, LoRA on the attention projections, TRL's `SFTTrainer`. Document the hyperparameters: rank, alpha, target modules, learning rate, epochs, batch size, warmup. These are the knobs that determined your result; pinning them is part of reproducibility. After training, merge the adapter into the base (`merge_and_unload`) to produce a single model you can quantize. The merge is the step that converts a swappable adapter (Layer 2) into a standalone model (Layers 1+2 fused) ready for export (Layer 4). A common mistake: skipping the merge and trying to quantize the adapter separately — GGUF export expects a merged model. ## Phase 4 — Eval (domain held-out + general benchmark) Two evaluations, both required: - **Domain held-out.** Run the fine-tuned model and the base on the `held_out` set from Phase 2. Compute the metric. Report lift. - **General benchmark.** Run both on a general benchmark (MMLU subset, GSM8K, or HellaSwag) to measure forgetting. The fine-tuned model will score slightly lower than the base on general tasks — this is expected (steering toward the domain steers slightly away from generality). The question is how much. A forgetting of <2 points is negligible; >5 points means your fine-tune was too aggressive (too many epochs, too high a learning rate) and you have traded away capability you should have kept. Report both numbers side by side. A submission with only the domain lift is incomplete — the forgetting number is what tells the reviewer the fine-tune was disciplined, not desperate. ## Phase 5 — GGUF export Quantize the merged model to GGUF (FT19). The target precision is a trade-off: Q4_K_M is the default for local serve (good quality, small footprint); Q8_0 if you have the RAM and want minimal loss; Q3 if you are severely constrained and accept the quality hit. Document your choice and why. The conversion uses `llama.cpp`'s `convert_hf_to_gguf.py` then `llama-quantize`. The output is a single `.gguf` file. Verify it loads before declaring Phase 5 done — a GGUF that fails to load is a common failure mode from a chat-template mismatch (FT07). ## Phase 6 — Local serve Serve the GGUF through Ollama or llama.cpp (FT20). For Ollama: create a Modelfile pointing at your GGUF, `ollama create`, `ollama run`. For llama.cpp: `llama-server -m your.gguf --port 8080`. Verify: - The model responds to a domain prompt with the fine-tuned behavior (not the base's default). - The latency is usable (measure tok/s). - No outbound network calls (run with the network off, or `tcpdump`, and confirm it still serves). This is the air-gap proof. ## Phase 7 — Reproducibility package The README is the deliverable. It contains: the domain rationale, the base choice and justification, the data prep recipe (publishable), the training command and hyperparameters, the eval methodology and the numbers (base, fine-tuned, lift, forgetting), the serve instructions, and a one-command entrypoint (`make pipeline` or equivalent) that chains the phases. Pin every dependency. Set seeds. This is what makes the capstone a portfolio asset rather than a result. --- # CAP1.5 — The Solution Key (What a Passing Submission Looks Like) A passing submission is a GitHub repo with a README that reads like a short technical report and a pipeline that runs. Concretely: - **README structure.** Title, one-paragraph domain rationale, the pipeline diagram (the seven phases), the results table (base / fine-tuned / lift / forgetting / latency), the reproducibility section (how to run), and the data-safety note (provenance and de-identification). - **Results table.** Something like: Base MMLU-medical 42.1, Fine-tuned 51.3, lift +9.2. Base GSM8K 38.5, Fine-tuned 37.9, forgetting -0.6. Serve latency 14 tok/s on M2 Pro, Q4_K_M, 4GB. These are illustrative — your numbers will differ — but the shape is what a reviewer expects. - **One-command run.** `make pipeline` clones the base, runs data prep, trains, evaluates, exports, and serves — or fails loudly at the first broken phase. A reviewer who runs this and gets your numbers within tolerance passes you. - **Data safety.** A section documenting that the training data is PubMedQA (open) plus 2000 synthetic examples generated via Distilabel from a teacher, deduped with MinHash, decontaminated against the held-out set. No PHI. Recipe publishable. ## What fails - **A notebook only.** No pipeline, no pinned deps, no README with numbers. This is a result, not a deliverable. - **Lift without forgetting.** The submission reports +12 points domain lift but no general benchmark. The reviewer cannot tell if the fine-tune traded away half the model's reasoning to get there. Incomplete. - **Serve that needs the network.** The GGUF loads but Ollama phones home for something. The air-gap claim is false. Fail the HIPAA gate. - **Unjustified closed base.** The student used Llama 3.x "because it's better" without addressing the auditability requirement. The base choice is the foundation of the sensitive-domain argument; an unjustified closed base undermines it. --- # CAP1.6 — Why This Capstone Matters This capstone is the proof that the course's thesis works end to end. Fine-tuning steers behavior; it does not teach knowledge — and here you demonstrate that steering on a real domain, measure the lift and the cost, and ship the result to a place where it runs without anyone else's infrastructure. The three success criteria are the three claims of the course operationalized: domain-lift proves steering works; local serve proves the export layer; reproducibility proves you understood the whole pipeline, not just one phase of it. The air-gap is the point. A model you fine-tuned on a cloud GPU and serve through a vendor API is a model the vendor can observe, throttle, or revoke. A model you fine-tuned on your hardware and serve through Ollama, from a pipeline you can hand to a clinician, a lawyer, or an analyst, is a model that belongs to its domain. That is what P07 is about, and this capstone is where it stops being theory and becomes a thing you built. The next capstone (CAP2) is the synthesis: the same pipeline skills, applied to an uncensored model wrapped in an eval'd harness — the bridge to the Harness Engineering courses. The skills transfer directly; the domain shifts from "air-gapped and domain-steered" to "uncensored and harness-bounded." --- ## Anti-Patterns ### Fine-tuning to inject knowledge Training on a domain corpus expecting the model to "learn" the domain, then evaluating on memorized examples. The held-out set exists to catch this. If your lift vanishes on held-out, you memorized, you did not steer. Use RAG for knowledge; fine-tune for behavior. ### Serving the base instead of the fine-tune A frequent failure: the GGUF that loads is the base model's GGUF, not your merged-and-quantized one. Always verify the served behavior matches the fine-tuned behavior — run a domain prompt through the served model and through the HF model and compare. ### Skipping the forgetting measurement Reporting only the domain lift. A +15 lift with a -10 on GSM8K is a fine-tune that broke the model's reasoning to get domain style. The forgetting number is the discipline check; without it, the lift is uninterpretable. ### Treating the air-gap as a deployment feature Believing you can "add air-gap later." The air-gap is a property of the pipeline — if your data prep, training, or serve touches the network, the claim is false. Design for it from Phase 1. --- ## Key Terms | Term | Definition | | --- | --- | | **Air-gapped** | A pipeline whose data prep, training, eval, and serve make no outbound network calls; the model never leaves the owner's hardware | | **Domain lift** | The fine-tuned model's score minus the base's score on a held-out domain evaluation | | **Forgetting** | The fine-tuned model's degradation on a general benchmark (MMLU, GSM8K) relative to the base — the cost of steering toward the domain | | **Open-data base** | A base released with its training corpus (MiniCPM, OLMo, Tulu, SmolLM3) — auditable, the foundation of the HIPAA-safety argument | | **Held-out set** | Domain examples never seen during training; the test of whether the fine-tune generalized or memorized | | **Reproducibility package** | Pinned deps, set seeds, one-command run, numbers in the README — what converts a result into a deliverable | | **GGUF** | The quantized model format for local serve via Ollama / llama.cpp | | **QLoRA** | Quantized Low-Rank Adaptation — the fine-tuning method that fits a 7B model on a consumer GPU | --- ## Lab Exercise See `07-lab-spec.md`. The capstone build itself: seven phases, each with a deliverable and a verification step, culminating in a reproducible GitHub README asset. --- ## References 1. **Dettmers et al. (2023)** — *QLoRA*. arXiv:2305.14314. The fine-tuning method this capstone uses. 2. **Aghajanyan et al. (2020)** — *Intrinsic Dimensionality*. arXiv:2007.07784. Why the lift you measure is steering, not knowledge. 3. **NTIA (2024)** — *AI Open-Model Weights Report*. The government basis for open-data models in sensitive domains. 4. **Course 3, FT00–FT20** — The module stack this capstone integrates. 5. **Course 3, FT21/FT22** — HIPAA and government air-gap, the "why" behind this capstone.