{
  "module": "CAP1 — The Air-Gapped Domain Model",
  "course": "3 — LLM Fine-Tuning Masterclass",
  "version": "1.0.0",
  "duration_minutes": 45,
  "total_questions": 15,
  "bloom_distribution": {
    "target": "20% recall / 40% application / 40% analysis-design",
    "actual": { "recall": 3, "application": 6, "analysis": 6 }
  },
  "passing_score_percent": 70,
  "questions": [
    {
      "id": "Q01", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What is the deliverable of Capstone 1?",
      "options": [
        "A research paper comparing QLoRA to full fine-tuning on a medical dataset.",
        "A reproducible end-to-end pipeline (GitHub README asset): open-data base → HIPAA-safe data prep → QLoRA → eval → GGUF export → local serve, running on a single consumer GPU/Apple Silicon with zero telemetry.",
        "A deployed model accessible via a cloud API endpoint.",
        "A benchmark suite for evaluating open-data base models."
      ],
      "answer_index": 1,
      "rationale": "The deliverable is a reproducible pipeline shipped as a GitHub README asset, covering all seven phases, running fully local with zero telemetry. A cloud API deployment contradicts the air-gap requirement."
    },
    {
      "id": "Q02", "bloom": "recall", "type": "multiple_choice",
      "prompt": "Name the three success criteria for Capstone 1.",
      "options": [
        "Speed, cost, and accuracy of the fine-tuned model.",
        "Domain lift vs base, successful local serve, and reproducibility.",
        "Model size, quantization level, and inference latency.",
        "Training time, dataset size, and GPU utilization."
      ],
      "answer_index": 1,
      "rationale": "The three success criteria are: (1) domain lift — fine-tuned measurably outperforms base on a held-out domain eval; (2) successful local serve — GGUF loads via Ollama/llama.cpp with no telemetry; (3) reproducibility — clone → one command → your numbers within tolerance."
    },
    {
      "id": "Q03", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What is the passing threshold for the Capstone 1 evaluation rubric, and which dimension is a gate?",
      "options": [
        "10/15 with no gate; all dimensions are weighted equally.",
        "12/15 or higher with no dimension below 2, and HIPAA-safety is a gate (a 1 fails regardless of total).",
        "15/15 required; every dimension must be excellent.",
        "8/15 with HIPAA-safety as the only scored dimension."
      ],
      "answer_index": 1,
      "rationale": "Passing is 12/15+ with no dimension below 2. HIPAA-safety is a gate — a score of 1 fails the submission regardless of the total, because a model with unclear data provenance is not safe to hand to a clinician."
    },
    {
      "id": "Q04", "bloom": "application", "type": "multiple_choice",
      "prompt": "Your fine-tuned model shows +12 points domain lift but you did not run a general benchmark. Is the submission complete, and why?",
      "options": [
        "Yes — the lift is strong, which is the primary criterion.",
        "No — reporting only the lift is incomplete. The forgetting number is required to prove the fine-tune was disciplined, not desperate. A +12 lift with -10 on GSM8K would mean reasoning was traded away for domain style.",
        "Yes — general benchmarks are optional for domain-specific capstones.",
        "No — you must run three general benchmarks, not just one."
      ],
      "answer_index": 1,
      "rationale": "Lift without forgetting is incomplete. The forgetting number (base minus fine-tuned on a general benchmark) is the discipline check. Without it, the reviewer cannot tell whether the fine-tune traded away capability. A submission with only domain lift fails the 'forgetting control' dimension."
    },
    {
      "id": "Q05", "bloom": "application", "type": "multiple_choice",
      "prompt": "You want to use Llama 3.x as your base because it benchmarks higher than MiniCPM. Is this acceptable for Capstone 1?",
      "options": [
        "Yes — higher benchmark scores always justify the base choice.",
        "Only if you explicitly justify it. The default is open-data (MiniCPM/OLMo/Tulu) because you must be able to audit the training corpus. Llama 3.x is open-weights-only — acceptable as a fallback with justification, but an unjustified closed base fails the HIPAA-safety dimension.",
        "No — Llama 3.x is never permitted in Course 3.",
        "Yes — open-weights and open-data are equivalent for this purpose."
      ],
      "answer_index": 1,
      "rationale": "Open-data is the default because auditability is the foundation of the HIPAA-safety argument. Llama 3.x (open-weights-only) is a fallback requiring explicit justification. An unjustified closed base undermines the sensitive-domain claim."
    },
    {
      "id": "Q06", "bloom": "application", "type": "multiple_choice",
      "prompt": "Your GGUF loads and generates, but you served it without checking for outbound network calls. What is the status of the 'local serve' criterion?",
      "options": [
        "Passed — the model loads and generates, which is the requirement.",
        "Not yet passed — 'local serve' requires confirming no telemetry. You must run with the network off or tcpdump to verify zero outbound calls. Without this, the air-gap claim is unverified.",
        "Failed permanently — you must restart the entire pipeline.",
        "Passed conditionally, pending a faster GPU."
      ],
      "answer_index": 1,
      "rationale": "Successful local serve requires confirming no telemetry. The air-gap is a verified property, not an assumption. Run with the network off or capture packets — if it still serves with zero outbound calls, the criterion is met."
    },
    {
      "id": "Q07", "bloom": "application", "type": "multiple_choice",
      "prompt": "You trained a LoRA adapter and want to export to GGUF. What step must happen before quantization?",
      "options": [
        "Re-train the base model from scratch.",
        "Merge the adapter into the base (merge_and_unload) to produce a single model. GGUF export expects a merged model, not a separate adapter.",
        "Quantize the adapter separately, then combine.",
        "Nothing — export the adapter directly to GGUF."
      ],
      "answer_index": 1,
      "rationale": "merge_and_unload() folds the adapter (Layer 2) into the base (Layer 1), producing a standalone model ready for export. A common mistake is skipping the merge — GGUF export of a separate adapter fails."
    },
    {
      "id": "Q08", "bloom": "application", "type": "multiple_choice",
      "prompt": "Your held-out domain lift is +0.5 points. What does this indicate, and what should you do?",
      "options": [
        "The fine-tune succeeded; +0.5 is a valid positive lift.",
        "The lift is noise (+0.5 is trivial). Either the fine-tune failed to steer, or the held-out set is contaminated/memorized. Investigate: check if held-out was truly unseen, verify the training actually changed behavior, and reconsider the domain (the base may already do the task, or may not have the capability).",
        "You need a larger model.",
        "You should quantize more aggressively to improve the score."
      ],
      "answer_index": 1,
      "rationale": "A +0.5 lift is noise, not a pass. The pass bar is a non-trivial lift (+3 to +15). This indicates either a failed steer or a memorization problem (held-out contaminated). The diagnosis per FT00's three-outcome test: if the base already does the task reliably, fine-tuning won't move the needle; if it genuinely can't, fine-tuning won't help."
    },
    {
      "id": "Q09", "bloom": "application", "type": "multiple_choice",
      "prompt": "You served your model via Ollama and it responds to a domain prompt with the base's default behavior, not your fine-tuned style. What went wrong?",
      "options": [
        "The quantization was too aggressive; use Q8.",
        "You likely served the base GGUF, not your merged-and-quantized one. Verify the Modelfile points at YOUR .gguf, and confirm the served behavior matches the HF fine-tuned model on the same prompt.",
        "Ollama cannot serve fine-tuned models.",
        "The LoRA rank was too low."
      ],
      "answer_index": 1,
      "rationale": "A frequent failure: the GGUF that loads is the base's, not the merged fine-tune. Always verify the served model exhibits your steering (same format, same domain style). Run a domain prompt through both the served and HF models and compare."
    },
    {
      "id": "Q10", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "How does the held-out domain evaluation test the course thesis (fine-tuning steers, not teaches)?",
      "options": [
        "It doesn't — the held-out set only measures accuracy, not the steering mechanism.",
        "If fine-tuning were injecting knowledge, the held-out lift would be small (knowledge doesn't generalize from memorization). If it's steering behavior, the held-out lift is real but bounded — the behavior generalizes because it's a low-rank redirect of existing capability, not memorized content. Held-out lift proves steering; flat held-out proves memorization.",
        "It proves the model learned new facts from the training set.",
        "It tests whether the base model is large enough."
      ],
      "answer_index": 1,
      "rationale": "The held-out set is the steering-vs-knowledge test. Steering (low-rank behavior redirect) generalizes to held-out examples. Knowledge injection (memorization) does not — it appears to work on training examples but fails on held-out. A real held-out lift is evidence of steering; a flat held-out with high training-set accuracy is evidence of memorization."
    },
    {
      "id": "Q11", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why is the air-gap described as a property of the whole pipeline rather than a deployment feature?",
      "options": [
        "Because it only matters at deployment time.",
        "Because if ANY phase (data prep, training, eval, export, serve) touches the network, the air-gap claim is false. You cannot bolt it on at the end — every phase must be designed local-first. A training run that downloaded data from a vendor, even if served locally, is not air-gapped.",
        "Because the air-gap is configured in the serve stack's settings file.",
        "Because it depends on the operating system's firewall."
      ],
      "answer_index": 1,
      "rationale": "The air-gap is a pipeline property. Data prep must use local data; training runs on local hardware with cached models; eval is local; export is local; serve is local with verified zero outbound calls. If any phase touched the network, the model's provenance is not fully owner-controlled."
    },
    {
      "id": "Q12", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "A student's submission reports +14 domain lift and -1 forgetting but uses a closed-weights base (Llama 3.x) with no justification. The rubric scores: lift 3, forgetting 3, serve 3, reproducibility 3, HIPAA-safety 1. Total 13/15. Does it pass?",
      "options": [
        "Yes — 13/15 exceeds the 12/15 threshold.",
        "No — HIPAA-safety is a gate. A score of 1 fails the submission regardless of total. An unjustified closed base means the training corpus cannot be audited, undermining the sensitive-domain claim. The student must either switch to an open-data base with justification.",
        "Yes, but with a warning about the base choice.",
        "It depends on the reviewer's discretion."
      ],
      "answer_index": 1,
      "rationale": "HIPAA-safety is a gate. 13/15 exceeds the threshold, but a 1 in HIPAA-safety fails the submission. The auditability requirement is foundational for sensitive domains — a model whose training corpus cannot be verified is not safe to deploy in medical/legal/security contexts. The dimension is non-tradeable."
    },
    {
      "id": "Q13", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why does a submission with only a notebook (no pipeline, no pinned deps, no README numbers) fail, even if the results are good?",
      "options": [
        "Because notebooks are not professional tools.",
        "Because the capstone deliverable is the reproducible PIPELINE, not the result. A reviewer cannot clone a notebook and reproduce the numbers — unpinned deps drift, manual steps are undocumented, and the numbers aren't stated for comparison. The pipeline + pinned deps + README numbers convert a result into a portfolio asset.",
        "Because notebooks cannot run QLoRA.",
        "Because notebooks are slower than scripts."
      ],
      "answer_index": 1,
      "rationale": "The deliverable is the reproducible pipeline. A notebook is a result-exploration tool, not a deliverable. Reproducibility (criterion 3) requires pinned deps, one-command run, set seeds, and numbers in the README. Without these, the work is not verifiable by others — it fails the reproducibility dimension."
    },
    {
      "id": "Q14", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "How does Capstone 1's pipeline architecture reflect the Steering Stack from FT00?",
      "options": [
        "It doesn't — the capstone uses a different architecture.",
        "Each phase maps to a stack layer: Phase 1 = Layer 1 (Base), Phase 2 = substrate (the steering wheel), Phase 3 = Layers 2+3 (Adapter + Steer), Phase 5 = Layer 4 (Export), Phase 6 = Layer 4 serve. The capstone is the stack climbed end to end, with the swappability property visible in the merge step.",
        "It only covers Layer 3 (the Steer).",
        "It only covers Layer 5 (the Boundary)."
      ],
      "answer_index": 1,
      "rationale": "The capstone climbs the stack: Phase 1 chooses Layer 1 (base); Phase 2 prepares the substrate; Phase 3 trains Layers 2+3 (adapter + steer) and merges them; Phase 5 exports Layer 4; Phase 6 serves Layer 4. The merge step demonstrates the swappability property (adapter folds into base). The pipeline IS the stack, operationalized."
    },
    {
      "id": "Q15", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "What distinguishes a fine-tune that 'steered' from one that 'memorized', and how does the capstone's eval design detect the difference?",
      "options": [
        "Steering produces high training-set accuracy; memorization produces low training-set accuracy.",
        "Steering generalizes to held-out examples (real held-out lift); memorization appears to work on training examples but produces flat or negative held-out lift. The capstone detects this by requiring a HELD-OUT set never seen in training — if the lift vanishes on held-out, the fine-tune memorized, it did not steer. This is the course thesis made testable.",
        "Steering uses QLoRA; memorization uses full fine-tuning.",
        "There is no reliable way to detect the difference."
      ],
      "answer_index": 1,
      "rationale": "The held-out set is the steering-vs-memorization discriminator. A steering fine-tune (low-rank behavior redirect) generalizes — held-out examples benefit because the behavior, not the content, was learned. A memorizing fine-tune (overfit to training content) fails on held-out. The capstone's required held-out eval makes this distinction falsifiable."
    }
  ]
}