Diagrams — Module FT04: Dataset Formats

Module: FT04 — Dataset Formats Diagram count: 5 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.

Diagram 1 — The Three Canonical Formats (side by side)

Type: Comparison Purpose: The single mental model for "what shape does my data arrive in." Three input formats, one normalized internal representation. Reading the diagram: Left to right. Three input shapes on the left all funnel through .map(to_messages) into the single messages column on the right, which is what every downstream tool consumes.

flowchart LR
  subgraph Inputs["THREE INPUT FORMATS"]
    direction TB
    F1["Raw messages\n[{role, content}]"]
    F2["Instruction / response\n(instruction + output)"]
    F3["ShareGPT\n[{from, value}]"]
  end
  Conv["to_messages()\nnormalize once"]
  Out["messages column\n[{role, content}, ...]\nTHE INTERNAL LINGUA FRANCA"]

  F1 --> Conv
  F2 --> Conv
  F3 --> Conv
  Conv --> Out

  style F1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style F2 fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style F3 fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Conv fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray:4 2,color:#5eead4
  style Out fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8

Diagram 2 — The apply_chat_template Flow

Type: Pipeline Purpose: The single right way to turn messages into model-ready text. The anti-pattern (hand-concatenation) is shown as a forbidden branch. Reading the diagram: Top-down. messages enters, the tokenizer's own Jinja template renders the bytes, the model sees the exact format it was instruction-tuned against. The hand-concat path is struck through because it produces wrong bytes.

flowchart TD
  Msgs["messages\n[{role, content}, ...]"]
  Tok["tokenizer\n(shipped WITH the model)"]
  Tpl["chat_template.jinja\n(publisher's exact format)"]
  Text["rendered string\nexact bytes the model expects"]
  IDs["input_ids\nsingle-token specials\n(no merging bugs)"]

  Hand["HAND-CONCATENATION\nf'<user>...' + q + ..."]
  HandBugs["token-boundary errors\nspecial tokens as text\nwrong role tokens"]
  Result["SILENT TRAINING FAILURE\nloss drops, model subtly wrong"]

  Msgs --> Tok
  Tok --> Tpl
  Tpl --> Text
  Text --> IDs
  IDs --> Model["MODEL"]

  Msgs -.->|"ANTI-PATTERN"| Hand
  Hand --> HandBugs
  HandBugs --> Result

  style Msgs fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Tok fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Tpl fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Text fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style IDs fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Model fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Hand fill:#14141f,stroke:#f08080,stroke-dasharray:4 2,color:#f08080
  style HandBugs fill:#14141f,stroke:#f08080,color:#f08080
  style Result fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080

Diagram 3 — SFT Data vs Preference Data

Type: Comparison / structure Purpose: Two training objectives, two data shapes. Mixing them is a bug. Reading the diagram: Left = SFT shape (one correct response, cross-entropy on the assistant turn). Right = preference shape (chosen vs rejected sharing the same prompt, contrastive loss). The defining property of preference data is at the bottom.

flowchart LR
  subgraph SFT["SFT DATA — one correct response"]
    direction TB
    SP["messages:\nsystem / user / assistant"]
    SL["loss on the assistant turn only\n(cross-entropy)"]
    ST["-> SFTTrainer (FT12)"]
  end
  subgraph PREF["PREFERENCE DATA — chosen vs rejected"]
    direction TB
    PP["prompt: [{user...}]\nchosen:   [{assistant...good}]\nrejected: [{assistant...bad}]"]
    PL["contrastive loss\n(chosen > rejected)"]
    PT["-> DPOTrainer / KTO / IPO (FT13)"]
  end

  Rule["DEFINING PROPERTY:\nchosen and rejected share the same prompt\nand differ only in the final assistant turn"]

  SFT --> Rule
  PREF --> Rule

  style SP fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style SL fill:#14141f,stroke:rgba(94,234,212,0.4),color:#9494a0
  style ST fill:#08080c,stroke:rgba(94,234,212,0.3),color:#5eead4
  style PP fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style PL fill:#14141f,stroke:rgba(94,234,212,0.4),color:#9494a0
  style PT fill:#08080c,stroke:rgba(94,234,212,0.3),color:#5eead4
  style Rule fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8

Diagram 4 — The Inspection Loop

Type: Cyclic process Purpose: The five-second check that catches every formatting bug. Do this once per dataset. Reading the diagram: Clockwise from the top. After apply_chat_template and tokenize, decode the input_ids back to text and read it. If you see any of the four bug signatures, fix and re-inspect. Only when the decoded text is clean do you train.

flowchart TD
  A["messages"]
  B["apply_chat_template\ntokenize=False"]
  C["tokenizer(text)\n-> input_ids"]
  D["tokenizer.decode(input_ids[0])\nTHE INSPECTION"]
  E{"Decoded text clean?"}
  Bugs["BUG SIGNATURES TO SCAN FOR:\n- missing EOS at end\n- specials rendered as text\n- trailing spaces\n- user turns in the loss"]
  Fix["Fix the source format\n(re-run conversion)"]
  Train["TRAIN\nthe data is verified"]

  A --> B
  B --> C
  C --> D
  D --> E
  E -->|"NO — something looks wrong"| Bugs
  Bugs --> Fix
  Fix --> B
  E -->|"YES — bytes match the template"| Train

  style A fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style B fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style C fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style D fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style E fill:#08080c,stroke:rgba(94,234,212,0.5),color:#5eead4
  style Bugs fill:#14141f,stroke:#f08080,color:#f08080
  style Fix fill:#14141f,stroke:#f08080,color:#f08080
  style Train fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa

Diagram 5 — The Bug Taxonomy

Type: Matrix / classification Purpose: The checklist of format bugs you will encounter, ranked by how silently they fail. Reading the diagram: Left column = the bug. Middle = the symptom (what you see at inference). Right = the root cause (where to look). Every row is caught by Diagram 4's inspection loop; none is caught by loss curves.

flowchart LR
  subgraph Bugs["THE FORMAT BUG TAXONOMY"]
    direction TB
    B1["Missing EOS"]
    B2["Wrong role tokens"]
    B3["Specials as text"]
    B4["Trailing spaces"]
    B5["Loss on user turns"]
    B6["Mixed formats"]
  end
  subgraph Symptom["SYMPTOM (at inference)"]
    direction TB
    S1["Model never stops"]
    S2["Can't follow instructions"]
    S3["Trains on literal text"]
    S4["Stopping drifts"]
    S5["Model echoes the prompt"]
    S6["Half dataset malformed"]
  end
  subgraph Cause["ROOT CAUSE"]
    direction TB
    C1["No closing im_end / eot_id"]
    C2["ChatML template on Llama-3 (or vice versa)"]
    C3["Hand-concat instead of template"]
    C4["Whitespace before closing token"]
    C5["labels not -100-masked"]
    C6["Didn't normalize before map()"]
  end

  B1 --> S1 --> C1
  B2 --> S2 --> C2
  B3 --> S3 --> C3
  B4 --> S4 --> C4
  B5 --> S5 --> C5
  B6 --> S6 --> C6

  Note["ALL CAUGHT BY:\ntokenizer.decode(input_ids[0])\nNONE CAUGHT BY:\nloss curves"]

  Bugs --> Note

  style B1 fill:#14141f,stroke:#f08080,color:#f08080
  style B2 fill:#14141f,stroke:#f08080,color:#f08080
  style B3 fill:#14141f,stroke:#f08080,color:#f08080
  style B4 fill:#14141f,stroke:#f08080,color:#f08080
  style B5 fill:#14141f,stroke:#f08080,color:#f08080
  style B6 fill:#14141f,stroke:#f08080,color:#f08080
  style S1 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style S2 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style S3 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style S4 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style S5 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style S6 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style C1 fill:#14141f,stroke:#9494a0,color:#9494a0
  style C2 fill:#14141f,stroke:#9494a0,color:#9494a0
  style C3 fill:#14141f,stroke:#9494a0,color:#9494a0
  style C4 fill:#14141f,stroke:#9494a0,color:#9494a0
  style C5 fill:#14141f,stroke:#9494a0,color:#9494a0
  style C6 fill:#14141f,stroke:#9494a0,color:#9494a0
  style Note fill:#08080c,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa

Validation notes

All five diagrams use the course design system colors: #14141f panel fill, #5eead4 accent for primary, #f08080 danger for bugs/anti-patterns, #82e0aa success for verified/clean, #9494a0 muted for secondary, #e4e4e8 for primary text.
Paste each into Mermaid Live Editor to render. All use stable Mermaid syntax (flowchart, subgraph) supported in current Mermaid (v10.4+).
For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.

# Diagrams — Module FT04: Dataset Formats **Module**: FT04 — Dataset Formats **Diagram count**: 5 **Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live). --- ## Diagram 1 — The Three Canonical Formats (side by side) **Type**: Comparison **Purpose**: The single mental model for "what shape does my data arrive in." Three input formats, one normalized internal representation. **Reading the diagram**: Left to right. Three input shapes on the left all funnel through `.map(to_messages)` into the single `messages` column on the right, which is what every downstream tool consumes. ```mermaid flowchart LR subgraph Inputs["THREE INPUT FORMATS"] direction TB F1["Raw messages\n[{role, content}]"] F2["Instruction / response\n(instruction + output)"] F3["ShareGPT\n[{from, value}]"] end Conv["to_messages()\nnormalize once"] Out["messages column\n[{role, content}, ...]\nTHE INTERNAL LINGUA FRANCA"] F1 --> Conv F2 --> Conv F3 --> Conv Conv --> Out style F1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style F2 fill:#14141f,stroke:#5eead4,color:#e4e4e8 style F3 fill:#14141f,stroke:#5eead4,color:#e4e4e8 style Conv fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray:4 2,color:#5eead4 style Out fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 ``` --- ## Diagram 2 — The apply_chat_template Flow **Type**: Pipeline **Purpose**: The single right way to turn messages into model-ready text. The anti-pattern (hand-concatenation) is shown as a forbidden branch. **Reading the diagram**: Top-down. `messages` enters, the tokenizer's own Jinja template renders the bytes, the model sees the exact format it was instruction-tuned against. The hand-concat path is struck through because it produces wrong bytes. ```mermaid flowchart TD Msgs["messages\n[{role, content}, ...]"] Tok["tokenizer\n(shipped WITH the model)"] Tpl["chat_template.jinja\n(publisher's exact format)"] Text["rendered string\nexact bytes the model expects"] IDs["input_ids\nsingle-token specials\n(no merging bugs)"] Hand["HAND-CONCATENATION\nf'<user>...' + q + ..."] HandBugs["token-boundary errors\nspecial tokens as text\nwrong role tokens"] Result["SILENT TRAINING FAILURE\nloss drops, model subtly wrong"] Msgs --> Tok Tok --> Tpl Tpl --> Text Text --> IDs IDs --> Model["MODEL"] Msgs -.->|"ANTI-PATTERN"| Hand Hand --> HandBugs HandBugs --> Result style Msgs fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Tok fill:#14141f,stroke:#5eead4,color:#e4e4e8 style Tpl fill:#14141f,stroke:#5eead4,color:#e4e4e8 style Text fill:#14141f,stroke:#5eead4,color:#e4e4e8 style IDs fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Model fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Hand fill:#14141f,stroke:#f08080,stroke-dasharray:4 2,color:#f08080 style HandBugs fill:#14141f,stroke:#f08080,color:#f08080 style Result fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080 ``` --- ## Diagram 3 — SFT Data vs Preference Data **Type**: Comparison / structure **Purpose**: Two training objectives, two data shapes. Mixing them is a bug. **Reading the diagram**: Left = SFT shape (one correct response, cross-entropy on the assistant turn). Right = preference shape (chosen vs rejected sharing the same prompt, contrastive loss). The defining property of preference data is at the bottom. ```mermaid flowchart LR subgraph SFT["SFT DATA — one correct response"] direction TB SP["messages:\nsystem / user / assistant"] SL["loss on the assistant turn only\n(cross-entropy)"] ST["-> SFTTrainer (FT12)"] end subgraph PREF["PREFERENCE DATA — chosen vs rejected"] direction TB PP["prompt: [{user...}]\nchosen: [{assistant...good}]\nrejected: [{assistant...bad}]"] PL["contrastive loss\n(chosen > rejected)"] PT["-> DPOTrainer / KTO / IPO (FT13)"] end Rule["DEFINING PROPERTY:\nchosen and rejected share the same prompt\nand differ only in the final assistant turn"] SFT --> Rule PREF --> Rule style SP fill:#14141f,stroke:#5eead4,color:#e4e4e8 style SL fill:#14141f,stroke:rgba(94,234,212,0.4),color:#9494a0 style ST fill:#08080c,stroke:rgba(94,234,212,0.3),color:#5eead4 style PP fill:#14141f,stroke:#5eead4,color:#e4e4e8 style PL fill:#14141f,stroke:rgba(94,234,212,0.4),color:#9494a0 style PT fill:#08080c,stroke:rgba(94,234,212,0.3),color:#5eead4 style Rule fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 ``` --- ## Diagram 4 — The Inspection Loop **Type**: Cyclic process **Purpose**: The five-second check that catches every formatting bug. Do this once per dataset. **Reading the diagram**: Clockwise from the top. After `apply_chat_template` and tokenize, decode the `input_ids` back to text and read it. If you see any of the four bug signatures, fix and re-inspect. Only when the decoded text is clean do you train. ```mermaid flowchart TD A["messages"] B["apply_chat_template\ntokenize=False"] C["tokenizer(text)\n-> input_ids"] D["tokenizer.decode(input_ids[0])\nTHE INSPECTION"] E{"Decoded text clean?"} Bugs["BUG SIGNATURES TO SCAN FOR:\n- missing EOS at end\n- specials rendered as text\n- trailing spaces\n- user turns in the loss"] Fix["Fix the source format\n(re-run conversion)"] Train["TRAIN\nthe data is verified"] A --> B B --> C C --> D D --> E E -->|"NO — something looks wrong"| Bugs Bugs --> Fix Fix --> B E -->|"YES — bytes match the template"| Train style A fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style B fill:#14141f,stroke:#5eead4,color:#e4e4e8 style C fill:#14141f,stroke:#5eead4,color:#e4e4e8 style D fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style E fill:#08080c,stroke:rgba(94,234,212,0.5),color:#5eead4 style Bugs fill:#14141f,stroke:#f08080,color:#f08080 style Fix fill:#14141f,stroke:#f08080,color:#f08080 style Train fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa ``` --- ## Diagram 5 — The Bug Taxonomy **Type**: Matrix / classification **Purpose**: The checklist of format bugs you will encounter, ranked by how silently they fail. **Reading the diagram**: Left column = the bug. Middle = the symptom (what you see at inference). Right = the root cause (where to look). Every row is caught by Diagram 4's inspection loop; none is caught by loss curves. ```mermaid flowchart LR subgraph Bugs["THE FORMAT BUG TAXONOMY"] direction TB B1["Missing EOS"] B2["Wrong role tokens"] B3["Specials as text"] B4["Trailing spaces"] B5["Loss on user turns"] B6["Mixed formats"] end subgraph Symptom["SYMPTOM (at inference)"] direction TB S1["Model never stops"] S2["Can't follow instructions"] S3["Trains on literal text"] S4["Stopping drifts"] S5["Model echoes the prompt"] S6["Half dataset malformed"] end subgraph Cause["ROOT CAUSE"] direction TB C1["No closing im_end / eot_id"] C2["ChatML template on Llama-3 (or vice versa)"] C3["Hand-concat instead of template"] C4["Whitespace before closing token"] C5["labels not -100-masked"] C6["Didn't normalize before map()"] end B1 --> S1 --> C1 B2 --> S2 --> C2 B3 --> S3 --> C3 B4 --> S4 --> C4 B5 --> S5 --> C5 B6 --> S6 --> C6 Note["ALL CAUGHT BY:\ntokenizer.decode(input_ids[0])\nNONE CAUGHT BY:\nloss curves"] Bugs --> Note style B1 fill:#14141f,stroke:#f08080,color:#f08080 style B2 fill:#14141f,stroke:#f08080,color:#f08080 style B3 fill:#14141f,stroke:#f08080,color:#f08080 style B4 fill:#14141f,stroke:#f08080,color:#f08080 style B5 fill:#14141f,stroke:#f08080,color:#f08080 style B6 fill:#14141f,stroke:#f08080,color:#f08080 style S1 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8 style S2 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8 style S3 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8 style S4 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8 style S5 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8 style S6 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8 style C1 fill:#14141f,stroke:#9494a0,color:#9494a0 style C2 fill:#14141f,stroke:#9494a0,color:#9494a0 style C3 fill:#14141f,stroke:#9494a0,color:#9494a0 style C4 fill:#14141f,stroke:#9494a0,color:#9494a0 style C5 fill:#14141f,stroke:#9494a0,color:#9494a0 style C6 fill:#14141f,stroke:#9494a0,color:#9494a0 style Note fill:#08080c,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa ``` --- ## Validation notes - All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `#f08080` danger for bugs/anti-patterns, `#82e0aa` success for verified/clean, `#9494a0` muted for secondary, `#e4e4e8` for primary text. - Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart`, `subgraph`) supported in current Mermaid (v10.4+). - For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.