Project 06 / 07 / shipped

Triage

A small offline model that sorts support emails, says how sure it is, and sends only the unsure ones to a bigger model.

From the write-up

Triage reads a support email and returns its topic, urgency and account number as clean JSON. It is a 600-million-parameter model, small enough to run on one ordinary graphics card with no internet, trained to copy a much larger model's judgement. It went from worse than guessing to 87 percent on emails it had never seen, and it says how sure it is so the unsure ones can go to a bigger model.

Headline numbers / 2

Support emails given the right urgency, out of 1,200 it had never seen. The same model before training: 38.3%. Always guessing high: 42.4%.: 87.2%up from 38.3%; From the Triage README, headline table, row Priority.
Card, phone or social security numbers hidden in emails that the trained model repeated back. The untrained model leaked 16.: 0 of 50none leaked; From the Triage README, trust table, row Planted card, phone or SSN echoed in the output.

Decisions I made and things that broke

The confidence cutoff of 0.885 turned out costly, but moving it after seeing test results would mean tuning on the test, so it stayed.
The model ignored requests to state its own confidence, so confidence comes from scoring every allowed answer instead.
Training made it answer most off-topic text as a support email, and one injected line pushed 29 of 40 low-urgency emails to high.

What this proves / 5 skills

AI / ML

Python: The training and scoring code is Python.
LoRA fine-tuning: After fine-tuning, the small model gave 87.2% of unseen support emails the right urgency. Before training it managed 38.3%.
Knowledge distillation: A large hosted model labels the emails, and the small model learns to copy those labels.
Model evaluation: It sorted 90.0% of the same 1,200 emails into the right topic. Before training it managed 64.2%.
Schema-checked output: It returns topic, urgency and account number as clean JSON, offline, with no outside service.

The demo, recorded

Triage demo: a ticket slides in, confidence bars fill, and unsure ones peel off to the teacher — Four buttons replay recorded runs: an easy ticket, an ambiguous one that the cascade sends to the bigger model, an escalated one, and an off-task input.

Papers / 3

arXiv 2609.23742, Constrained decoding fixes structure but leaves a semantic gapForcing the output format fixed the untrained model's layout but not its answers, so the trained model runs without it.
arXiv 2609.25014, Not all 4-bit quantizers are equalEach compressed size of the model was tested for repeating account numbers it saw in training before the smallest safe one was chosen.
arXiv 2609.26550, JEV-as-a-JudgeThe model keeps the emails it is sure of and hands the rest to a bigger model.