Model card

Model details

• Three independent classification tasks over the same input text: Sentiment (Negative / Neutral / Positive), Support routing (Technical / Billing / Account / Escalation) and Severity (Severity 1 Urgent through Severity 4 Low). Each axis is trained separately: sentiment never sees the severity label, and vice versa, so emotional tone stays distinct from business impact.
• Per task, two interpretable scikit-learn models sharing one TF-IDF representation (unigrams + bigrams, 1,500 alphabetic features): Multinomial Naive Bayes and Logistic Regression. No neural networks, no LLM on any prediction path.
• Every prediction ships with its evidence: per-class probabilities and signed token contributions computed directly from model internals (no SHAP/LIME).
• Built as a TU Dublin ML certification deliverable, 2026.

Intended use

• Primary: education and demonstration — showing how interpretable text classification works, end to end, with honest evaluation.
• Plausible application shape: first-pass triage of incoming support cases, with a human owning the final routing decision.
• Out of scope: unsupervised production use on real customer traffic, any consequential decision about a real person, evaluation of the staff handling a case, and text domains far from infrastructure support (see limitations).

Training data

• Fully synthetic — 2,999 Red Hat support first comments, generated offline by a large language model in 120 small batches, each pinned to a distinct persona / industry / scenario slice. No real customer data anywhere.
• Independently blind-labelled by a second model that never saw the labels the generator intended, with inter-annotator agreement measured by Cohen's kappa. Severity is derived from the business impact stated in the comment.
• Live class distribution (train + test):
  Sentiment — Neutral 999 · Negative 1000 · Positive 1000
  Support routing — Technical 1743 · Escalation 200 · Billing 548 · Account 508
  Severity — Severity 1 (Urgent) 224 · Severity 2 (High) 620 · Severity 3 (Normal) 1312 · Severity 4 (Low) 843
• Validated before use. Generated text can look varied to a reader while staying trivially separable to a classifier. An ablation gate (scripts/leakage_probe.py) trains each task on the full comment, on the opening alone, and on the comment with its opening removed, and compares all three against a no-learning floor. It blocks the build if a label scores suspiciously well or fails to beat its own floor — a one-sided check would wave through a column of noise.
• Cosmetic case metadata (case IDs, customer and product names, dates) is synthetic and unused by the models — they read the comment text only.

Evaluation — live numbers

Scored on a stratified 25% held-out split the models never saw in training, with stratified 5-fold cross-validation on the training split and dummy-classifier floors for context.

Every figure above is reproducible from the shipped corpus: the random seed is pinned and so is the order in which cases are read for training. Routing is the strongest axis and severity the hardest — it asks the model to infer business impact from described consequences rather than from vocabulary alone. All three comfortably clear their no-learning floor, which is the only thing that makes an accuracy figure meaningful.

Known limitations

• Bag-of-words blindness. TF-IDF has no grammar: negation, sarcasm and word order are invisible. Documented failure case, measured on the deployed models: "Currently the cluster operations are on hold and we are running out of downtime" is filed as Neutral at 91% confidence by Logistic Regression. The phrase "running out of downtime" carries the urgency, and the model cannot read phrases — only the words it knows, weighted independently. The thin-evidence warning does not fire here, because 5 of the 13 words are in vocabulary. That guard catches verdicts resting on text the model cannot see; it does not catch verdicts where the model sees the words but misses the meaning. High confidence is not correctness.
• 1,500-term vocabulary ceiling. Words absent from the training corpus contribute nothing to a verdict. Out-of-domain text silently degrades to whatever few words are known — which is exactly what the thin-evidence warning detects.
• Synthetic-data ceiling. The corpus was written by a language model, so it carries that model's distributional biases and its idea of how a support ticket reads. Performance on real, human-written tickets is unknown and would need re-evaluation on real, consented data before any deployment.
• No human annotation. Labels come from a second model, not a human panel. Generator and labeller share a training lineage, so their agreement overstates what human annotators would reach.
• Class imbalance. Escalation is the smallest routing class; Naive Bayes recall on it is markedly weaker than Logistic Regression's (visible in the per-class table and confusion matrix). Macro F1 and κ expose this; accuracy alone hides it.
• Severity is the weakest axis. It clears its baseline by a clear margin but sits well below the other two, and Severity 2 (High) is the class most often confused with its neighbours.

Ethical considerations & recommendations

• No personal data was used; the corpus is entirely synthetic. Any adaptation to real support traffic requires consent review, PII handling, and bias evaluation on the real population before deployment.
• Sentiment classification of customers is sensitive: misreading a frustrated customer as Positive (see the failure case above) could delay help for the people who most need it. Keep a human in the loop for anything consequential.
• Prefer the transparency surfaces over blind trust: the evidence ledger, thin evidence warnings, and disagreement flags exist so users can audit every single prediction rather than accept a score.