The most useful thing I've learned from shipping machine learning is when to argue against it.
That isn't skepticism about the technology. I've built transformer-based conversational systems and image classifiers that did exactly what they were supposed to do. It's a claim about selection: the projects that succeeded had a specific structural property, and the ones that struggled were missing it.
The property that matters
A model earns its place when the problem has high volume, tolerable error, and expensive human judgment. All three, not two.
Drop volume and you've built infrastructure to automate something that happens twice a week. Drop error tolerance and you need a confidence threshold so high the model rarely fires. Drop the cost of human judgment and you've spent months replacing something that was already cheap.
Support triage has all three: thousands of conversations, a wrong first response is recoverable, and human attention is the scarcest resource in the system. That's why it works.
The rule-based baseline you skipped
Before any model, write the dumb version. Regex, lookup table, a hundred lines of conditionals. It takes an afternoon.
def classify(ticket: str) -> str | None:
"""Deliberately naive baseline. If this gets you to 70%,
the remaining 30% is the actual machine learning problem."""
text = ticket.lower()
for label, terms in RULES.items():
if any(term in text for term in terms):
return label
return None # Unhandled -> human, and a labeled training example.Two things happen. Sometimes the baseline is good enough, and you've saved a quarter. More often it isn't — but now you know precisely which cases it fails on, which is the real specification for the model you're about to train. You also have a fallback for when the model is down, which you were going to need anyway.
Deployment is part of the model
A classifier sitting in a notebook at 96% accuracy is a research result. It becomes a product when something else can call it without knowing anything about machine learning.
That means a boring, stable HTTP contract that doesn't break when you retrain. It means returning calibrated confidence alongside the prediction, so consumers can set their own risk tolerance rather than inheriting yours. A graded signal lets a team adopt incrementally — auto-handle the confident cases, route the rest to a human — where a bare label forces an all-or-nothing bet.
I now treat the integration surface as part of the model's design, decided before training rather than after. It's the difference between a model that ships and one that gets admired in a review meeting.
The honest summary
Use a model where scale makes human judgment the bottleneck and imperfection is survivable. Everywhere else, the conditional statement is faster to build, easier to debug, and considerably easier to explain when something goes wrong at 2am.