A new arXiv paper explores cost-aware abstention for Large Language Models (LLMs) in IT ticket classification. The research evaluates how abstention, or deferring a decision to a human, can reduce costs associated with incorrect automatic routing. By calibrating policies on explicit costs, the study found that even simpler models like TF-IDF with logistic regression significantly lowered costs and wrong routes compared to always routing tickets. While LLMs like GPT-4o mini and Haiku were tested, their performance in terms of calibration and discrimination varied, with some configurations deferring all tickets at certain probability thresholds. AI
IMPACT This research suggests methods to improve the cost-efficiency of LLM deployments in enterprise IT support by intelligently deferring tasks.
RANK_REASON Research paper published on arXiv detailing a novel approach to LLM application. [lever_c_demoted from research: ic=1 ai=1.0]
- arXiv
- DistilBERT
- GPT-4o mini
- Haiku
- IT ticket classification
- Large Language Models
- logistic regression model
- tf–idf
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
