Teaching a Reranker the Language of Security Tickets (+41% MRR@10)
A security operations team enhanced their RAG pipeline by fine-tuning a reranker model using existing ticket data. This approach, which leveraged implicit relevance judgments found in analyst close-notes, resulted in a 41% increase in mean reciprocal rank without changing the model architecture or embedding model. The team discovered that by analyzing how analysts referenced previous tickets, they could generate valuable training data for the reranker, improving its ability to ground LLM answers in relevant historical security tickets. AI
IMPACT Demonstrates a cost-effective method for improving RAG systems by leveraging implicit user feedback within existing data, potentially reducing reliance on expensive explicit labeling.