New TriCalRAG benchmark evaluates on-premise LLMs for AIOps root cause analysis

A new benchmark called TriCalRAG has been developed to evaluate on-premise Large Language Models (LLMs) for AIOps root cause analysis, addressing privacy and cost concerns associated with cloud-hosted models. The benchmark, tested on a single high-memory workstation GPU, compares two open-weight models, Qwen2.5-14B and Mistral-Small, using zero-shot, few-shot, and retrieval-augmented generation (RAG) prompting strategies. Results indicate that RAG significantly improves model accuracy and calibration, though the choice between models depends on whether peak performance or predictable behavior is prioritized. AI

IMPACT This benchmark could accelerate the adoption of on-premise LLMs for critical AIOps tasks by providing a standardized evaluation framework.

RANK_REASON The item is a research paper introducing a new benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• AIOps
• LLM
• Mistral Small
• Nvidia RTX Pro 6000 Blackwell Workstation Edition
• OpenStack
• Qwen2.5:14b
• retrieval-augmented generation
• Susil Kumar Mohanty
• Thunderbird
• TriCalRAG
• vLLM

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →