LLMs evaluated by LLMs on Jira backlog analysis

A developer explored the effectiveness of using Large Language Models (LLMs) to grade other LLMs by comparing the performance of Claude Sonnet 4.5 and GPT-5.5 in analyzing Jira backlog tickets. The experiment involved two distinct pipelines: one using sentence embeddings and another using TF-IDF, with both LLMs processing outputs from both pipelines. A third LLM, Gemini 3.1 Pro, was used to score the twelve resulting outputs based on criteria such as strategic alignment and recommendation specificity, with the developer also performing a manual comparison. AI

IMPACT Provides insights into the comparative strengths and weaknesses of different LLMs for practical tasks like backlog management.

RANK_REASON Developer's comparative analysis of LLM performance on a specific task.

Read on dev.to — LLM tag →

• Apache ZooKeeper
• Claude Sonnet 4.5
• Gemini 2.5-Flash
• Gemini 3.1 Pro
• GPT-5.5
• Jira
• k-means clustering
• retrieval-augmented generation
• tf–idf

• product
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →