How Many Tools Should an LLM Agent See? A Chance-Corrected Answer
Researchers have developed a chance-corrected metric called Bits-over-Random (BoR) to evaluate the optimal number of tools an LLM agent should consider for a given query. This metric helps determine if success at a certain tool shortlist depth is better than random selection. Applying this principle through reinforcement learning, an agent learned to adapt its tool shortlist size per query, significantly reducing the number of tools presented while maintaining or improving coverage and LLM selection accuracy. AI
IMPACT Optimizes LLM agent efficiency by reducing unnecessary tool considerations, potentially improving response times and accuracy.