Researchers have introduced AeroCopilotBench, a novel two-tier benchmark designed to evaluate Large Language Model (LLM) agents in aviation scenarios. This benchmark includes a virtual cockpit environment called the AeroCopilot Operational Environment (ACOE) and a set of 1,200 multiple-choice questions for knowledge assessment. The second tier focuses on procedural execution and safety compliance, featuring 73 emergency and abnormal tasks derived from Pilot's Operating Handbooks. Initial testing across 12 models revealed that while knowledge can be assessed, procedural execution remains a significant challenge, with the highest success rate reaching only 72.6%. Analysis of failures highlighted issues in procedural completeness, state feedback utilization, and long-horizon execution management. AI
IMPACT This benchmark could accelerate the development and safety validation of LLM agents for critical operational roles in aviation and other complex domains.
RANK_REASON The item describes a new academic benchmark and evaluation environment for LLM agents in a specific domain (aviation). [lever_c_demoted from research: ic=1 ai=1.0]
- AeroCopilotBench
- AeroCopilot Operational Environment
- alphaXiv
- CatalyzeX
- DagsHub
- Gotit.pub
- Hugging Face
- Large Language Model agents
- Pilot's Operating Handbooks
- ScienceCast
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
