Researchers have introduced JFinBench, a new benchmark designed to evaluate large language models (LLMs) on Japanese financial tasks. This benchmark is continuously refreshable, using recent certification exam questions to mitigate data contamination issues common in static benchmarks. In evaluations, Claude Fable 5 and GPT-6 Astra demonstrated high performance, surpassing 93% accuracy, while Kimi K3.1 Pro, an open-weight model, lagged behind. The study also found that providing models with tools like terminal and web access improved performance, though internal knowledge remained a significant factor. AI
IMPACT Establishes a new, continuously refreshable benchmark for evaluating LLMs in specialized financial domains, particularly for Japanese markets.
RANK_REASON Academic paper introducing a new benchmark dataset and evaluation results. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- Claude Fable 5
- DagsHub
- Gotit.pub
- GPT -6 Astra
- Hugging Face
- JFinBench
- Kimi K3.1 Pro
- ScienceCast
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →