New benchmark UNIKIE-BENCH evaluates large multimodal models for document information extraction

Researchers have introduced UNIKIE-BENCH, a new benchmark designed to systematically evaluate the performance of Large Multimodal Models (LMMs) in extracting key information from visual documents. The benchmark features two tracks: one for constrained-category KIE with predefined schemas and another for open-category KIE. Experiments using 15 state-of-the-art LMMs highlighted significant performance drops when dealing with varied schemas, long-tail information, and complex layouts, indicating ongoing challenges in accuracy and reasoning for LMMs in this domain. AI

影响 Provides a standardized evaluation framework for LMMs in document information extraction, highlighting current limitations.

排序理由 This is a research paper introducing a new benchmark for evaluating LMMs.

在 arXiv cs.CV 阅读 →

• LMMs
• UNIKIE-BENCH
• Zhenghao Liu
• Large Multimodal Models

• 论文
• 其他

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →