A new study published on arXiv evaluates various methods for using large language models (LLMs) to assist in academic paper peer review. The research compared five different paradigms across three input representations and three LLMs, using 232 stratified submissions from the International Conference on Learning Representations (ICLR) 2025. The findings indicate that direct absolute rating performs comparably to comparative ranking in predicting human scores, but is more efficient in terms of API calls. Inputting only the abstract proved more effective than using full-paper summaries, and the choice of LLM significantly impacted results. Notably, none of the evaluated LLM paradigms outperformed a baseline of always rejecting papers in terms of accuracy, though human scores correlated better with citations, with Group Rating being a partial exception. AI
IMPACT This research provides insights into the effectiveness and efficiency of using LLMs for academic peer review, potentially guiding future development of AI-assisted scholarly evaluation tools.
RANK_REASON Academic paper detailing research findings on LLM applications. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- DagsHub
- Direct Absolute Rating
- Elo
- Gotit.pub
- Group Rating
- Hugging Face
- Influence Flower
- International Conference on Learning Representations
- ScienceCast
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
