LLMs struggle to generate efficient code for specialized architectures

Researchers have introduced CodegenBench, a new benchmark suite to evaluate the ability of large language models (LLMs) to generate efficient parallel code across diverse hardware architectures. The benchmark includes standard BLAS routines and specialized kernels for x86_64, Sunway, and Kunpeng platforms. Initial evaluations show that while LLMs perform well on common architectures, they struggle with domain-specific architectures lacking extensive public documentation and training data, indicating limitations in cross-platform generalization. AI

IMPACT Highlights limitations in LLM code generation for specialized hardware, suggesting a need for improved cross-platform generalization.

RANK_REASON The cluster contains an academic paper introducing a new benchmark for evaluating LLM code generation capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• BLAS
• CodegenBench
• LLMs
• Sunway
• x86_64

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →