New benchmark suite evaluates LLMs for translating natural language to SHACL

Researchers have introduced NL2SHACL-Bench, a new benchmark suite designed to evaluate the translation of natural language requirements into SHACL shapes for validating RDF knowledge graphs. The benchmark was used to assess four large language models, revealing that while current models can generate syntactically correct SHACL, they struggle with producing semantically equivalent constraints for complex logical and structural patterns. This new suite aims to provide a meaningful basis for measuring progress in the NL2SHACL domain. AI

IMPACT This benchmark could accelerate the development of AI systems capable of understanding and generating formal constraints from natural language, improving data validation and knowledge graph management.

RANK_REASON The cluster describes a new academic paper introducing a benchmark suite for a specific AI task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• NL2SHACL
• NL2SHACL-Bench
• Resource Description Framework
• SHACL

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →