Bleu
PulseAugur coverage of Bleu — every cluster mentioning Bleu across labs, papers, and developer communities, ranked by signal.
- instance of ROUGE-L Score 90%
- instance of Comet 90%
- instance of PixelBank 90%
- competes with Comet 80%
- used by BERTScore: Evaluating text generation with BERT 70%
- used by Gotit.pub 70%
- used by alphaXiv 70%
- instance of Meteor 70%
- used by Meteor 70%
- used by CatalyzeX 70%
- authored Gotit.pub 70%
- instance of chrF++ 70%
8 day(s) with sentiment data
-
LLM evaluation tools: Choosing the right scorer for accuracy and nuance
Evaluating Large Language Model (LLM) applications requires a nuanced approach, moving beyond simplistic metrics like BLEU and ROUGE or relying solely on LLM judges. The key is to select the most appropriate and cost-ef…
-
LLMs evaluated for English-Arabic dialectal translation quality
A new research paper evaluates the performance of various large language models (LLMs) in translating between English and dialectal Arabic across 16 different dialects. The study utilized the MADAR dataset and employed …
-
New Arabic Datasets Boost NLP for Primary Science Education
Researchers have introduced ArSciKids, a new collection of Arabic datasets designed to advance natural language processing for primary education. The datasets include scientific experiment records and multiple-choice qu…
-
New method uses instruction-tuned LLMs to generate BPMN diagrams
Researchers have developed InstruBPM, a method for instruction-tuning open-weight language models to generate Business Process Model and Notation (BPMN) diagrams from natural language descriptions. This approach uses pa…
-
Survey details LLM agent evaluation for multi-turn conversations
A new survey published on arXiv details evaluation methods for large language model (LLM)-based agents in multi-turn conversations. The paper, structured using a PRISMA-inspired framework, systematically reviewed nearly…
-
New multilingual dataset enhances AI crisis warning generation
Researchers have developed CrisiLangs, a new multilingual dataset designed to improve crisis warning generation by language models. This resource extends the existing CrisiText dataset to include warning messages for cr…
-
New RAG evaluation protocol for breast lesion literature detailed
Researchers have developed an evidence-aware evaluation protocol for a retrieval-augmented generation (RAG) system focused on breast lesion management literature. The study utilized an 18-document corpus and an 18-query…
-
New TITAN framework enhances low-resource multilingual translation in LLMs
Researchers have developed a new framework called TITAN to understand and improve translation capabilities in multilingual large language models, particularly for low-resource languages. This framework uses causal inter…
-
New MeddeX system uses LLaMA-3 and Gemini Vision for medical report analysis
Researchers have developed MeddeX, a system designed to analyze and summarize complex medical reports. MeddeX integrates a fine-tuned LLaMA-3 model with retrieval-augmented generation and a Gemini Vision layer to provid…
-
LLMs show varied performance in English vs. Bengali summarization
A new study published on arXiv evaluates the performance of eight large language models (LLMs) on multilingual abstractive summarization, focusing on English and Bengali. The research found significant cross-lingual per…
-
New benchmark and model advance multilingual patent translation
Researchers have introduced PatMT2, a two-stage domain adaptation pipeline for multilingual patent translation, improving upon existing models like Hy-MT2-7B and Qwen3-32B. This adaptation process enhances translation q…
-
Small Language Models Framework for SUD Patient Dialogue Generation
Researchers have developed a framework using Small Language Models (SLMs) to generate dialogue for patients with substance use disorder (SUD). This approach addresses the limitations of larger models in clinical setting…
-
Offline speech translation system self-corrects on edge devices
Researchers have developed an offline speech-to-speech translation system that can run on a Jetson Nano edge device and self-correct its translations without needing retraining. The system utilizes a Whisper-tiny ASR mo…
-
LLM deployment best practices: Business needs first, model selection second
Developing and deploying a successful LLM solution requires a structured approach, prioritizing business needs over model selection. The process involves five key steps: understanding the business problem and defining b…
-
New framework evaluates LLM context understanding using knowledge graphs
Researchers have developed a new framework to evaluate the contextual understanding of large language models (LLMs) in question-answering tasks. This framework utilizes knowledge graphs and a novel metric called Semanti…
-
Hidden dates in LLM prompts skew evaluations, research finds · 2 sources tracked
A new research paper highlights a significant, previously overlooked factor impacting Large Language Model (LLM) evaluations: the automatic injection of the current date into system prompts. This hidden date variable ca…
-
First English-Syriac machine translation model developed using Bible corpus
Researchers have developed the first phrase-based Statistical Machine Translation (SMT) model for English-to-Syriac, addressing the challenge of translating an endangered language with complex orthography. The study cre…
-
New EviSI agent improves evaluation of simultaneous translation
Researchers have developed EviSI, a new evaluation agent designed for simultaneous speech-to-speech translation systems. Unlike traditional metrics like BLEU and COMET, EviSI incorporates Multidimensional Quality Metric…
-
New RepoExec benchmark reveals LLMs struggle with code dependency accuracy
A new benchmark called RepoExec, introduced in a paper from FPT Software AI Center, evaluates Large Language Models on code generation by considering both correctness and dependency invocation rate. The benchmark reveal…
-
New adaptive LLM evaluation method uses continuous scores with fewer items
Researchers have developed a new method for evaluating Large Language Models (LLMs) that adapts principles from Computerized Adaptive Testing (CAT) to continuous scoring metrics. This approach, detailed in a recent arXi…