Bleu
PulseAugur coverage of Bleu — every cluster mentioning Bleu across labs, papers, and developer communities, ranked by signal.
- instance of Comet 90%
- instance of PixelBank 90%
- competes with Comet 70%
- used by BERTScore: Evaluating text generation with BERT 70%
- used by Comet 70%
- used by CatalyzeX 70%
- used by alphaXiv 70%
- used by Gotit.pub 70%
- used by Meteor 70%
- instance of chrF++ 70%
- used by ROUGE L Score 60%
- competes with Meteor 50%
8 day(s) with sentiment data
-
First English-Syriac machine translation model developed using Bible corpus
Researchers have developed the first phrase-based Statistical Machine Translation (SMT) model for English-to-Syriac, addressing the challenge of translating an endangered language with complex orthography. The study cre…
-
New EviSI agent improves evaluation of simultaneous translation
Researchers have developed EviSI, a new evaluation agent designed for simultaneous speech-to-speech translation systems. Unlike traditional metrics like BLEU and COMET, EviSI incorporates Multidimensional Quality Metric…
-
New RepoExec benchmark reveals LLMs struggle with code dependency accuracy
A new benchmark called RepoExec, introduced in a paper from FPT Software AI Center, evaluates Large Language Models on code generation by considering both correctness and dependency invocation rate. The benchmark reveal…
-
New adaptive LLM evaluation method uses continuous scores with fewer items
Researchers have developed a new method for evaluating Large Language Models (LLMs) that adapts principles from Computerized Adaptive Testing (CAT) to continuous scoring metrics. This approach, detailed in a recent arXi…
-
AI advances sign language translation and video generation
Researchers are exploring advanced AI techniques for sign language translation and generation. One study investigates the impact of different T5 model scales and motion features on translating Indian Sign Language to te…
-
Human evaluation remains critical for LLM quality assessment
Human evaluation is crucial for assessing Large Language Models (LLMs) because automated metrics like BLEU scores often fail to capture nuanced qualities such as coherence, creativity, and factual accuracy. This approac…
-
New AlphaMWE Corpus Exposes LLM Translation Blind Spots for Multiword Expressions
Researchers have developed the AlphaMWE corpus to test the capabilities of large language models (LLMs) in machine translation, specifically focusing on Multiword Expressions (MWEs). The study evaluated 31 MT systems ac…
-
New LLM agents evaluate simultaneous interpreting, improving on BLEU and COMET
Researchers have developed new methods for evaluating simultaneous interpreting systems, addressing limitations of existing metrics like BLEU and COMET. One approach, detailed in an arXiv paper, uses a LoRA-adapted COME…
-
LLMs show promise in extracting software design decisions from code commits
A preliminary study explored the ability of four Large Language Models (LLMs) to extract Architectural Design Decisions (ADDs) from source code commits. Models including Gemini 3-Pro, DeepSeek-R1, Kimi K2, and Qwen3 wer…
-
SimpCue prompts boost multilingual text simplification modestly
Researchers have developed SimpCue, a novel prompting technique for multilingual text simplification using the Qwen3_8B language model. This method involves enriching prompts with explicit linguistic cues about sentence…
-
New LUX architecture enhances explainable endoscopic image captioning
Researchers have developed LUX, a novel graph-conditioned vision-language architecture designed for explainable endoscopic image captioning. This system addresses the limitations of current deep learning models by const…
-
LLM framework enhances similarity analysis for large-scale assessments
Researchers have developed a new framework called AISA, which utilizes Large Language Models (LLMs) to analyze incidental content similarity in large-scale assessments. This dual-dimensional approach, operationalizing s…
-
First machine translation systems developed for English-Pnar language pair
Researchers have developed the first machine translation systems for the English and Pnar language pair, an Austroasiatic language spoken by approximately 0.4 million people. Using articles from the Wyrta newspaper, a p…
-
LLM benchmark suites: Measuring progress with standardized metrics
Benchmark suites are essential for objectively measuring the progress of large language models (LLMs) by providing standardized testing frameworks. These suites aggregate various individual benchmarks to offer a holisti…
-
New SAraBERT model enhances Arabic document summarization with novel similarity metric
Researchers have developed SAraBERT, an improved version of the AraBERT model specifically designed for extractive summarization of Arabic documents. This new model incorporates inter-sentence transformer layers to enha…
-
New Denoising-Aware Inversion Method Exposes Privacy Risks in Noisy Text Embeddings
Researchers have developed a new method called Denoising-Aware Inversion (DAEI) to address privacy risks in text embeddings that have been protected by adding Gaussian noise. Standard inversion attacks struggle with the…
-
RAG vs Direct Context: LLM Test Reveals Retrieval Failures
A recent test compared Retrieval-Augmented Generation (RAG) with direct context answering using the BGE-M3 embedding model and Qwen3 LLM. The RAG approach, which retrieves relevant text chunks before answering, performe…
-
BLEU and ROUGE metrics explained for language model evaluation
BLEU and ROUGE are key metrics used to evaluate the performance of language models, particularly in tasks like machine translation and text summarization. BLEU focuses on precision of n-grams and includes a penalty for …
-
LLM-as-a-Judge: Using AI to Evaluate AI Output
The "LLM-as-a-Judge" technique utilizes a large language model to evaluate the output of other models, addressing the bottleneck of performance assessment in AI development. This method acts as a scalable and explainabl…
-
AI models win pun translation contest with novel multi-agent approach · 3 sources tracked
Researchers have developed a novel approach to translate puns from English to French, achieving first and second place in the CLEF JOKER 2025 Task 2 competition. The method combines large language models with specialize…