PulseAugur
EN
LIVE 13:56:04

New models correct Zipf's and Heaps' laws using hapax rate analysis

By PulseAugur Editorial · [1 sources] ·

This research paper introduces modifications to Zipf's and Heaps' laws by developing models for the proportion of words that appear only once (hapaxes). The study assumes a standard urn model for word token sampling and posits that the hapax rate is a function of text length. Four such functions—constant, cancelation, linear, and logistic—were examined, with the logistic model showing the best fit for a sample of English texts. The paper also discusses the need for more complex mixture models for larger text corpora. AI

IMPACT Provides theoretical linguistic insights potentially relevant to natural language processing model development.

RANK_REASON Academic paper detailing theoretical corrections to linguistic laws. [lever_c_demoted from research: ic=1 ai=0.4]

Read on arXiv cs.CL →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

New models correct Zipf's and Heaps' laws using hapax rate analysis

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
Academic paper detailing theoretical corrections to linguistic laws. [lever_c_demoted from research: ic=1 ai=0.4]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
paper, other
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
Standard
On-topic for AI-industry coverage; kept in the public index.
Story freshness
65 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

Full methodology in our editorial standards.

COVERAGE [1]

  1. arXiv cs.CL TIER_1 English(EN) · {\L}ukasz D\k{e}bowski ·

    Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models

    arXiv:2307.12896v5 Announce Type: replace Abstract: The article introduces corrections to Zipf's and Heaps' laws based on systematic models of the proportion of hapaxes, i.e., words that occur once. The derivation rests on two assumptions. The first one is the standard urn model …

RELATED ENTITIES

RELATED TOPICS