A new research paper proposes that the distribution of stopwords and function words in language deviates from Zipf's law, instead fitting a Beta Rank Function (BRF). The study suggests that non-stopwords and non-function words also deviate from Zipf's law, but are better described by a quadratic function of log-token-count over log-rank. The researchers developed a stopword/function word/subset selection model based on word rank, which they validated with independent text collections and demonstrated analytically to explain the observed rank-frequency distributions. AI
RANK_REASON The cluster contains an academic paper detailing a new linguistic distribution model. [lever_c_demoted from research: ic=1 ai=0.4]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
