Scaling laws and fluctuations in the statistics of word frequencies

The frequency of an individual word varies significantly across different texts, meaning that its usage cannot be described alone by a single global frequency [38–40]. For example, consider the usage of the (topical) word 'network' in all articles published in the journal PlosOne. It has an overall rank ${{r}^{*}}=428$ and a global frequency, ${{F}_{{{r}^{*}}=428}}\approx 2.9\times {{10}^{-4}}$, see figure 2(a). The local frequency obtained from each article separately varies over more than one decade, see figure 2(b). Note that, although in this case the local rank-ordering differs from document to document, the index r still refers to the globally determined rank and is used as a unique label for each word.

Zoom In Zoom Out Reset image size

One popular approach to account for the heterogeneity in the usage of single words is topic models [41]. The basic idea is that the variability across different documents can be explained by the existence of (a smaller number of) topics. In the framework of a generative model, it assumes (i) that individual documents are composed of a mixture of topics (indexed by $t=1,..,T$), with each topic represented in an individual document by the probabilities ${{P}_{{\rm doc}}}({\rm topic}=t)$ and (ii) that the frequency of each word is topic dependent, i.e., ${{F}_{r}}({\rm topic}=t)$, which leads to a different effective frequency in each document, ${{F}_{r,{\rm doc}}}=\sum _{t=1}^{T}{{P}_{{\rm doc}}}(t){{F}_{r}}(t)$. One particularly popular variant of topic models is Latent Dirichlet Allocation (LDA) [42], which assumes that the topic composition ${{P}_{{\rm doc}}}({\rm topic})$ of each document is drawn from a Dirichlet distribution, ${{P}_{{\rm Dir}}}$, such that only a few topics contribute substantially to each document. Given a database of documents, LDA infers the topic-dependent frequencies, ${{F}_{r}}({\rm topic})$, from numerical maximization of the posterior likelihood of the generative model [43]. As an illustration, in figure 2(c), we show ${{F}_{{{r}^{*}}}}({\rm topic})$ obtained using LDA for the word 'network' in the PlosOne database. As expected from a meaningful topic model, we see that the conditional frequencies vary over many orders of magnitude, and that the global frequency ${{F}_{{{r}^{*}}}}$ is governed by few topics. The advantage of LDA is that, instead of measuring the distribution of frequencies of each individual word (or two-point distributions for assessing correlations) over different documents, it estimates the frequency of individual words for a finite (and small) number of topics. In combination with the generative model (e.g., drawing ${{P}_{{\rm doc}}}({\rm topic})$ from a Dirichlet distribution), this not only yields a more compact description of topicality by dramatically reducing the number of parameters, but also allows for an easy extrapolation to unseen texts from a small training sample [42].

In this section, we show how topicality can be included in the analysis of the vocabulary growth. The simplest approach is to consider again that the usage of each word is governed by Poisson processes, but this time to consider that frequencies are not fixed but are themselves random variables that vary across texts.

In this setting, the random variable representing the vocabulary size, N, for a text of length M can be written as

$$\begin{eqnarray}&&N(M)=\mathop{\sum }\limits_{r}I\left[ {{n}_{r}}(M,{{F}_{r}}) \right],\end{eqnarray} \tag{ 7 }$$

in which n_r is the integer number of times the word r occurs in a Poisson process of length M with frequency F_r and $I[x]$ is an indicator-type function, i.e., $I[x=0]=0$ and $I[x\geqslant 1]=1$. The calculation of the expectation value now consists of two parts: (i) the average over realizations i of the Poisson processes $n_{r}^{(i)}(M,F_{r}^{(j)})$ for a given realization j of the set of frequencies $F_{r}^{(j)}$ and (ii) the average overall possible realizations j of the sets of frequencies $F_{r}^{(j)}$ (which vary due to topicality). In this framework, expectation values correspond to quenched averages (denoted by subscript q)

$$\begin{eqnarray}&&{{\mathbb{E}}_{q}}\left[ N(M) \right]={{\left\langle N{{(M)}^{(i,j)}} \right\rangle }_{i,j}}=\mathop{\sum }\limits_{r}{{\left\langle I\left[ n_{r}^{(i)}(M,F_{r}^{(j)}) \right] \right\rangle }_{i,j}}=\mathop{\sum }\limits_{r}1-{{\langle {{{\rm e}}^{-MF_{r}^{(j)}}}\rangle }_{j}},\end{eqnarray} \tag{ 8 }$$

where we used

$$\begin{eqnarray}&&{{\left\langle I\left[ n_{r}^{(i)}(M,F_{r}^{(j)}) \right] \right\rangle }_{i}}=1-P({{n}_{r}}=0;M,F_{r}^{(j)})=1-{{{\rm e}}^{-MF_{r}^{(j)}}}.\end{eqnarray} \tag{ 9 }$$

The last equation corresponds to the probability of word r not occurring for a Poisson process of duration M with frequency $F_{r}^{(j)}$, as in equation (4). For simplicity, hereafter $\left\langle \ldots \right\rangle \equiv {{\left\langle \ldots \right\rangle }_{j}}$ (the average over realizations of sets of frequencies $F_{r}^{(j)}$).

Using the inequality between arithmetic and geometric mean

$$\begin{eqnarray}&&{{{\rm e}}^{\left\langle {\rm ln} x \right\rangle }}={{\left\langle x \right\rangle }_{{\rm geometric}}}\leqslant {{\left\langle x \right\rangle }_{{\rm arithmetic}}}=\left\langle {{{\rm e}}^{{\rm ln} x}} \right\rangle ,\end{eqnarray} \tag{ 10 }$$

we obtain that

$$\begin{eqnarray}&&{{\mathbb{E}}_{q}}\left[ N(M) \right]=\mathop{\sum }\limits_{r}1-\left\langle {{{\rm e}}^{-M{{F}_{r}}}} \right\rangle \leqslant \mathop{\sum }\limits_{r}1-{{{\rm e}}^{-M\left\langle {{F}_{r}} \right\rangle }}\equiv {{\mathbb{E}}_{a}}\left[ N(M) \right].\end{eqnarray} \tag{ 11 }$$

The right-hand side corresponds to the result of the Poisson null model (with fixed ${{F}_{r}}=\langle {{F}_{r}}\rangle$), see equation (4), and can be interpreted as an annealed average (denoted by subscript a). This implies that the heterogeneous dissemination of words across different texts leads to a reduction of the expected size of the vocabulary, in agreement with the first deviation of the Poisson null model reported in figures 1(b), (e) and (h).

For the quenched variance, we obtain (see appendix C)

$$\begin{eqnarray}&&{{\mathbb{V}}_{q}}\left[ N(M) \right]\equiv {{\mathbb{E}}_{q}}\left[ N{{(M)}^{2}} \right]-{{\mathbb{E}}_{q}}{{\left[ N(M) \right]}^{2}}\end{eqnarray} \tag{ 12 }$$

$$\begin{eqnarray}&&\qquad \qquad =\mathop{\sum }\limits_{r}\left\langle {{e}^{-M{{F}_{r}}}} \right\rangle -{{\left\langle {{e}^{-M{{F}_{r}}}} \right\rangle }^{2}}+\mathop{\sum }\limits_{r}\mathop{\sum }\limits_{r^{\prime} \ne r}{\rm Cov}[{{e}^{-M{{F}_{r}}}},{{e}^{-M{{F}_{r^{\prime} }}}}]\end{eqnarray} \tag{ 13 }$$

where ${\rm Cov}[{{{\rm e}}^{-M{{F}_{r}}}},{{{\rm e}}^{-M{{F}_{r^{\prime} }}}}]\equiv \langle {{{\rm e}}^{-M{{F}_{r}}}}{{{\rm e}}^{-M{{F}_{r^{\prime} }}}}\rangle -\langle {{{\rm e}}^{-M{{F}_{r}}}}\rangle \langle {{{\rm e}}^{-M{{F}_{r^{\prime} }}}}\rangle$. Comparing to the Poisson case in equation (5), we see that the quenched average yields an additional term containing the correlations of different words. In general, this term does not vanish and is responsible for the anomalous fluctuation scaling with $\beta =1$ observed in real text, explaining the second deviation from the Poisson null model reported in figures 1(c), (f) and (i).

In this section, we compute the general results from equation (8), (13) for particular ensembles of frequencies $F_{r}^{(j)}$ and compare them to the empirical results. In the absence of a generally accepted parametric formulation of such an ensemble, we propose two nonparametric approaches explained in the following.

In the first approach, we construct the ensemble $F_{r}^{(j)}$ directly from the collection of documents, i.e., the frequency $F_{r}^{(j)}$ corresponds to the frequency of word r in document j, such that

$$\begin{eqnarray}&&\left\langle {{{\rm e}}^{-M{{F}_{r}}}} \right\rangle =\frac{1}{D}\mathop{\sum }\limits_{j=1}^{D}{{{\rm e}}^{-MF_{r}^{(j)}}},\end{eqnarray} \tag{ 14 }$$

where D is the number of documents in the data, see figure 2(b).

In the second approach, we construct the ensemble from the LDA topic model [42], in which $F_{r}^{(j)}={{F}_{r}}({\rm topic}=j)$ corresponds to the frequency of word r conditional on the topic $j=1,\ldots ,T$, see figures 2(c) and (d). In this particular formulation, each document is assumed to consist of a composition of topics, ${{P}_{{\rm doc}}}({\rm topic})$, which is drawn from a Dirichlet distribution, such that we get for the quenched average

$$\begin{eqnarray}&&\left\langle {{{\rm e}}^{-M{{F}_{r}}}} \right\rangle =\int {\rm d}\theta {{P}_{{\rm Dir}}}(\theta |\alpha ){{{\rm e}}^{-M{{F}_{r}}(\theta )}},\end{eqnarray} \tag{ 15 }$$

in which $\theta =({{\theta }_{1}},\ldots ,{{\theta }_{T}})$ are the probabilities of each topic, ${{F}_{r}}(\theta )=\sum _{j=1}^{T}{{\theta }_{j}}{{F}_{r}}({\rm topic}=j)$, and the integral is over a T-dimensional Dirichlet-distribution ${{P}_{{\rm Dir}}}(\theta |\alpha )$ with concentration parameter α. We infer the ${{F}_{r}}({\rm topic})$ using Gensim [43] for LDA with T = 100 topics.

The results from both approaches are compared to the PlosOne database in figure 3. Figure 3(a) shows that both methods lead to a reduction in the mean number of different words. Whereas the direct ensemble, equation (14), almost perfectly matches the curve of the data, the LDA-ensemble, equation (15), still overestimates the mean number of different words in the data. This is not surprising because, due to the fewer number of topics (when compared to the number of documents), it constitutes a much more coarse-grained description than the direct ensemble. Additionally, the LDA-ensemble relies on a number of ad-hoc assumptions, e.g., the Dirichlet-distribution in equation (15) or the particular choice of parameters in the inference algorithm, which were not optimized here. More importantly, both methods correctly account for the anomalous fluctuation scaling with $\beta =1$ observed in the real data, see figure 3(b), and even yield a similar proportionality factor in the quantitative agreement with the data. The comparison of the individual contributions to the fluctuations, equation (13), shown in the inset of figure 3(b) shows that the anomalous fluctuation scaling is due to correlations in the co-occurrence of different words (contained in the term ${\rm Cov}[{{{\rm e}}^{-M{{F}_{r}}}},{{{\rm e}}^{-M{{F}_{r^{\prime} }}}}]$).

Zoom In Zoom Out Reset image size

In thermodynamic terms, Heaps' law (as other allometric scalings) implies that the vocabulary size is neither extensive nor intensive ($N(M)\lt N(2\;M)\lt 2N(M)$, also for $M\to \infty$). Although this can be seen as a direct consequence of Zipfʼs law, our results show that Heaps' law depends also sensitively on the fluctuations of the frequency of specific words across different documents. To illustrate this, consider the problem of doubling the size of a text of size M. This can be done either by simply extending the size of the same text up to size $2\;M$ (denoted by ${{M}^{\prime }}=2\cdot M$) or by concatenating another text of size M (denoted by ${{M}^{\prime }}=2\times M$). The Poisson model (fixed frequency or annealed average) predicts the same expected vocabulary for both procedures

$$\begin{eqnarray}&&{{\mathbb{E}}_{a}}[N(2\cdot M)]={{\mathbb{E}}_{a}}[N(2\times M)]=\mathop{\sum }\limits_{r}1-{{{\rm e}}^{-2\,M\left\langle {{F}_{r}} \right\rangle }}.\end{eqnarray} \tag{ 16 }$$

Taking fluctuations of individual frequencies across documents (quenched average) into account yields (see appendix D for details)

$$\begin{eqnarray}&&{{\mathbb{E}}_{q}}[N(2\cdot M)]=\mathop{\sum }\limits_{r}1-\left\langle {{{\rm e}}^{-2\,M{{F}_{r}}}} \right\rangle \ \ {\rm and}\ \ {{\mathbb{E}}_{q}}[N(2\times M)]=\mathop{\sum }\limits_{r}1-{{\left\langle {{{\rm e}}^{-M{{F}_{r}}}} \right\rangle }^{2}}.\end{eqnarray} \tag{ 17 }$$

Using equation (10) and the fact that $\langle {{x}^{2}}\rangle \geqslant {{\left\langle x \right\rangle }^{2}}$, we obtain the following general result

$$\begin{eqnarray}&&{{\mathbb{E}}_{q}}[N(2\cdot M)]\leqslant {{\mathbb{E}}_{q}}[N(2\times M)]\leqslant {{\mathbb{E}}_{a}}[N(2\times M)]={{\mathbb{E}}_{a}}[N(2\cdot M)].\end{eqnarray} \tag{ 18 }$$

This is consistent with the intuition that the concatenation of different texts (e.g., on different topics) leads to larger vocabulary than a single longer text. The preceding calculations remain true if the text is extended by a factor k (instead of 2), even for $k\to \infty$.

The fluctuations around the mean show a more interesting behavior, as revealed by repeating the preceding calculations for the variance. We consider the case of k texts each of length M, such that $M^{\prime} =k\times M$, and focus on the terms containing correlations between different words shown to be responsible for the anomalous fluctuation scaling (see appendix D for details):

$$\begin{eqnarray}&&{{\mathbb{V}}_{q}}[N(k\times M)]\sim \mathop{\sum }\limits_{r,r^{\prime} }{{\left\langle {{{\rm e}}^{-M{{F}_{r}}}}{{{\rm e}}^{-M{{F}_{r^{\prime} }}}} \right\rangle }^{k}}-{{\left\langle {{{\rm e}}^{-M{{F}_{r}}}} \right\rangle }^{k}}{{\left\langle {{{\rm e}}^{-M{{F}_{r^{\prime} }}}} \right\rangle }^{k}}.\end{eqnarray} \tag{ 19 }$$

The individual terms can be written as

$$\begin{eqnarray}&&\mathop{\sum }\limits_{r,r^{\prime} }{{\langle {{{\rm e}}^{-M{{F}_{r}}}}{{{\rm e}}^{-M{{F}_{r^{\prime} }}}}\rangle }^{k}}={{\langle {{[\mathop{\sum }\limits_{r}{{{\rm e}}^{-Mk\bar{F}_{r}^{(k)}}}]}^{2}}\rangle }_{{{j}_{1}},\ldots ,{{j}_{k}}}},\end{eqnarray} \tag{ 20 }$$

$$\begin{eqnarray}&&\mathop{\sum }\limits_{r,r^{\prime} }{{\langle {{{\rm e}}^{-M{{F}_{r}}}}\rangle }^{k}}{{\langle {{{\rm e}}^{-M{{F}_{r^{\prime} }}}}\rangle }^{k}}={{[{{\langle \mathop{\sum }\limits_{r}{{{\rm e}}^{-Mk\bar{F}_{r}^{(k)}}}\rangle }_{{{j}_{1}},\ldots ,{{j}_{k}}}}]}^{2}},\end{eqnarray} \tag{ 21 }$$

in which ${{\langle \cdot \rangle }_{{{j}_{1}},\ldots ,{{j}_{k}}}}$ denotes the averaging over the realizations $({{j}_{1}},\ldots ,{{j}_{k}})$ of frequencies $F_{r}^{({{j}_{i}})}$ in each single text $i=1,\ldots ,k$ and $\bar{F}_{r}^{(k)}=\frac{1}{k}\sum _{i=1}^{k}F_{r}^{({{j}_{i}})}$ is the k-sample average frequency based on the realizations $({{j}_{1}},\ldots ,{{j}_{k}})$. In the limit $k\to \infty$ : $\bar{F}_{r}^{(k)}\to \left\langle {{F}_{r}} \right\rangle$ such that

$$\begin{eqnarray}&&\mathop{\sum }\limits_{r,r^{\prime} }{{\left\langle {{{\rm e}}^{-M{{F}_{r}}}}{{{\rm e}}^{-M{{F}_{r^{\prime} }}}} \right\rangle }^{k}}-{{\left\langle {{{\rm e}}^{-M{{F}_{r}}}} \right\rangle }^{k}}{{\left\langle {{{\rm e}}^{-M{{F}_{r^{\prime} }}}} \right\rangle }^{k}}\to 0\end{eqnarray} \tag{ 22 }$$

for $k\to \infty$. This implies that, for $k\gg 1$, (adding many different texts) the fluctuations in the vocabulary across documents (and therefore the correlations between different words) vanish and normal fluctuation scaling ($\beta =1/2$) is recovered. This prediction can be tested in data. Starting from a collection of documents, we create a new collection by concatenating k randomly selected documents (each document is used once). We then compute for each concatenated document the number of distinct words N up to size M for increasing M, $\mathbb{E}[N(M)],$ and $\mathbb{V}[N(M)]$. We observe a transition of the exponent β in the fluctuation scaling, equation (3), from $\beta \approx 1\to \beta \approx 1/2$.

When measuring vocabulary richness, we want a measure that is robust to different text sizes. The traditional approach is to use Herdanʼs C, i.e., $C={\rm log} N/{\rm log} M$ [25–27]. Although quite effective for rough estimations, this approach has several problems. An obvious problem is that it does not incorporate any deviations from the original Heaps' law (e.g., the double scaling regime [19]). More seriously, it does not provide any estimation of the statistical significance or expected fluctuations of the measure. For instance, if two values are measured for different texts, one cannot determine whether one is significantly larger than the other. Our approach is to compare observations with the fluctuations expected from models in the spirit of section 3.2.

The computation of statistical significance requires an estimation of the probability of finding N different words in a text of length M, $P(N|M)$, which can be obtained from a given generative model (e.g., as presented in section 3). For a text with $({{N}^{*}},{{M}^{*}})$, we compute the percentile $P(N\gt {{N}^{*}}|{{M}^{*}})$, which allows for a ranking of texts with different sizes such that the smaller the percentile, the richer the vocabulary. An estimation of the significance of the difference in the vocabulary can then be obtained by comparison of the different percentile.

For the sake of simplicity, we illustrate this general approach by approximating $P(N|M)$ using a Gaussian distribution. In this case, the percentile are determined by the mean, $\mu (M)=\mathbb{E}[N(M)]$, and the variance, $\sigma (M)=\sqrt{\mathbb{V}[N(M)]}$, in terms of the z-score

$$\begin{eqnarray}&&{{z}_{(N,M)}}=\frac{N-\mu (M)}{\sigma (M)},\end{eqnarray} \tag{ 23 }$$

which shows how much the measured value (N, M) deviates from the expected value $\mu (M)$ in units of standard deviations (${{z}_{(N,M)}}$ follows a standard normal distribution: $z\mathop{\sim }\limits^{d}\mathcal{N}(0,1)$). If we consider our quantitative result on fluctuation scaling in the vocabulary in equation (6), i.e., $\sigma (M)\approx 0.1\mu (M)$, we can calculate the z-score of the observation (N, M) as

$$\begin{eqnarray}&&{{z}_{(N,M)}}\approx \frac{N-\mu (M)}{0.1\mu (M)}=10\left( \frac{N}{\mu (M)}-1 \right),\end{eqnarray} \tag{ 24 }$$

in which we need to include the expected vocabulary growth, $\mu (M)$, from a given generative model (e.g., Heaps' law with two scalings [19]). We can now: (i) for a single text (N, M), assign a value of lexical richness, the z-score ${{z}_{(N,M)}}$, considering deviations from the pure Heaps' law that should be included in $\mu (M)$; (ii) given two texts $({{N}_{1}},{{M}_{1}})$ and $({{N}_{2}},{{M}_{2}})$, compare directly the respective z-scores ${{z}_{({{N}_{1}},{{M}_{1}})}}$ and ${{z}_{({{N}_{2}},{{M}_{2}})}}$ to assess which text has a higher lexical richness independent of the difference in the textlengths; and (iii) estimate the statistical significance of the difference in vocabulary by considering $\Delta z:={{z}_{({{N}_{1}},{{M}_{1}})}}-{{z}_{({{N}_{2}},{{M}_{2}})}}$, which is distributed according to $\Delta z\mathop{\sim }\limits^{d}\mathcal{N}(0,2)$ because $z\mathop{\sim }\limits^{d}\mathcal{N}(0,1)$. Point (iii) implies that the difference in the vocabulary richness of two texts is statistically significant on a 95%-confidence level if $|\Delta z|\gt 2.77$, i.e., in this case there is at most a 5% chance that the observed difference originates from topic fluctuations. As a general rule, for two texts of approximately the same length ($N(M)\approx \mu (M)$), the relative difference in the vocabulary must be larger than $27.7\%$ to be sure on a 95%-confidence level that the difference is not due to expected topic fluctuations.

We illustrate this approach for the vocabulary richness of Wikipedia articles. As a proxy for the true vocabulary richness, we measure how much the vocabulary of each article, N(M), exceeds the average vocabulary ${{N}_{{\rm avg}}}(M)$ with the same textlength M empirically determined from all articles in the Wikipedia. In practice however, when assessing the vocabulary richness of a single article, information of ${{N}_{{\rm avg}}}(M)$ from an ensemble of texts is usually not available and measures such as the ones described previously are needed. In figure 4, we compare the accuracy of measures of vocabulary richness according to Herdanʼs C, figure 4(a), and the z-score, figures 4(b) and (c). For the latter, we use equation (24) and calculate $\mu (M)$ from Poisson word usage by fixing Zipfʼs law and assuming Gamma-distributed word frequencies across documents, see appendix E for details. We see in figure 4(a) that Herdanʼs C shows a strong bias towards assigning high values of C to shorter texts: following a line with constant C, we observe for $M\gtrsim 10$ articles with a vocabulary below average, whereas for $M\gt 1000$ articles with a vocabulary above average. A similar (weaker) bias is observed in figure 4(b) for the calculation of the z-score for the case in which we consider deviations from the pure Heaps' law but treat frequencies of individual words as fixed, i.e., ignoring topicality. The z-score calculations including topicality in figure 4(c) show that we obtain a measure of vocabulary richness which is approximately unbiased with respect to the textlength M (contour lines are roughly horizontal). Furthermore, in contrast to the two other measures, we correctly assign the highest z-score to the article with the highest ratio $N(M)/{{N}_{{\rm avg}}}(M)$. Altogether, this implies that it is not only important to consider deviations from the pure Heaps' law, but that it is crucial to consider topicality in the form of a quenched average.

Zoom In Zoom Out Reset image size