Power-law distributions in empirical data · HackerPapers discussions

Power-law distributions in empirical data — mentioned by bikenaga

Thu, 12 Feb 2026 17:30:20 GMT

"Abstract: We formalize and study a generalized form of the Pareto principle or '20/80–rule' as a property of bounded cumulative processes. Modeling such processes by non-negative gain densities, we first show that any such process satisfies a generalized Pareto principle of the form 'fraction p of inputs yields fraction 1 − p of outputs'. To obtain a non-trivial and unique characterization, we define the generalized Pareto principle via the decreasing rearrangement of the gain density function. Within this framework, we analyze both constructed gain densities that exemplify the framework and…

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by PaulHoule

Fri, 07 Feb 2025 17:38:41 GMT

That Science paper https://www.science.org/doi/10.1126/science.adq7055 ought to be retracted, it is using the same ad-hoc methods that we used to wrongly fit power laws to statistical distributions when I was a grad student 30 years ago. I wrote a trash paper too and I'm still angry about not feeling psychologically safe to bring the issue up. A postdoc who was there wrote a paper many years later about the right way to do it https://arxiv.org/abs/0706.1062 I don't think anybody read it. Note that the Science paper uses the "Pearson product moment correlation" which is "can only be used to…

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by PaulHoule

Tue, 29 Aug 2023 12:08:35 GMT

... less power It is a sore point for me that physicists have long wrongly plotted distributions on log-log paper, drawn a line, and pronounced they discovered a "universal" power law without ever once using words like "estimator", "significance", etc. My first paper was wrong because of this and when I was asked what I thought the cause of our observation was I didn't give my real answer (which I'd already had given and had rejected) which was that our analysis was bogus. Mark Newman was at our lab at this time and later on he wrote some good papers on this project.…

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by soVeryTired

Fri, 01 Jun 2018 09:40:37 GMT

For what it's worth, showing a straight line on a log-log plot isn't really enough to demonstrate existence of a power law. Most papers that test for the existence of power laws don't test goodness-of-fit for other distribtions. The lognormal distribution is often just as good a fit to the data. This paper covers a lot of the frequent problems in academic literature that tries to fit power laws: https://arxiv.org/abs/0706.1062

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by curuinor

Tue, 14 Feb 2017 23:25:23 GMT

Clauset Shalizi Newman 2007 has not-nice things to say about the classic physicist's idiot trick of fitting power law distributions by drawing a straight line on a log-log graph: it's got huge bias. https://arxiv.org/abs/0706.1062 However, the other difficult thing about power law distributions is that the dataset size requirements for proper determination of the fact that it's a power law distribution are occasionally incredibly difficult. So their critique is very strong, given the comparative lack of data. It is often the case that computer systems, with the overflowing reams of data, are…

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by smallnamespace

Sat, 24 Sep 2016 21:16:29 GMT

being roughly straight on a log-log plot is a necessary but not sufficient condition for power-law behavior Source: https://arxiv.org/abs/0706.1062, p.15 pp. 24-29 have nice log-log plots of various data sets, along with whether they can plausibly described as power law distributions. 'Wealth', interestingly enough, looks like a pretty straight line, but fails the statistical test badly. Btw, if you just take logs and run a regression to estimate the exponent, you get a highly biased estimate. In general, it's much better to do a direct statistical test, although the method described in the…

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by mattkrause

Wed, 27 Jul 2016 21:28:34 GMT

>A large number of natural distributions follow a power-law distribution. This is surprisingly debatable. People like to find power-law/scale-free distributions because it implies a neat generative story, but a lot of the "evidence" for power-law distributions is pretty weak. For example, you cannot just show that a log-log plot is linear--lots of other distributions can produce similar plots. Clauset, Shalizi, and Newman have a very readable paper where they describe 1) how to properly test for a power-law distribution, and 2) use those tests to assess the validity of some claims from the…

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by deaneckles

Tue, 29 Apr 2014 21:28:16 GMT

I think log-normal distributions usually fit this kind of data as well as power laws do. Anyway, such a plot is not the best way to choose between those two models (see http://arxiv.org/abs/0706.1062). We don't actually do any such test in the paper though (or say that it is a power law or log-normal). You might want to check out the paper for more details. https://www.facebook.com/publications/244240069095667/

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by szhorvat

Thu, 25 Apr 2013 15:39:20 GMT

About the "fictive power law" thing: this is THE paper to read: http://arxiv.org/abs/0706.1062 (it's an easy read, explaining what the maximum likelihood method is, etc.). Despite what they say, fitting the log-log CDF usually gives pretty good results when done right (fitting the PDF does not)

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by Dabacon

Thu, 31 Mar 2011 19:13:13 GMT

"Mark wrote a paper about this ten years later, after physicists had published thousands of bogus papers using bogus statistics." Agreed, but to be fair a lot of non-physicists were also doing this. For example in their excellent review article http://arxiv.org/abs/0706.1062 Mark Newman, Aaron Clauset, and Cosma Shalizi show that a claimed power law in bytes received per http request was wrong. I'm not going to defend the sloppiness of physicists, but I would note that other fields don't fare as well in this either. And I wish you or Mark had written that paper earlier too!

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by mayank

Thu, 31 Mar 2011 16:32:16 GMT

Actually, the paper you should be reading is this: "Power-law distributions in empirical data", Aaron Clauset, Cosma Rohilla Shalizi, M. E. J. Newman http://arxiv.org/abs/0706.1062

Paper and discussion history. A mention is not an endorsement.

Power-law distributions in empirical data — mentioned by kurtosis

Mon, 23 Nov 2009 04:53:55 GMT

If you follow the literature referenced in this recent blog post you should be able to find what you are looking for. Also pay close attention to the content of shalizi and newman's review article - there is a lot of notorious and well known statistical sloppiness in the literature on power law distributions. http://cs.unm.edu/~aaron/blog/archives/2009/11/power_laws_an... It's especially interesting that they call the power law distribution of wealth into question. EDIT: the link to the shalizi and newman review in the blog post is a link to a "buy this .pdf" site. Here's the mentioned…

Paper and discussion history. A mention is not an endorsement.