[We would like to thank Paul Jakus (@paulj) of the Dept. of Applied Economics at Utah State University for this summary of research presented at the 2024 Phish Studies Conference. -Ed.]

The first two blogposts in this series can be found here and here. This post will address the statistical biases believed to be present in the data, and how anomalous raters may contribute to bias.

Statistically, a show rating represents our best point estimate of an unobservable theoretical construct: the “true” show rating. To the degree that an estimated show rating deviates from its true value, the error is composed of sampling variance and bias. In the figure below, think of the bullseye as the true show rating, and the red dots as our estimates (best guesses) of the true value.

Variance acknowledges an unavoidable aspect of any statistic based on sampling: we’ll probably get it wrong because we can’t ask everybody about their rating of every show. That said, if we have a large, unbiased sample, we’ll come really close to hitting the bullseye (the lower left target). A smaller but still unbiased sample will miss by a larger amount but, on average, the misses will be randomly spread around the true value (lower right target.)

Bias measures the degree to which we systematically miss the true value. That is, there’s something about the people who rate shows, or how they rate shows, that means we will always miss the bullseye regardless of the amount of variance (the targets in the top row). We have two major forms of bias:

Sampling Biases

We do not have a truly random sample because people who rate shows—and the shows they choose to rate—are self-selected.

1. Potential raters are not drawn with equal probability from a sample of Phish fans. Instead, all raters must be registered users of Phish.Net, who are likely to be more enthusiastic than the typical Phish fan. Even then, only a fraction of all registered users have ever rated a show. The ratings database is unlikely to be representative of the general population of fans.

2. Most Phish.net raters do not rate all the shows they’ve heard on tape, seen on video, or attended in person. If people selectively rate only those shows they consider the best, or those shows they have heard are the best, or only those shows they have attended, then the shows raters choose to rate are no longer random, and we are likely to systematically miss the true show rating.

Response Biases

1. Attendance bias. Ratings for shows that one has attended are alleged to be biased upwards (fluffed) because attending a show is the best way to enjoy Phish.

2. Recency bias. Rating a show too soon after its conclusion is alleged to bias show ratings upward (fluffed) because the “warm glow” of a solid performance has not yet faded.

3. Herding effect. Phish.Net shows any rater the average show rating of all previous raters. For example, you think a show earned a ‘3’, but then you see that on .Net that 165 people rate that show as ‘4.3’. The product marketing literature has found that people adjust from their initial rating so as to follow the herd (i.e., maybe you submit a ‘4’ instead of a ‘3’.)

4. Deliberate distortion. Here we’re talking about people who rate almost every show a ‘1’ (bombers) or a ‘5’ (fluffers) regardless of show quality. Are these people simply telling their own truth, or are their ratings intended to bias the rating estimate away from its true value?

While we can do little to control for sampling biases, we may be able to use the average deviation and entropy metrics to mitigate the various response biases—if these biases are associated with anomalous rating behavior. That will be the topic of the fourth, and final, blogpost.

If you liked this blog post, one way you could "like" it is to make a donation to The Mockingbird Foundation, the sponsor of Phish.net. Support music education for children, and you just might change the world.