[We would like to thank Paul Jakus (@paulj) of the Dept. of Applied Economics at Utah State University for this summary of research presented at the 2024 Phish Studies Conference. -Ed.]

This is the fourth and final blogpost regarding the current rating system. Previous posts can be found here, here and here.

Post #2 showed how two metrics—average deviation and entropy—have been used by product marketers to identify anomalous raters; Post #3 showed how anomalous users may increase bias in the show rating. Many Phish.Net users have intuitively known that anomalous raters increase rating bias, and have suggested using a rating system similar to that used by rateyourmusic.com (RYM). RYM is an album rating aggregation website where registered users have provided nearly 137 million ratings of 6.2 million albums recorded by nearly 1.8 million artists (as of August 2024).

Similar to Phish.Net, RYM uses a five-point star rating scale but, unlike .Net, an album’s rating is not a simple average of all user ratings. Instead, RYM calculates a weighted average, where the most credible raters are given greater weight than less credible raters. Weights differ across raters on the basis of the number of albums they have rated and/or reviewed, the length of time since their last review, whether or not the reviewer provides only extreme ratings (lowest and/or highest scores), and how often they log onto the site, among other measures. These measures identify credible reviewers and separate them from what the site describes as possible “trolls”. Weights are not made public, and the exact details of the weighting system are left deliberately opaque so as to avoid strategic rating behavior.

Read more...