The Ratings Conundrum

Some time back, the Oregonian's main movie reviewer was a guy named Ted Mahar. His reviews were generally positive--and this was a problem. It got to be that we nick-named him "three-star Mahar" because he never really panned or gushed about a movie. Everything was, in the manner of the children of Lake Wobegon, above average. And therefore his reviews were of little use.

Not too long ago, a friend of mine castigated me for turning into a three-star Mahar. It's true--most of my ratings fall in the range of A- to C+ (here's what they mean). I plead helplessness, though: most beers here fall in that range, too. Obviously, few beers qualify for the "world class" standard ("a superlative example of the style or an exceptionally original beer"). But few fall below the standard attained by a C+ beer, either ("a well-made beer that is a fairly common example of its style or a near miss on originality").

I am, obviously, a homer, and so you have to suspect me of putting my thumb on the scale. But scan through the aggregate scores on BeerAdvocate and RateBeer--they're no better. This may be due to different factors, but the upshot's the same. With only very rare exceptions do breweries put out either badly ill-conceived recipes or beer with off-flavors. We have a problem of compression.

I'm almost to the point of abandoning ratings altogether and just letting the descriptions stand on their own. (I've actually done that more and more with recent reviews.) It's easy enough to change the scale so there are more calibrations between "common" and "world class," but that doesn't exactly resolve the problem. Worse, it exaggerates the effect of personal preference. I'd be willing to entertain some wholly novel style of ratings, though, if only I could think of one. Any suggestions? How do you make sense of subtle differences between beers?
_____________
Share

Banish All Ratings!

So says Joe Sixpack, via Appellation Beer:

FURTHER EVIDENCE of the dumbing down of America is now on display on the necks of Pilsner Urquell. The bottles are adorned with a big "93," its most recent score from Chicago's Beverage Testing Institute....

But who needs a score to choose a beer?

I dunno - someone who wants to impress his friends, maybe, but doesn't have a clue about style, flavor or quality? Who is so filled with self-doubt, he can't spend a few lousy bucks without an expert holding his hand? Whose vocabulary is so limited he needs a number to describe what's in his glass?

A wee bit harsh, but overall, good analysis. Ratings are tough. I really dislike RateBeer and the Beer Advocate's rating system, and wondered if I should have them at all. I do, but uneasily.

New Ratings!

I have decided to abandon my old rating system. Although I maintain the philosophical underpinning was sound, as a method of communication, it is perhaps inadequate. While I know what I mean when I say "average" or "excellent," apparently it's not a universal nomanclature. So instead I'm moving to the classic school-based, letter-grading system.

• A ___ World class; a superlative example of the style or an exceptionally original beer.
  
• B+ to A- __Technically flawless, just short of the kind of character that distinguishes it as as the best in its style or as a truly original beer.
  
• C+ to B __A well-made beer that is a fairly common example of its style or a near miss on originality.
  
• D+ to C __Nothing stands out; beer doesn't have off-flavors but fails to impress.
• D- to D __Off-flavors mar the recipe.
• F ___ Off-flavors so profound the beer is undrinkable

This gives me a little more leeway to distinguish within broad categories, and to distinguish the transcendentally sublime (A) from the merely extraordinary (A-).

[Note: This post has been updated to correct numerous embarrassing errors. See comments for further analysis and documentation of my crimes and misdemeanors.]