Blind Tasting: Unreliable but Necessary

Blind tasting is an artificial experience. Consumers don’t taste wines blind, except as a game. If you really want to enjoy a glass of wine, you need to know what it is. Where it came from. Who made it. Its story.
But blind tasting is essential for critics. It’s the only fair way to discover unknown wines, and to judge famous wines without succumbing to reputation.

Sauternes+Barsac, blind tastingA big question with blind tasting is, is it repeatable? Will a wine tasted by the same critic on two different occasions get the same score?
A multi-year study of judges at the California State Fair wine competition showed that even for most wine professionals, it is not.
Top critics have a lot at stake in convincing you that their ratings are immutable: that a 92 on Tuesday is a 92 when you buy the wine a month later and a 92 when you drink it on your birthday.
I have put off quoting Steve Heimoff here, but a comment he wrote on his blog last week caused me to (figuratively) tear up the column I had written for Palate Press this month and write this screed on blind tasting instead. Thanks a lot, Steve.

Heimoff often writes to provoke; I have some familiarity with that. He’s often outlandish in his opinions but he’s an excellent blogger because of it: people may read him to get outraged, but they read him.
Still, even by his standards, this comment demands further consideration. Heimoff, who reviewed California wines for Wine Enthusiast for nearly 20 years, wrote:

“I suggest that if we, or any critic, were to taste the same wine multiple times, that scores would vary. And how much they would vary is tied into the nature of statistical analysis. The more frequently we tasted the wine, the more variance in scores there would likely be.”

Whoa. Think about that for a second. The more a critic tastes a wine, the less reliable their scores would be. Heimoff, meet Heisenberg.

It must be pointed out that Heimoff now works in public relations for Jackson Family Wines. He no longer needs to defend the work of the wine critic, and possibly it benefits Jackson Family Wines to diminish the importance of critical scores.

However, I’m going to give Heimoff the benefit of the doubt and say that, as often happens on his blog, he just wrote what came into his head, without any larger agenda.

It’s likely that Wine Spectator will fall over itself saying its own scores are repeatable; that Heimoff is not as accomplished a taster as their own superannuated panel. And maybe they’re right.
It’s a good chance to look at how blind tasting should and shouldn’t be used by critics.

On its site, Wine Spectator has a short mission statement titled, “Why We Taste Blind.” I like it a lot. It reads in part, “Impartiality is crucial to fairness; good judges base their decisions on facts, not favoritism.”
Without naming the Wine Advocate, whose critics don’t taste blind, it says that the Spectator does not believe a critic can “overcome the expectations that are inevitably triggered by knowing the identity of a wine, its reputation and its price.”

Studies have shown this to be true. Tell people that a wine costs more in a blind tasting, and they like it more. And why wouldn’t you be predisposed to like a barrel sample poured for you by a winemaker you know and like?

In his book How to Love Wine, Eric Asimov speaks of his own dislike for the blind tastings he conducts at The New York Times. Asimov writes, “It’s not unusual for a wine that in the past I’ve liked very much to receive a low rating from the wine panel. Ordinarily I take this in stride. But sometimes I find it disturbing.”

That is one of the most useful features of blind tasting: to be disturbed. To ask yourself, do I really like what I like? Or do I like it because I think I should?

Some wines don’t do well in blind tasting. Asimov writes of a German Riesling producer whose young wines taste sulfurous, whereas he likes them as they age. As one of his readers, I think it’s useful for him to realize just how off-putting those young wines are; it would be easier to ignore it in a non-blind tasting.

It has taken me a long time to say Pinot Blanc, when made well, is one of my favorite white grapes, in part because it never does well in blind tastings. The grape has a snap of bitterness at the end; it just doesn’t taste all that good by itself. With food, that finish becomes refreshing and palate-resetting, and keeps what initially seems like a fairly simple wine interesting ithroughout an entire bottle. But a Pinot Blanc would never get 95 points unless it was oaked into a perversion of itself.

The most useful aspect of blind tasting is discovery. At the Concours Mondial de Bruxelles this year my panel got a flight of red wines that we liked and couldn’t identify. Turned out they were from Lebanon, and they were nothing like the musty wines of Chateau Musar, the only Lebanese winery I had heard of. I can’t wait to try some Lebanese wines non-blind; the country of origin is a story in itself.

A great wine critic tells the story of the wine, because that enhances the readers’ appreciation, but doesn’t forget how the wine tasted without it.

7 Responses

  1. Joe Gargiulo

    Blake, I believe there would a huge difference in comparing the consistency of blind tasting in the realm of media/blogger critics (where breaks/rest are an option) v. wine competitions (where judges suffer from palate fatigue due to heavy volume). Joe Gargiulo, JAG Public Relations and Marketing

  2. Jamal Rayyis

    Like many writers/critics, I taste both ways. As you say, Blake, it’s about the context, as well as demands of editors who want the appearance of quantifiable rigor. Tasting blind is can be a great exercise, especially when you taste a wine blind and, after learning its identity, realize that you tasted it unblind some time before. It’s interesting to see how similar (or not) one’s comments are in both situations. The idea that scores could be replicated over time is of course as absurd as the notion that scoring on a numerical scale is anything more than wishful science. A 92 one day isn’t a 92 a second
    or third or fourth day since 1) wine critics do not taste in laboratories.and 2) as much as they believe, they don’t precisely control for all sorts of variables such as temperature (of the wine and of the environment), humidity or barometric pressure (yes, they make a difference), size of the pour (the smaller the pour, the more aeration the wine will receive), time of day, what one had for breakfast, the
    nagging sensation of hangover from the night before, etc, etc. 3) what in the world does a 92 mean, anyway? Yes, I know, people will break it down, but we know for a fact that people’s palates, even trained, are more sensitive to some scents/flavors than others. Some of that sensitivity is innate, some of it is cultural, but, it differs. That said, it is liberating to be free from preconceived notions of how a
    particular, identified wine, should taste. Of course, we also have to know what “blind” means, anyway. Double blind? Triple blind? How many wines? Is it in a competition with a dozen wines in a flight and a dozen flights per day? Or, is it in the calmish confines of magazine’s office with only a few set before noble critic?

    Surely, the critic who knows the vintage and appellation of a wine (or, perhaps grape variety) is susceptible (though not the cuvee and producer) to bias, too. It is doubtful that a Montrachet is going to be treated in the same way as a chardonnay from Margaret River. Perhaps that difference will work in Montrachet’s favor, perhaps, not (if it doesn’t match expectations, it will suffer. Presumably expectations for the Aussie white will be lower). And, if the Montrachet is evaluated alongside other Montrachets, then its score might be influenced by comparison to the others rather than some objective standard of what a chardonnay wine from wherever, should taste like. (There are controls
    for such issues, but even so, these are problematic). So, does a 92 for a Montrachet, even if tasted blind, without knowledge of the producer, mean the same as a 92 for a Margaret River chard tasted a week later? Likely, not.

    As for Chateau Musar, I wouldn’t (generally) call them musty. Perhaps a bit high toned, certainly esoteric. Certainly, they are their own thing, and do not resemble almost anything else from Lebanon. Reds in Lebanon, however, are not made from varieties unique to the country, or even to the Levant. Overwhelmingly, they are blends of classic Bordeaux varieties – with cabernet sauvignon taking the lead, and Rhone varieties. Interestingly, cinsault assumes a large role in Lebanon’s reds, and is more substantial than it is in France. Carignan, too, for that matter. As for the whites, a bit more mixed. Obeidah and merwah, two indigenous varieties, often make it to blends, but so too, does chardonnay, semillon (classic, French), and a host of Rhone whites, too. Overall quality of Lebanese wines is rather high, even on the simplest level. Indeed, I would argue that taken as a whole, Lebanese wines probably reach among the highest average quality levels of any nations wines. And, no, I’m not Lebanese.

  3. Latest Wine Tasting News - Follow Your True Passions .com

    […] Blind Tasting: Unreliable but Necessary But blind tasting is essential for critics. It's the only fair way to discover unknown wines, and to judge famous wines without succumbing to reputation. Sauternes+Barsac, blind tasting A big question with blind tasting is, is it repeatable? Will a … Read more on Palate Pres […]

  4. Bob Henry (wine professional)


    Great (and one hopes, non-rhetorical) questions:

    “That is one of the most useful features of blind tasting: to be disturbed. To ask yourself, do I really like what I like? Or do I like it because I think I should?”

    Being self-aware of your biases is a good thing. Being open-minded by suspending your preconceived notions and rejecting the conventional/received wisdom is even better.

    ~~ Bob

  5. Brian Swisher

    Its hard to imagine anyone truly believing that judging ANYTHING by humans, or by machines made by humans, will be perfectly unbiased. Of course there is variability in judge’s scores, but this doesn’t mean blind judging/tasting is without merit. Its the best method we’ve come up with thus far and its not perfect. How else might we rank wines, beers, cheeses, etc. with less bias?