Broad Swords and Beer Awards

It’s nearly two weeks since the biggest night in New Zealand’s beer calendar: the Brewers Guild of New Zealand Awards.  It was a huge night for indy darlings Garage Project, who, only six years on from the 24/24 project that launched them into existence, took out the top gong and were crowned Champion Brewery.  My beloved northern breweries did exceptionally well, too; more than a third of the trophies given for beer styles went to breweries north of the Harbour Bridge.

Being a self-confessed data nerd – like someone else I know – I always look forward to the full list of medals and trophies that the Brewer’s Guild publish on the Sunday following the awards.  This year, the list very usefully included all the beers that were entered for judging, and then stated whether they received a medal or not.  For a data nerd, it’s paradise: having a denominator allows you to figure-out fun stuff, like the proportion of beers entered by a brewery that received a medal.  But most of all, I like to see how the blossoming breweries are doing – contract brewers like Elation Brewing Company, for whom a Silver medal (for their delicious APA Chopper) is just the kind of validation that a newbie needs to keep their dreams alive.

As in previous years, at the end of the report there was a summary of the awards – written by competition organiser Craig Bowen, the founder of beer distribution company BeerNZ.  Craig has been organising this event since 2009, and from all reports has done an exceptional job of creating a world-class event.  In this regard, my hat is firmly tipped in his direction: organising events as large as this one must be akin to herding cats in a yarn-making factory.

In his summary, Craig mentioned that the proportion of beers entered in the awards that went on to receive a medal was 52% – meaning that around half of the entered beers were deemed worthy of either a Gold, Silver or Bronze medal. (A quick refresher about how most beer awards work: breweries enter beers into one of a number of beer style categories, and the beer is judged against a set of predefined criteria regarding what characteristics should exist in a beer of that style.  The degree to which an entry fits the criteria determines whether the beer receives a Gold, Silver, Bronze or nothing at all.)  While Craig noted that the 52% was an encouraging increase from 2016’s 46%, his remarks then proceeded to wander into territory that they really shouldn’t have.

Of the number of entered beers that received medals, Craig – bearing in mind, he is speaking as the organiser of New Zealand’s flagship beer awards – had this to say: “While this increase is encouraging, I believe it is still far lower than where it should be…If only +/- 50% of entries have been judged as medal worthy when received straight from the brewery, then it raises the question as to the overall quality of product purchased by the consumer.” 

Craig’s comments were akin to a teacher scalding his students for a lacklustre mid-term exam result.  But what he’s saying here isn’t a new idea: it’s something I’ve heard a bunch of times from various folk in the industry.  The theory goes something like this: if a beer is entered in the Brewer’s Guild Awards, and it has been well-made, then it should receive a medal.  If it doesn’t, then it’s of poor quality.

But here’s the thing: this theory is, at best, a logical fallacy – a conclusion based on a flawed line of reasoning.  And at worst, it’s a steaming pile of bullshit.  I’m firmly in the Calling Bullshit corner – and I’ve outlined my reasons for this below.  (I apologise in advance for the spittle; but I’m annoyed, so you’ll just have to bear with me.)

Firstly, concluding that a beer’s inability to medal is directly related to its quality makes a fairly substantial assumption about the validity of the system that divvies out the medals.  In other words, it ignores the possibility – probability – that there are flaws in the judging process that may lead to some beers erroneously receiving a medal, and vice-versa.  Craig Bowen (and others) have drawn a straight line between the beer that leaves the brewery and the award outcome: however, that particular causal diagram has multiple other factors that mediate this relationship.  To name a few: the quality of the cold chain from the brewery to the awards; how long a beer sits before being presented to the judges; the experience and quality of the judges; systemic judge bias towards giving low scores; what the judges had for lunch that day, or how much sleep they had the night before; etc., etc., etc.

This isn’t the judge’s fault, nor the organisers: it’s the plight of all beer awards worldwide.  Taste is a subjective sense – some are better at it than others, but there is (as yet) no such thing as a tasting robot.  Even the judges with the most time around the table would agree that, despite all efforts, it isn’t a perfect process.

But even if we were to believe in the robustness of the system – which we can’t, or at least shouldn’t; but perhaps we could assume that the impact of the system’s inherent problems are evenly-spread across breweries and entries – the logical fallacy remains more-or-less entirely intact.  Here’s why.

In order to make a subjective process as objective as possible, the judges – even if they have a bionic taste apparatus – are judging beers against a set of predefined style guidelines.  As mentioned above, the degree to which the beer meets those style guidelines is what determines whether a beer receives a Gold, Silver, Bronze, or nothing at all.  A beer which might win a Gold as a Strong Ale, but which is entered by the brewery as a Pale Ale, will get jack-shit.  Some of the best beers I’ve had in the last month received no medal at all at the awards; and yet somehow – miraculously – they were outstanding beers.  The absence of a medal bears little relationship with whether a beer is Good; equally, it has little to do with whether a beer is Shit.

(Example: Both Garage Project Pernicious Weed and Emersons BirdDog – two perennial favourites of mine, of which I’ve never had a bad batch – didn’t medal at all.  Does that mean they are bad beers?  Of course not.)

And there can’t be a direct relationship between whether a beer medals and whether it’s ridden with yicky faults, either; a full quarter of the beers submitted for judging by Our Humble And Great Brewing Overlord Lion – including some of beer-nerdery’s favourite fridge-stuffers, like Mac’s Hop Rocker and Steinlager Classic – didn’t medal at all.  Given that Lion spent Seventy Bajillion Dollars (a conservative estimate) on the fancy-schmancy equipment one needs to ensure a faultless product, I find it very hard to believe that the lack of medals for 25% of Lion’s entries is a solid indication of problems with quality control at their South Auckland plant.

So, the relationship between whether a beer receives a medal at the Brewer’s Guild Awards has, by its own design, little to do with whether the beer is either a) good, or b) faulty.  It can only possibly tell us how well a given beer fits within the a priori schedule of characteristics that distinguish one style from another.  After years of judging notes that said “excellent beer, wrong category”, Garage Project wizard Pete Gillespie admits to sitting down with his team before this year’s awards, and spending some time on shepherding entries toward the categories that gave them their best chance of success.  And that’s the game: Pete and his team were simply calibrating their aim to give them their best chance of hitting the target.  Just as you can’t blame brewers for poring over the style guidelines in an effort to force a beer into a category, you really can’t conclude that a beer is bad just because it didn’t fit.

To sum-up: we mustn’t be fooled into thinking that the beer awards are a pseudo-indication of the state of beer quality in New Zealand.  For the reasons I’ve outlined here, to do so is not just overreaching – it’s full-blown fallacy.   And far from being innocuous, it’s dangerous to the reputation of our blossoming corner of the industry: make no mistake, it’s a big deal that the organiser of our nation’s preeminent beer awards stated in a public document that the results indicated a problem with the quality of beer leaving New Zealand’s breweries.  We’re just fucking lucky that Fairfax and NZME are too busy with Jacindamania to give a toss about us right now.

I’ve said it before, and I’ll say it again: I’m not suggesting that there isn’t a problem with beer quality in New Zealand – I’m merely suggesting that we ought to measure this properly before lobbing misguided grenades and blowing ourselves up. | | Twitter: @jasegurney

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s