How Good was Our Ad?

Ad-tests typically benchmark performance against a large back-catalogue. We discuss how this is misleading and it’s necessary to test the target ad alongside a selection of relevant “competitor contrast ads”.

In 1724 a Polish glass blower put a sealed tube of mercury into a bucket of freezing seawater and marked the mercury’s level with a “zero”.  He then warmed it up to body temperature and labelled it “100”.  Daniel Fahrenheit had created the first modern thermometer.  And this relationship to objective human experience remains a recurring theme in measurement.  The mile is based on marching Roman soldiers and time is built around seasons and days¹.  But what are the equivalent tangible scales in human psychology?  More practically, just how funny is a funny advert?

Attempting to define “100 hoots” as the point at which an average person laughs is a doomed endeavour.  Comedic tastes are changeable and unstable.  Jokes are soon antiquated, as my six year old often points out.  Indeed, at base this is the philosophical question of “qualia”.  How can one define people’s mental experiences?  This is a question that Thomas Nagel concluded had no meaningful answer when he delightfully asked “What is it like to be a bat?” and soberly avoided the response “quite squeaky”².  So, an audience’s reaction to an advert has no objective scale.  It must inevitably be judged against other ads.  Crucially, then, which ads?

Sample of Supermarket Adverts

Traditionally this has been solved by using an entire back-catalogue of previously tested adverts.  But the graphic illustrates how this can be completely misleading.  A typical ad-test asks people to gauge various emotional and cognitive dimensions.  In this example, respondents judge how much they’ve learned (Informative) and how much excitement they experienced (Arousal).  The chart shows the performance of four Valentine’s Day ads.

The M&S advert scores around 56 on both dimensions.  In truth, it’s a fairly worthy trade-driving meal-deal ad.  This is reflected by the wider set of supermarket ads denoted by the blue dots.  Against these, Arousal attains 50th percentile and Informative rates 69th.  So not a particularly successful advert, you might conclude.  But this comparison set includes all kinds of adverts, spanning an Aldi price-promotion, a Waitrose Christmas tear-jerker, and a controversial Sainsbury’s WWI ad.  By contrast, the red dots show that M&S actually beat the other three Valentine’s Day ads that year.  In its correct context, this was a successful ad.

This example demonstrates the crucial role played by the reference set.  Our clients tend to want to know how their advert performed against competitors’ immediate equivalents, so we test up to four “primary contrasts” alongside the target advert.  By extension, this also negates the value of having a vast back-catalogue which contains Japanese automotive ads from the 90s.  More general comparisons should be focussed on recent adverts from the same geography and sector.  Reports using a wider reference set should get the Fahrenheit 451 treatment.

1. The origins of the mile can be traced back to the distanced travelled by a Roman legion in 1,000 paces (i.e. two steps). An hour is defined by dividing the day into night and day and then subdividing those into twelve units (there being approximately twelve lunar cycles in a solar year).

2. Nagel, T. (1974). What Is It Like to Be a Bat? The Philosophical Review, 83(4), 435-450.

3. Waitrose Valentine’s Day ad; Tesco Valentine’s Day ad; M&S Valentine’s Day ad; Sainsbury’s Valentine’s Day ad