Here’s what Freakonomics gets wrong about advertising

Freakonomics recently applied their formula of Pop Economics to put a challenge to the efficacy of advertising: “Does Advertising Actually Work?” where the hosts lay out a compelling narrative to suggest that advertising may have little uplift… but it’s just not that simple.

“Guess what? Advertising doesn’t work!” makes a great narrative hook for a podcast, but it’s not helpful if what you’re trying to do is measure advertising effectiveness.

My concern here is that a lot of decision-makers believe Freakonomics’ general premise that advertising is “guilty until proven innocent.” However, anecdotes aren’t a substitute for a rigorous measurement strategy bespoke to a brand’s characteristics and conditions.

Instead of trying to find “smoking guns” that highlight the inadequacy of advertising, the more productive path for a CEO or CMO is to understand the inadequacy of measuring advertising correctly.

The <3 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

How do I know? Over the last 20 years, I’ve measured advertising for hundreds of brands and witnessed the refinement of measurement to a point where advertisers can reliably predict business outcomes.

So let’s do a point-by-point walkthrough of how to think about advertising measurement in a more productive, less knee-jerk way:

Freakonomics conclusion #1 — Teasing out the causal part of TV uplifts is a near-impossible task

I went back to this company and I said, ‘I’m really sorry to say, but with the data you have, with nothing like a randomized experiment, it’s just possible that the return on investment could be anywhere from zero to infinity.’

Freakonomics presumes here that academic studies have failed a number of times to understand the causal impact of ads on sales because there was no randomized experiment to examine.

While randomized experiments are preferable when conducting research, they aren’t always achievable, especially in the case of advertising where countless variables must be accounted for.

Evaluation techniques, such as Market Mix Modelling (or MMM) can pull apart all sales drivers without the need for prior tests to be set up.

Freakonomics conclusion #2 — You can’t disentangle seasonality and media uplifts

 So, of course there was a correlation between the TV advertising and store sales. But it’s not necessarily or even primarily because of the ads. It’s because the company knows when the big selling days are, and they target the ads around it.

Brands will indeed — and quite rightly — advertise during seasonal peaks. The podcast puts forward an endogeneity problem: brands that advertise over seasonal peaks are getting advertising uplift confused with seasonality (seasonal demand spikes that would have happened anyway) and “it’s impossible to disentangle” the two.

Approaching this problem the wrong way can cost brands a great deal in mis-allocated spending. Seasonality and media uplift can indeed be disentangled by analyzing a longer time series of data.

For example, analyzing three years of data builds up as much real-world behavior as possible to inform the measurement model, giving you three chances of isolating seasonal effects (e.g., 3x Christmas, 3x Easter, etc).

Also, advertising when customers organically look for your product doesn’t have to be a bad thing. For some brands, seasonal peaks are the highest-ROI opportunity for advertising, so it makes the most financial sense to advertise over the highest possible baseline of demand.

Freakonomics conclusion #3 — Using randomized experiments to measure advertising uplifts in its entirety

So, one of the challenges with measuring the effects of advertising is that firms aren’t out there assigning their advertising randomly across geographies and across time periods.

Freakonomics presents randomized experiments — or A/B testing — as the best method to measure media uplifts.

While I think A/B testing is great — it is after all one of the most common means of measuring uplifts — it has its shortfalls. It can miss the mark on advertising uplift as it:

1. May only measure the initial sales lift and doesn’t measure the long tail effect or memorized effect of advertising; and
2. Can miss local nuances. For example, a geographically split experiment may miss: different responsiveness to price by geographic area (DMA, postal code, etc), climate factors, store closures, local marketing, etc.

Market Mix Modelling considers the initial sales lift of the advertising, as well as measuring the short to medium-term impacts, sometimes lasting up to three years. These kinds of longer-term branding impacts can add around 250% on the initial uplift.

Freakonomics conclusion #4 — TV ads are unprofitable

We find that almost all brands seem to be over-advertising, and that they are earning a negative R.O.I. from advertising in an average week. And if they were to instead decide not to advertise in a given week, they would earn higher profits.

It’s put forward that TV is unprofitable, and the uplifts are underwhelming. Now to break it down, I’d say there are two things going here: one is the uplift, and the second is the way ROI is measured.

Firstly, the initial uplift suggested by Freakonomics of between 1% and 10% rings about true, but this is true of a consumer packaged goods (CPG) brand, which is the use case the paper is derived from.

Media uplifts for CPG are notoriously small, not because the media impacts are nominal, but because CPG brands tend to spend 10x their total media budgets on trade promotions (temporary price reductions, multibuys, etc.) and this dwarfs media uplifts. Outside of CPG, you can expect to see media uplifts anywhere between 5% and 40% of sales.

Secondly, the ROI calculation is a bit squiffy — they are calculating the marginal ROI rather than the absolute ROI. Freakonomics puts it thusly:

The vast majority of brands over-invest in advertising and could increase profits by reducing their advertising spending

This is a very different point to whether the advertising is paying back or not. It is simply saying the ROI will increase if the budget is reduced. It is not an absolute ROI.

Increases in spending will tend to lead to lower ROI/efficiency, but not the absolute profit uplift. In my experience, 8 times out of 10 clients can increase their spend on media and enjoy higher absolute profits.

Advertising doesn’t always work — the key is being able to tell when

While entertaining, sensational anecdotes from Freakonomics aren’t helpful for brands and decision-makers when it comes to understanding ad effectiveness. The key is to measure and find out what works specifically for your brand.

In your personal life, I hope I haven’t diminished your enjoyment of Freakonomics. In your professional life though, I hope this has been useful input for the next time you are drawn into an “advertising doesn’t work” debate.