If I told you I had the winning lottery numbers for next week’s £40m jackpot – and that I was willing to share them with you, the lucky reader – what would be the very first question you’d ask?

Quite rightly, you’d ask how I came to know these numbers.

I’d explain that I own a wonderful machine called the LottoMetron – a device capable of churning out the winning numbers week in, week out, thanks to some very clever mathematics.

At a cost of just a couple of pounds a ticket, you might take a punt. You might not ask too many questions and simply accept the black box.

But what if I told you that it wasn’t £2 a ticket? What if it required hundreds of thousands, if not millions, of pounds up front? Suddenly you’d want to understand more about how this machine works – and, crucially, how reliable its predictions really are.

All of this feels perfectly logical when you’re dealing with a hypothetical lottery. But here’s the question: why does that logic vanish the moment we replace lottery tickets with advertising budgets, and the LottoMetron with marketing mix modelling (MMM)?

I’m weary of conversations with clients and planners who are basing multimillion-pound decisions on MMM outputs with little idea of how those results came into being – or how reliable they really are.

Perhaps my experiences are skewed. Perhaps many of you do ask the right questions. In fact, if you’ve ever received an MMM model output and pressed for at least three of the following, then please consider yourself excused from this rebuke:

• Can I see a confidence interval for each channel?
• Has this model been tested for multicollinearity, and can I see the results?
• Have you tested coefficients across rolling windows of data? Were the coefficients stable?
• Did you use a holdout data set, and if so, did it reveal signs of overfitting?
• Can you give me a full list of all the variables this model accounts for?
• Can I see a measure of gross profit, not just ROI?
• Have decay rates been applied, what are they and can you outline how these were derived?

If you’ve asked at least a handful of these, I suspect you’re doing your job – you’re protecting your brand’s money.

But the rest of you? The vast majority? You’re waving through decisions worth tens of millions on the blind faith that the LottoMetron must be right. And that, to put it bluntly, is indefensible.

‍

Cards on the table

I am not an econometrician. I couldn’t build an MMM model – nor would I want to. But like many of you, I am routinely asked to work with MMM outputs. Until about 12 months ago, I did exactly what I now recognise 99% of clients and planners still do: pore over histograms, each bar slightly smaller than the last, each boasting its own lovely ROI.

With no grounding of my own in regression modelling (for that is all an MMM really is), I asked no further questions. The MMM gods had spoken – simply bow, nod, and accept that the channel with the biggest bar must be the best place to stick the cash. Sod the rest.

But that was 12 months ago. Since then, I’ve been on a bit of a journey of discovery, and I’ve realised something important: it is entirely possible – for anyone – to develop an intuitive grasp of multiple regression, econometrics and MMM models. And let me tell you: once you start to understand these models – how they’re actually built – you’ll see that these towering monuments to marketing effectiveness are, more often than not, perched on crumbling, worthless foundations.

At the very least familiarise yourself with how these models work. Learn how to interrogate them.

Now, before I’m formally hanged, drawn and quartered by the eminent Grace Kite (seriously, love your work) and the other luminaries in her camp, let me make myself clear. I recognise that in the absence of insights from rigorous marketing experiments, based on single-source data, MMM models are often the only flicker of light in an otherwise dark cave. And if that’s the only light you have, then of course you’ll make it your north star.

But, for the love of God, if you are in this position, at the very least familiarise yourself with how these models work. Learn how to interrogate them and how to build media plans that give modellers half a chance of producing something vaguely valid.

‍

Understanding the basics of MMM

At an intuitive level, these models are simple creatures. You feed them time-series data covering all the variables you think influenced sales over the last two or three years (an improbable task in itself, but that’s a tale for another time). Then you throw in your sales data for the same period.

What the model then does is scour all this information looking for patterns. It asks: when sales went up, what else was happening at the same time? Was TV on air? Did digital spend spike? Was there a price promotion running? Did the weather turn hot? Was the moon waning gibbous or waxing crescent? The model crunches through thousands of these overlaps, testing which variables appear to move in tandem with sales and which don’t.

From there, it assigns weights to each factor, essentially saying: “Given the historical patterns I’ve spotted, here’s how much I think TV contributed, here’s how much digital contributed, here’s how much was price, seasonality or something else entirely.” All of this rests upon the model’s attempt to conjure a neat ‘baseline’ (something only an econometrician could say with a straight face) of what sales would have been without advertising. From that baseline, it infers the incrementality the ads delivered.

The end product is a tidy-looking set of numbers, neat bar charts and an average ROI figure. And that’s the crucial bit – unless otherwise specified, the ROI isn’t a verdict on your last campaign, or the one before. It’s a locked-in coefficient spread across everything you fed in. That means weaker campaigns pull the number down, stronger ones are watered down and the individuality of each effort is lost – along with any chance of learning what really worked in a specific campaign.

The black box doesn’t get a free pass just because someone’s slapped a shinier label on it.

Those outputs may look convincing, but underneath the polish the model is still just making educated guesses, shaped entirely by the data you provided and the assumptions baked into its equations. Miss out key variables, run channels together or keep spend too steady and the model can be easily fooled, giving credit where it isn’t due or stripping it away from where it should be.

To show just how easy it is to fool a model – or rather, how easily a model can find patterns in pure noise – consider this: in a test described by Armstrong (1970), a model was built with eight variables and an impressive-looking fit (R² = 0.85, for the mathematically gifted). The only problem? Every single number was random.

In fact, with so many variables to juggle and so much random noise in the system, it’s no surprise that Dawes et al, in their sublime critique ‘Forecasting advertising and media effects on sales: Econometrics and alternatives’, conclude: “The assumption that developing realistic – and hence predictively valid – econometric models for advertising and media decision making is in practice rather heroic.”

Of course, some clever souls will tell you that newer, fancier flavours of MMM – Bayesian, stochastic, take your pick – can magic away a lot of these problems. Maybe they can smooth some rough edges, but the point stands: it’s still a model, not gospel. If you’re going to use it, you need to learn how it works and ask questions. The black box doesn’t get a free pass just because someone’s slapped a shinier label on it.

‍

Hiding in plain sight

At this point, it would be easy to cast econometricians as the villains of the story – but that couldn’t be further from the truth. I’ve had the pleasure of talking through my ideas with several brilliantly talented econometricians, all of whom are fully aware of the limitations of their craft. Many openly admit they’re surprised at how rarely their findings are challenged, and how seldom clients ask to see the battery of statistical tests that would reveal how much the results can really be trusted.

One modeller put it to me candidly: “I like to assume people know how to test a model and simply choose not to ask – but I certainly don’t offer it up on a plate. Why shatter the illusion? Who am I to tell them Father Christmas isn’t real?”

To my mind, that’s a perfectly reasonable stance. What sort of salesperson undermines their own product? The modeller is, after all, selling something – a belief in the model’s authority. If I’m buying a second-hand car, I don’t expect the seller to reel off everything that could go wrong; I accept that if I want to know, I’ll need to ask the right questions – and that means learning enough about the engine to know what to ask.

The good news is that, while modellers won’t necessarily volunteer evidence of model validity – they are, in my experience, always happy to oblige once asked.

‍

Knowing what to ask for

As you’ve perhaps already gathered, unfortunately, knowing what to ask and how to interpret what you’re given will require some learning – it’s complex stuff and understanding complex stuff takes effort.

That said, in a bid to make your MMM journey slightly less painful than mine, I’ve been working on an intuitive guide to questioning MMM models – ‘From mythical to mastered: Making sense of MMM’. No complex maths, just simple intuitive explanations (all sense-checked and rubber-stamped by an IPA-accredited econometrician) and brought to life through the unmistakable imagery of Dan White (see main image, above).

Read more about the guide and download the whitepaper here.

Knowing what to ask and how to interpret what you’re given will require some learning – it’s complex stuff and understanding complex stuff takes effort.

The guide covers 12 key questions – what I like to call the Dirty Dozen of MMM.

One of the twelve? Ask whether the model has been tested for multicollinearity – when two channels always run together, the model can’t separate their effects. If OOH and TV always overlap, the model may give all the credit to whichever channel shows slightly more variation in the data, leaving the other unfairly penalised. A simple request for a correlation matrix or a variance inflation factor (VIF) score will quickly tell you whether this is a problem.

Another? Demand to see confidence intervals alongside those shiny ROI figures – because without them you’ve no sense of how reliable the number actually is. A channel boasting £5 back for every £1 spent sounds great, until you realise the real answer could be anything from £0.50 to £9.

One more? Ask whether the model has been cross-checked against experimental data. If MMM says TV underperforms but geo-tests show a lift, the model’s missing something. The smart move is to ask: where is the model blind, and can experiments fill the gap?

The other nine?  If you can’t already guess them, then you’re exactly who this guide was written for.

‍

Last spin of the LottoMetron

As I write this, the EuroMillions jackpot stands at £26m and a quick glance at the LottoMetron says: 16, 40, 23, 17, 21 (36, 42).

No questions? Fine. Good luck.

Update: the winning numbers were 10, 14, 28, 38, 39 (02, 04). The LottoMetron has now been quietly consigned to the bin – right next to any faith I ever had in black boxes.

‍

*Article originall published in Marketing Week.