MMM: How consistent are the channel inputs in this model?
Is the advertising being well managed in the model?
MMM averages historic performance over time to predict how well it will perform in the future.
Any changes over time, to media strategy, creative quality, campaign messaging, formats within channels, audience targeting, will impact how the model is able to determine meaningful marketing effectiveness results. Unless specifically told otherwise, the model can’t distinguish between past and present activity. The danger is:
- older poor executions of creative or planning will drag down the coefficient the model assigns to the channel.
- Older, more successful creative and planning where work is now weaker, the model will overstimate the impact of current activity.
Out of Home advertising is extremely sensitive to creative execution and environment, where there are lots of choice of formats. A simple average across the channel could be unreliable.
Are past campaigns comparable to today’s execution - or is the model averaging very different strategies?
YES!
Creative and deployment are broadly consistent.
Good - the model is learning from comparable inputs.
Ensuring campaign strength (whether consistently strong or weak) and checking current and historic campaign creative, targeting and formats will provide better data for the model to work with.
NO!
Execution has changed significantly.
The model may be misrepresenting current performance by averaging unlike campaigns, blurring true performance differences and distorting the future estimated effects.
Watch out for: valuation bias, a lack of supplementary evidence, poor segmentation of the model inputs such as format or campaign quality, and using historic coefficients as gospel without the correct application or data.
For more information: