So, were all those multi-touch attribution models validated?
The natural experiment of the Facebook outage
Unless you were living under a rock, you noticed that Facebook and Instagram went down for a while yesterday. Since one out of every 4 dollars spent on online advertising goes to the big blue, lots of marketers’ ad dollars were not getting spent when normally they would.
For the purposes of analysis, the outage probably didn’t last long enough to make reliable inferences, but as a thought experiment I thought the out-of-the-blue outage was interesting regardless. If ads on Facebook were truly driving a lot of sales, we’d expect to see a corresponding drop in sales during the outage (and after, for however long the adstock effect lasts for, for Facebook, for that specific product); if not, well, we’d see no impact.
Generally speaking, marketers use a set of tools called “attribution models” to attempt to figure this all out. The idea is that you give credit to a specific advertising channel for each customer. Basically you say “this customer came to us through Facebook”. Often you give credit to multiple channels, through a “multi-touch” model, which gives, say, 50% of the credit to Facebook and 50% to Google. There are first-touch and last-touch models, models that work at the level of the order, not the customer, and sometimes even the order item, and every variation in between. Once you can assign credit like this, you can then do some arithmetic to work out how effective the marketing spend in each channel was.
Somewhat snarkily, I wrote on Twitter:
Why would I write that? Here’s the problem: lots of people hang out on Facebook, and lots of people also buy products. Just because someone did both doesn’t mean that the Facebook ad made any difference in their purchasing decision.
Let’s say you’re definitely going to buy the BrandCo Widget 3000. If you’re gonna buy the Widget 3000, more likely than not, you’re also the type of person that buys a Widget 3000. And if you’re the type of person that buys a Widget 3000, Facebook knows it. Google knows it. Amazon knows it. This isn’t new: direct mail vendors in 1980 would have known it too. Sorry, what people are likely to buy is a lot easier to predict than you think.
Since you’re the type of person to buy the Widget 3000, Facebook serves you an ad for it.
So you get served an ad on Facebook, and you buy the Widget 3000. BrandCo’s attribution model sees these two things happen and assigns credit for your purchase to Facebook. BrandCo’s marketers estimate that Facebook is causing lots of purchases.
But this isn’t true! You would have bought it anyway. The ad you saw had no effect.
This isn’t a new problem, but it’s pervasive and pernicious. One of Recast’s clients recently had to turn off their marketing spend due to supply chain constraints and go to waitlist-only. Their attribution model told them Facebook was driving about 25% of their purchases. So when they turned Facebook (and everything else) off, what happened? Nothing. Sales (this time to reserve a place on the waitlist) stayed right where they were.
So if you’re a marketer, what happened yesterday?