DoubleClick outage highlights Google’s ‘single-point-of-failure’ problem

By Ricardo Bilton • November 12, 2014 •

When Google catches a cold, the entire Internet gets sick.

Publishers across the Web were crippled this morning when DoubleClick for Publishers (DFP), Google’s ad serving software, went down for over an hour, preventing many sites from running ads.

While the outage was clearly a big headache for Google, whose business is almost entirely advertising-supported, it’s arguably a bigger one for publishers, who miss out on significant sums of money for every second that DFP was down. Major sites like the Wall Street Journal, BBC, Wired and even YouTube were all affected by the outage.

Quick! Doubleclick ad server is down. Time to look at all the internet. It’s free!

— dave epstein (@epstein) November 12, 2014

And, understandably, publishers aren’t happy about the way things, quite literally, went down.

“Google has obviously become a single point of failure,” said the CEO of one publisher, who asked to remain anonymous. “I get that the Internet breaks, tech breaks, things go down, but that’s all magnified when they go down. We’re still not sure how much this is going to cost in revenue.”

Ad rendering preventing in staging

Ad position: web_incontent_pos1

The exact magnitude of publisher losses is still unknown, but publishers may have lost as much as $1 million collectively during the outage, according to AppNexus CEO Brian O’Kelley. Some publishers wonder Google will compensate them for the revenue they lost during the outage.

Fill in the blank – Every second that DFP is down, the ad industry loses ___ amount of money. #DFP

— Tim (@timtrev) November 12, 2014

Ad rendering preventing in staging

Ad position: web_incontent_pos2

“DoubleClick for Publishers experienced an outage this morning impacting publishers globally, across their video, display, native and mobile formats,” Google said in a statement. “Our team has worked quickly to fix the software bug and DFP is now back up and running, so our publisher partners can return to funding their content.”

The stakes here are hard to ignore. Google commands 31 percent of the $120 billion global digital ad market, according to eMarketer, and DoubleClick for Publishers is a major revenue source for many sites.

While few publishers have any misconceptions about the fragile nature of the Web and the advertising infrastructure built on top of it, the DFP outage does wake them up to the risks of relying too heavily on a single technology, even if that technology comes from Google.

“The outage was only an hour and a half, so the revenue loss wasn’t significant,” said Alex Magnin, CRO of Thought Catalog. “But the whole experience certainly does highlight our reliance on DFP, and how we take it as a given that DFP will work.”

Magnin said that while Thought Catalog isn’t in a position to reduce its reliance on DFP, it’s already exploring backup plans that would let it switch to hard-coded tags from other networks in case DFP goes down again.

Companies that rely less exclusively on Google fared much better during the outage.

“We’re probably less affected by this than most publishers,” said Kevin Johannesen, VP of sales at Someecards. “We generate the lion’s share of our revenue through branded content and skin/wallpaper display ads that we deliver through our CMS.”