Mythbuster: How the inconsistent definition of click-through rates affects publishers and their advertisers

Two different email service providers (ESPs) — including Beehiiv and Substack — calculate CTR using what is actually adopted by the industry as CTOR, while Mailchimp calculates CTR (or what the company calls “click rate”) based on the number of emails sent, according to a Digiday analysis and confirmations by Mailchimp and Substack (Beehiiv did not respond to requests for comment).

CTR is defined as the number of unique recipients who clicked on an email divided by unique emails sent, said Dan Oshinsky, who helps publishers and brands with their email strategy through his consultancy Inbox Collective. CTOR means the number of unique recipients who clicked on an email divided by unique emails opened. Both are typically multiplied by 100 to get to a percentage rate.

“You’ll find that every [email service provider] tracks data a little differently,” Oshinsky said. 

Inbound marketing software company Hubspot also defines CTR as the number of people who clicked on a link divided by the number of unique emails sent, multiplied by 100. And email marketing software company Campaign Monitor defines CTR as “the total number of clicks an email receives, and dividing that number by the total number of delivered messages.” Editor’s note: Campaign Monitor is a contracted vendor with Digiday.

Some may not “realize that one person’s CTR is another’s CTOR,” Oshinksy said. “Advertisers should really check with brands to make sure they clearly understand what data is being presented to them.”

This isn’t a new issue, according to Oshinksy. And media buyers that spoke with Digiday seem well aware of the discrepancies — though they don’t seem too concerned, given the fact that traditional newsletter publishers charge advertisers CPM rates based on total email subscriber count or open rates.

“A lot of times we will look at both [CTR and CTOR], because different partners will send different click-through rates,” said Katie Driggs, media director at ad agency FerebeeLane. “You have to know which it is because if you’re evaluating multiple vendors and e-newsletters against each other [and] you’re comparing one click-through rate to [another]. It’s not apples to apples. You have to make sure you know the methodology behind that click-through rate.”

Why buyers prefer CTOR 

Two media buyers told Digiday they’re aware newsletter platforms often calculate CTR based on open rates rather than on total email sends (even though Apple’s Mail Privacy Protection feature introduced last September has caused open rates to be viewed as unreliable).

Reporting CTOR as CTR in this way leads to a “higher” number that ESPs and newsletter authors can share with advertisers, Driggs said. “It looks better. So you would think [ESPs] would want to use that [number],” she said.

Driggs believes the industry needs to use standardized definitions for metrics like CTR and CTOR. “That would help prevent planners from using the wrong metrics against each other when comparing,” she said.

But it seems buyers actually prefer the way some ESPs are calculating CTR, by measuring clicks divided by opens — or “impressions,” said David Mirsky, group director at ad agency Crispin Porter + Bogusky — as opposed to total emails sent.

“A send is not an impression,” Mirsky said. “We hit delete for a lot of things we get all the time. Just because it went into my inbox doesn’t mean it was an impression that was served… An impression is when a pixel has been able to fire and register that a device has engaged or seen an ad.”

It’s the responsibility of media buyers to ensure they are comparing the same metrics against each other from different newsletter publishers, by asking email service providers and publishers’ sales reps for their methodologies to get to those numbers or asking for the raw numbers to calculate CTR and CTOR themselves, Driggs said. Buyers’ own third-party ad servers can also track ads in the newsletters. 

Sometimes, media buyers will ask newsletter publishers for a screenshot of their dashboards to see the raw numbers and different metrics, or for an export of their data to load into buyers’ own analytics dashboards, Driggs said. 

Mirsky said there are two main ways his team verifies that data. The “ideal” method is through third-party trackers, where emails can incorporate image pixels that fire as soon as an email is opened and content is generated for the ad. “It gives us a real-time read on when the impression is registered on it,” he said. 

But some newsletters don’t incorporate that technology and have static, pre-written ads, he said. In that case, Mirsky said they do rely on newsletter publishers to provide their engagement data, and his team can track site or purchase conversions from there. It requires trust between the buyer and the publisher, Mirsky said.

Publishers say CTR is just one metric of many

Boston Globe’s The B-Side, a newsletter for Gen Z and millennial audiences, is hosted on Beehiiv. Andrew Grillo, director of new product at the Boston Globe, said his team primarily tracks open rate and CTR to measure the performance of the newsletter.

“What percentage of audiences are in your newsletter, and then clicking to dive deeper into an article or a link – that really shows the intent of the reader,” Grillo said. “It’s the percent of our audience that is actually engaging with your newsletter.” The average click-through rate [Editor’s note: meaning click-to-open rate] on The B-Side’s Beehiiv dashboard was 9%, he said.

Jack Appleby moved his Future Social newsletter to Beehiiv after leaving Morning Brew six months ago. Appleby said he tracks performance by looking at subscriber growth, open rate and clicks, the same metrics advertisers ask him for. Appleby said he’s “never had anyone ask” him to define CTR or any of the other metrics he provides to advertisers, and that the data is typically taken “at face value.” 

Aazar Shad, who runs a few marketing and local news newsletters on Beehiiv, echoed Appleby’s description of the trust between newsletter publishers and advertisers when it comes to sharing their newsletter metrics, though both Appleby and Shad said advertisers sometimes do ask for screenshots of their analytics dashboards.

“Advertisers… care about how many people were converted,” Shad said. “They don’t care about what my numbers look like. My numbers vary every month and every post. If they get customers, if they get leads — as long as they get something tangible out of it, then they don’t care [about CTR definitions].”

Mailchimp reached out after this story’s publication to correct their original statement on their methodology to calculate CTR (which the company calls “click rate”). This story has been updated to reflect that Mailchimp calculates CTR based on emails sent, not emails opened.