Is there more to hidden traffic than 'dark social'?

In the keynote presentation at news:rewired on Friday (20 September), Jay Lauf, publisher of business news site Quartz, said an important question that publishers do not ask often enough is "where does our audience come from?".

He shared traffic source stats for Quartz, which is a year old this week and is averaging around 3.3 million unique users a month.

Between 10 and 15 per cent of traffic is from people coming directly to the homepage, and search engines provide around 20 per cent of traffic.

Social provides the majority of traffic to qz.com but Lauf divided this in to two categories: 'light social' and 'dark social'.

Light social is where referrer data is easy to spot in analytics and includes traffic from Facebook and Twitter. This constitutes 35 per cent of Quartz traffic.

Dark social is the more mysterious type of traffic where links are shared in a private yet social way, such as sent in an email, by instant messenger or in a text message. Between 25 to 30 per cent of Quartz traffic comes from dark social.

It was The Atlantic, also owned by Atlantic Media, which first identified dark social as a category. Senior editor Alexis Madrigal coined the term almost a year ago.

All publishers receive dark traffic, which shows as direct as there is no referrer data preserved, but now a data expert from analytics platform Parse.ly is proposing that it is not all social, but that "dark search" is a factor, plus traffic referred by secure sites.

What is dark social?

Look at your website's analytics and you will see 'direct' traffic showing as a source, in addition to referrers such as Google, Facebook and Twitter.

You may assume direct traffic is made up of people typing your site's homepage domain or a URL directly into a browser.

But how many readers will type a long article URL (such as www.journalism.co.uk/news/dark-social/a123456)? It is more likely that this link has been shared in a private network, such as by instant messenger or email, rather than typed.

As email clients and instant messaging platforms such as Skype do not show as referrers, links clicked on after being shared within these networks have had referral data stripped and show as direct traffic.

And if you regularly share such URLs with friends and colleagues via email and IM, you will be aware that this trading of links is common.

Alexis Madrigal realised dark social was a factor of direct traffic after a conversation with data experts at analytics platform Chartbeat. Chartbeat subsequently tweaked reporting so publishers which use the platform can work out which of the traffic is likely to be from dark social.

Dark mystery

But in the curious world of dark social perhaps not all is as it seems.

I think there is some hidden social activity going on in the direct category, but I think that other factors are actually causing it to become a bigger and bigger categoryAndrew Montalenti, Parse.ly

Andrew Montalenti, co-founder and chief technology officer at Parse.ly, an analytics platform which helps clients including Atlantic Media and Mashable "understand what their readers are interested in and why", thinks there are other 'dark' sources of traffic.

Parse.ly has data from around 160 million monthly unique visitors and about 5 billion monthly page views.

Montalenti has been thinking about dark social since it was first highlighted and believes "that the concept is quite wrong", he told Journalism.co.uk in an interview for this podcast.

"I think there is some hidden social activity going on in the direct category, but I think that other factors are actually causing it to become a bigger and bigger category," he said.

"I think that dark social is the wrong term," he added.

Dark search

Montalenti believes that some of this dark direct traffic is "dark search" from DuckDuckGo, a search engine that does not track users in the way Google does and strips all referral information.

"DuckDuckGo has had a big surge in popularity post the Snowden and NSA scandal as people are interested in searching online without being tracked, Montalenti said. "Publishers that get featured in that search engine still show up as direct traffic and it is not social."

Dark secure

Montalenti believes some direct traffic is due to changes in site security. You can spot a secure site as a padlock symbol appears in the browser when you type in a URL.

"Due to security concerns which now, with the NSA scandal, are heightened and on everyone's mind, a lot of sites started switching from an insecure mode of browsing, called HTTP, to a more secure mode of browsing, called HTTPS or secure HTTP."

That secure form of browsing strips referral information out to "respect people's privacy".

"As a result any link that is shared in a secure setting starts to show up as direct," he explained.

"This used to be a minor source of confusion in the direct category but as phishing scandals on the web, security scandals and privacy concerns have spread, people have increasingly made their sites secure by default."

One site which "should be on publishers' minds" is Hacker News. "This has been a very large start-up in the technology community and a big driver of traffic to publishers," Montalenti said.

And it is worth bearing in mind that news sites are the third biggest driver of traffic to other news sites, according to this Parse.ly study.

"A few months ago Hacker News switched the site to HTTPS and didn't change the hack so they could preserve the referrer. Suddenly all Hacker News pick-ups show up as direct for publishers, so this is a major effect on tech publishers in particular," Montalenti explained.

And in an email he sent to Journalism.co.uk following our interview Montalenti offered "more fodder" for his "dark social isn't really social" claim.

On 28 August 2013, Wikipedia "decided to 'go dark'", Montalenti explained, by switching browsing activity of logged in users to HTTPS.

This means that Wikipedia.org will not show as a referrer if the user is logged in and shows up in the direct category. It's "not social, yet it is certainly dark", Montalenti said.

Logged in users, those who edit Wikipedia pages, are no doubt a minor traffic source to most publishers at the current time, but the Wikimedia Foundation, publisher of Wikipedia, has made an official statement to say all users will eventually be switched to HTTPS.

That will see Wikipedia become part of the "dark direct" category, Montalenti said.

Testing the dark theory

When we approached Montalenti to speak to him about dark traffic, Parse.ly ran a test which found 12 per cent of traffic is "of the dark variety".

Analytics experts looked at section pages from a few major publishers. "We took sports section pages on the assumption that a sports section page would have a higher likelihood of actually being typed in," Montalenti explained.

"The assumption would be that if you have a favourite news site and you are a sports wonk you might more likely type in newssite.com/sports than you would any other section."

They then looked at the direct traffic category for all the posts within that section.

They established that 2 per cent of the traffic to a section page was typed in, as people had entered newssite.com/sports.

"Meanwhile 14 per cent to the traffic within those sections was marked 'direct', Montalenti said.

"You can assume that because those posts have very long URLs, definitely no more than 2 per cent of the traffic of those posts is type-in."

Montalenti believes that the rest is traffic of the "dark variety", which could include dark social, with the links being shared via IM and email, but that it probably includes some dark search (from DuckDuckGo), and dark secure (from logged in Wikipedia users and other secure sites).

"You can safely assume that 12 per cent of the traffic to these posts is this dark variety."

Free daily newsletter

If you like our news and feature articles, you can sign up to receive our free daily (Mon-Fri) email newsletter (mobile friendly).