Bots drove nearly 40% of internet traffic last year — and the naughty ones are getting smarter

Distil Networks today published its 2019 Bad Bot Report, which looks at the current state of automated web traffic, both good and bad. The paper highlights some startling figures, including that bots accounted for 37.9 percent of all online traffic during 2018. This is a admittedly a startling number, but it actually represents a welcome decrease on last year’s figures.

This decrease can be found across legitimate bot traffic (like, for example, the scrapers travel sites use to identify flight deals) as well as ill-intentioned bots (like those used to conduct denial-of-service attacks). These slumped by 14.4 percent and 6.4 percent respectively. The share of human traffic, on the other hand, grew by 7.5 percent to 62.1 percent.

It’s not clear why this shift took place, and Distil doesn’t try any qualitative analysis to figure out why. One plausible example could be the increasing levels of Internet access, largely coming from developed nations, which is bolstered by the proliferation of affordable smartphones and low-cost data plans.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

The paper does an excellent job of highlighting the challenges faced by those combating bot traffic. The current generation of bad bots are described as Advanced Persistent Bots (APBs), which have characteristics that make it difficult to mitigate against. APBs try to obfuscate their origin by relying upon anonymous proxies and other identity-hiding technologies, while simultaneously trying to appear to target sites as legitimate human traffic.

According to Distil, 49.9 percent of all bots appear as browsers running Google Chrome. A further 28.2 percent masquerade as other popular browsers, including Firefox, Internet Explorer, and Safari.

Surprisingly, 53.4 percent of all bad bot traffic comes from the United States, with the Netherlands, with 5.7 percent of all traffic, being the second most common country-of-origin. This is likely due to the proliferation of data centers and hosting providers in both countries.

Despite that, the most commonly IP-blocked countries are Russia and the Ukraine, highlighting the huge disparity in where people perceive attacks to originate from, rather than where they actually do.

These factors, in short, make bad bots harder to spot, and even harder to stop.

So, what are the targets of malicious bots? One of the most frequently attacked sectors is the ticketing industry. Distil’s 2019 Bad Bot Report says that 39.3 percent of all visits to ticketing websites come from bad bots. Furthermore, 24.1 percent of visits to ticketing sites come from the most sophisticated of bad bots.

It’s not hard to see why. By deploying bots, touts can acquire vast numbers of tickets to concerts and sporting events, which are sold for many times more than their list price. This is a big business. It’s evolved from sole-traders taking a punt on an event they think will be popular, to something that’s highly organized and technologically sophisticated.

Nations are also a common target. Distil Networks states that 29.9 percent of all traffic to government websites comes from bad bots, with voter registration pages among the most frequently targeted niches.

That makes sense. First-time voters tend to be younger. They often belong to minority ethnic groups. And, according to a 2018 Wall Street journal analysis of midterm polling data, they tend to lean towards more progressive candidates. If you want to influence the outcome of an election, a great way is to ensure certain voters don’t get to cast a ballot in the first place.

Distil Networks’ 2019 Bad Bots Report is available to read here. It offers a really interesting overview of the current bot landscape, how these nefarious programs work, and who they’re targeting. And, of course, if you want to read 2018’s report for comparison, you can find it here.

TNW Conference 2019 is coming! Check out our glorious new location, inspiring line-up of speakers and activities, and how to be a part of this annual tech extravaganza by clicking here.