In Asia, take comScore data with a huge grain of salt

Asia is gaining a reputation for the growth in the technology and Internet industries – and the numbers don’t lie either. As with any industry, most of the analysis in the region is derived from various forms of analytics and reports from research firms.

comScore is one such example, and it’s one of the world’s most influential research and analysis firms when it comes to technology. It is faced, however, with a critical challenge that questions the accuracy of its data.

In its reports, comScore clearly states that data relating to Internet traffic:

“Excludes visitation from public computers such as Internet cafes or access from mobile phones and PDAs.”

This caveat has prompted a small discussion among a group of tech pundits, mostly specializing in the Asian market, because in this region public computers and mobile Internet are important access points for users and missing out on this data makes the reliability and usefulness of the reports questionable.

Leading the discussion is Asia Correspondent’s Jon Russell, joined by Singapore-based social media analyst Simon Kemp and myself, among others, as we attempt to analyze the reliability of comScore data in Asia.

The Complex Asian Market

The problem is not exclusive to the Asian market, but in a region where mobile phones and public Internet access points play such a key part in bringing Web access to the masses, the reach of comScore’s research – i.e. fixed-line home Internet subscribers – leaves a lot of activity unaccounted for.

According to a mobiThinking report, the world’s most populous nations have the most mobile subscriptions, with two Asian countries, China and India, leading the growth. Additionally, it may come as no surprise to learn that there are more mobile Internet users in China than in any other country. The China Internet Network Information Center (CINIC) reports that as of July 2010, there were 277m mobile Web users in China, representing almost two-thirds of the country’s Internet users.

Public access points such as Internet cafes are also huge in Asia. According to a report, in China, the world’s single biggest Internet population, 20% of users access the Web in Internet cafes. And in Indonesia, where Internet uptake is starting to flourish and where Facebook has its second largest user-base, analyst firm Saling Silang claim that 83% of Indonesians use Internet cafes and 22% access the web through a smartphone. A similar picture exists across other Asian countries such as Malaysia, Taiwan, South Korea and the Philippines.

The list could go on but it only points to one thing: comScore data in Asia should be taken with a grain of salt.

Case Study: The Popularity of Social Networks in China

We will further illustrate these data discrepancies by comparing figures side by side, as compiled by Jon Russel, in a small case study. In an attempt to study Internet activity in China, particularly in social networking, we are given two opposing results coming from different analysis firms.

According to comScore, Internet usage in China is led by web portals, and social networks remain a smaller segment of the market. Based on its figures, social networks are used by 17.8% of Internet users, accounting for just 5.5% of time spent online, which gives the impression that social networking is more of a niche market in China.

On the other hand, we are presented with a completely different report from the Data Center of the Chinese Internet (DCCI), which claims that Chinese people now spend 41% of their online time on social networks, overtaking news, search, and videos – this indicates a major shift in how Chinese ‘netizens’ are no longer purely consumers of web content, but are now more open to communicating, sharing, and engaging online as well.

Russell states in his study:

“While we cannot be entirely sure what factors account for the vastly different projections of China’s digital landscape, and the role of social media, from these two reports (and it’s true that I do not have the raw data from DCCI) it is clear that the data from comScore does not include any Internet traffic from public networks – such as Internet cafes – or mobile devices.”

comScore’s Methodology

Jon told us that he contacted comScore regarding this matter, and he got this reply:

“The comScore panel is based on a 2 million personal panel globally that opt in to let us measure their online usage from home and work locations. Through this method we are able to accurately measure user visitation, engagement and specifically user demographics.

comScore data is accurate for the universe that we measure in Asia, which currently is home and work computer usage for those visitors age 15+. The reality of any market research method is that there may be limitations to what can be measured, but comScore data is the global standard for digital media measurement and uses the most robust methodology on numerous accounts.”

comScore Data: Misleading

Internet data can be tracked, gathered and interpreted in so many ways and it’s not an easy thing to do. I don’t intend to say that comScore data is absolutely useless. It’s a reputable company that, as they say, is able to accurately measure user visitation and engagement to some degree. However, given the significant discrepancies between results, as seen in the case study, its data is certainly misleading.

comScore makes the limitations of its data clear, so the issue is turned and thrown back to journalists, reporters and bloggers to be aware of this and ensure that inaccurate analysis of data is avoided, at least until comScore optimizes the way it collects its data.

Of course, the desired end result from all this is for comScore to include mobile and public access points in its research methodology. Asia is developing into a key growth market across many digital industries, and it’s imperative that comScore reconsiders its current research methodology.