DataSift launches two new ways for anyone to mine and filter social data

For those drowning in data, this week provides good news from DataSift, the social data platform company. The firm is launching Push and Query Builder to make it simple for almost anyone to mine and  filter through data across the social Web.

The company’s Push and Query Builder tools allow enterprises to easily create filters and combine data with their own enterprise tools to quickly analyze and make sense of important data about their customers, competitors and markets that are contained in social data sources.

Both of these developments have been made possible by DataSift’s unique cloud infrastructure, which has been architected and built specifically to handle massive volumes of real-time and historical Social Data.

So what are these new magical tools? Let’s take a look.

Push is a social big data delivery into business intelligence (BI) platforms. It’s a new system that gives companies more control over the data they receive and makes it easier to integrate and analyse that data alongside their own.

Push has been designed to integrate with existing BI applications, databases, data warehousing platforms and other cloud services, including Amazon DynamoDB, Amazon S3, MongoDB, CouchDB, FTP/SFTP, ElasticSearch and WebHooks.

“Moving Big Data around the Internet is non-trivial and we spent nine months building out an infrastructure that is capable of easily delivering terabytes of data,” said DataSift Founder & CTO, Nick Halstead. “We guarantee delivery of that data with ‘Push’ and already have connectors with many popular cloud services, and expect many more to be released in the coming months.”

Users can log in, set up feeds and then manage when, where and how the data will be pushed. Basically this helps when there is an overwhelming amount of data to work with and social media now kicks up so much data that there can be simply too much to cope with in real-time.

“We’ve been blown away by the amount of social data. When Mark Zuckerberg said the amount of data people are sharing is doubling every year, we thought that was maybe a bit aggressive but we’re finding that to be the case,” comments DataSift CEO, Rob Bailey. “Not only has there been incredible growth from primary networks like Facebook and twitter but also an explosion of creativity in smaller but fast growing networks like Pinterest.”

Graphical queries

Query Builder provides a simple, graphical interface within DataSift to create sophisticated filters that mine social Web data both real-time and historically all without needing to learn a programming language.

Available in September, the service is open source so that it may be embedded into customer products providing enterprises with the flexibility to create queries that are as simple or complex as required.

Query Builder was written in HTML5 and Javascript so the interface is  configurable and users can add and remove functions or data sources in a tailored fashion. Basically it provides nice big icons and simple methods of creating filters to get the results you want and pretty much anyone can understand the process.

“It used to be that if someone wanted to run queries on large amounts of data they would have to go out and hire expensive data scientists,” says Bailey. “Now they can do it for themselves. It’s very quick and easy and we’re expanding the audience of who can work with social data.”

DataSift aggregates billions of social interactions from more than a million public sources such as Twitter, Facebook, YouTube, blogs, forums and online message boards. With real-time and historical data on tap, that can be an intimidating load to work with if you’re not a data scientist.

Democratising the process means that new questions may arise from data when users can clearly work with it in a manageable way. It is unlikely to put the real number crunching minds out of a job, but it does give the rest of us a way to spot insights and trends that relate to markets, news, brands and public opinion all of which are useful when analysing the best ways to remain competitive.

If you like noodling about with data, it even works on a tablet. Take a look at the video below where Halstead sets up filters on an iPad.

DataSift launched in November last year and has been building on its firm infrastructure ever since. In May this year the firm raised $7.2m from existing backers GRP Partners and IA Ventures to help accelerate growth. In August it opened an office in New York to work closely with the US finance industry.

The company works with Fortune 500, social media monitoring, BI companies and news organisations to build applications that use social data. It takes care of the heavy lifting for enriching, cleaning and archiving the data so that it can be exported in a number of ways.

As the amount of social data continues to explode, making the process of analysis easier can only work in DataSift’s favour.

Image Credit: JD Hancock