Samsung Unboxes Visual Search & Machine Learning with Bixby

Samsung Unboxes Visual Search & Machine Learning with Bixby

Samsung unveiled their new ultra sleek S8 & S8 + smartphone earlier this week in New York, featuring tons of innovation that will soon be the critical building blocks for the coming wave of Mixed Reality.  Many of the capabilities featured in the new Galaxy S8 & S8 + will start to show up in Augmented Reality optics. When worn these glasses will turn our physical world into an endless source of digital information through the implementation of eye tracking, facial recognition, AI & AR. Having the Galaxy S8 & S8+ tuned up with so many of these advanced features, marks the beginning of one of the largest and most exciting digital transformations of all time which will bridge our digital and physical worlds together seamlessly. The Galaxy S8 and S8+ is by far one of the most advanced smartphones the world has ever seen.

At first glance, the curved “infinity display” on the 5.8-inch S8 and the 6.2-inch S8+ is simply stunning along with other key features like “Dex” which turns the phone into a desktop simply by docking it in a cradle and syncing to a monitor and wireless keyboard, offering an incredible mobility solution for frequent business travelers. But what really caught my immediate attention and excitement was Bixby, Samsung’s new Artificial Intelligence assistant which was created by Viv Labs (a startup founded by the creators of Siri that Samsung bought last year).

What is Bixby?

Bixby was created to provide a more personalized experience to enhance the way people interact with their phones and the world around them. Simply put, Bixby is not just an ordinary voice assistant, but rather an intuitive and comprehensive interface that leverages contextual awareness to learn users’ habits and respond accordingly.

Bixby offers four main features which include Voice, Vision, Reminder and Home.

Bixby’s Machine Learning & Artificial Intelligence:

With a friendly new interface that solves the problem of complexity, Bixby takes an approach that instead of humans learning how machines interact with the world, your machine (or smartphone) will now learn and adapt to your behavior.

Bixby’s Home screen lets users see the content they care most about with just a single swipe or a short press of the designated Bixby button.

But Home doesn’t just relay information; it puts it into context, paying attention to the user’s schedule, routine and personality each time it’s used. In this way, Bixby can actually anticipate what the user needs and respond accordingly. For example, various cards appear that will reveal meaningful information a user accessed from apps like Gmail, Facebook or Uber.

Bixby would then be able to organize the cards depending on the time of day. If a user commonly schedules an Uber for a ride to work in the morning for example, the Uber card would show up at the top of a group of cards every morning, along with morning news and weather. Or, Bixby could provide a reminder to read a document later when the user arrives to the office.

Sriram Thodla, senior director of intelligence at Samsung Electronics America explained that with other AI assistants, voice is “isolated and doesn’t understand what’s on a phone’s screen.” Bixby, knows the context and what’s on a screen and has the ability to”move seamlessly between voice and touch.”

In another example, to send directions to a contact named Cindy, Thodla opened up a maps application, then pressed the Bixby physical button on the side of the device and said, “Capture this and send to Cindy.” With multiple Cindy’s in his phone’s directory, he was then directed to touch the right one and off went the directions.

Visual Search:

Bixby is more than just a voice-powered AI assistant, the lens of the phones camera will unlock visual intelligence and place the internet on real world physical objects, eliminating the need for a search box. Through partnerships with Pintrest, Foursquare, Amazon, Google Translate & Vivino. Users can now point their phones at objects like a bottle of wine, a pair of sneakers, a landmark or even a building and Bixby will display recommendations and contextual information. Shopping for similar products or nearby locations of interest are now easily searchable in a frictionless way. Pinterest’s data and image recognition technology will be beneficial along with Foursquares massive datasets of location recommendations. I’m expecting the list of partners will continue to grow and evolve, offering an incredible ecosystem of visual search & AR experiences in the near future.

Landmarks & Location:

Traveling abroad and need some information on local landmarks or nearby restaurants? Bixby has got that covered. Bixby will even translate the menu for you in up to 8 different languages!

Object Recognition & Shopping:

Now objects and products are easier and faster to buy through Bixby. Simply snap a photo of a product and get instant shopping results through Amazon and other partners.

Vision can also be activated to read QR codes and, in select markets, barcodes that direct users to related information.

Samsung plans to eventually release a tool (in SDK) to enable third-party developers to make their applications and services Bixby-enabled easily.

Bixby & Samsung Connected Home:

Bixby will also connect to smart devices in your home, using the new Samsung Connect Home router which turns the phone into the universal remote for your life.

In the future we will start to see Bixby gradually integrated into all of Samsung’s IoT connected products. Soon you will be able to control your air conditioner, washing machine or TV through Bixby. And because it is a cloud service, all you need is an internet connection and a mic and Bixby will be set to task.

The Samsung Galaxy S8 and S8+ release date is set for April 21 in the US, and April 28 in the UK and the rest of Europe.

This post is part of our contributor series. It is written and published independently of TNW.

Read next: WhatsApp Messenger for Java Phones