MIT researchers create an algorithm that removes digicam photo reflections

Professional photographers understand the complexities of compensating for errant reflections when shooting a scene through a window. But what about the rest of us who are using arguably less than stellar equipment for the same kinds of shots?

MIT researchers are getting ready to present a new algorithm — at the Computer Vision and Pattern Recognition conference in June — that can automatically remove such reflections from images shot with a digital camera. The algorithm relies on photos showing two nearly identical, offset reflections.

Researcher, YiChang Shih, the first author of the paper, noted that the windows in Boston that he used were double-paned and that they produced double reflections.

“With that kind of window, there’s one reflection coming from the inner pane and another reflection from the outer pane. But thick windows are usually enough to produce a double reflection, too. The inner side will give a reflection, and the outer side will give a reflection as well.”

The double reflection is the key to this solution and relies on finding the edges of the reflection via a repeating offset pattern. Once the reflection is pinpointed, it can be reduced.

A new technique co-developed by Daniel Zoran and Yair Weiss of Hebrew University in Jerusalem created an algorithm that divides images into 8-by-8 blocks of pixels; for each block, it calculates the correlation between each pixel and each of the others. The aggregate statistics for all the 8-by-8 blocks in 50,000 training images proved a reliable way to distinguish reflections from images shot through glass.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

Shih and his colleagues searched for photos on Google and the Flickr using the terms, “window reflection photography problems” and turned up 96 images that had double reflections offset far enough for the algorithm to work.

What’s the point? Eventually, if such an algorithm is strengthened, it can be of widespread use in consumer digital photography to give people more control over their pictures, as well as assist robot vision technology for distinguishing glass reflections.

➤ Removing reflections from photos taken through windows