In recent announcement Yahoo’s image community Flickr revealed the new feature – Similarity Search. It’s deep neural network based search engine which is designed to search across the Flickr directory for similar images as you ask for.
It seems to be somewhat like Google’s “reverse image search” for quite a long. With Google reverse image search, you can upload any photo from your computer to seek any similar photos on the web. Or you can even provide a direct web link to a photo instead. Where the Flickr’s “Similarity Search” won’t let you upload a photo or input any specific photo to search for.
With Flickr’s new feature, you to search further for similar photos to what’s in front of you. That is, enter a query text in Flickr search box, then in results, you can further use a new menu button on a photo with “…” called “Similarity Pivot”. That will discover more photos much similar to the photo you choose to pivot.
The similarity pivot in the menu is the major addition to the user experience. It offers a new way to explore and discover the billions of photos and millions of photographers on Flickr. The new system will allow users to search for particular sense and style of an image without even converting your thoughts into text to search. Hover on a desired photo in the result page and choose Similarity Pivot from the menu to find similar photos.
“It allows people to look for images of a particular style, it gives people a view into universal behaviors, and even when it “messes up,” it can force people to look at the unexpected commonalities and oddities of our visual world with a fresh perspective.” Flickr stated in a blog post.
The bad thing in Flickr’s similarity search is that you can only use it when you are on the search result page – as demoed above. Means that you can’t do it even with your own Flickr photos unless they appear in search results.
Similarity and Neural Network
Similarity in photos can be understood in various ways — two or more photos can contain a similar person, a similar thing, color or even a style. Observe the notion of similarity in each of the following four groups of photos depicting identity, semantic, texture and color similarities.
While Flickr showed interest in its blog post to use all of these approaches to do “similarity search”, it first preferred to go with semantic similarity — the one based on semantic content of photos. Flickr thinks it’s the best approach to facilitate Flickr community for similar photos discovery and implements deep neural networks to perform discovery. You can expect other approaches as well for later improvements.
Following flow of the system is easy to understand as how an image passes through Flickr’s neural network and are tagged with different elements from within the photo after identifying.
“We have been using deep neural networks at Flickr for a while for various tasks such as object recognition, NSFW prediction, and even prediction of aesthetic quality. For these tasks, we train a neural network to map the raw pixels of a photo into a set of relevant tags.”
With the semantic similarity, Flickr discards a lot of information from the image and only uses the tagged elements for further searching the similar images. In this way, Flickr ignores the colors, style or anything else but the tagged information. With the above example, Flickr will look out for any other images that contain the “hot air balloon”, the “sky”, “clouds”, and the “sunset”.
By looking at the illustration below, you can imagine, how the network output is then used to result the similar images. “Points in the neighborhood around the query image are semantically similar to the query image, whereas points in neighborhoods further away are not.”
Flickr also believes that this algorithm is not perfect and will be constrained by only tasks, the network is trained for. However, it’s enough effective for their purpose, Flickr says. “It contains information beyond merely the semantic content of the image, such as appearance, composition, and texture.”
The “Reverse Image Search” Services
There are several other services providing reverse image search most like Google’s including Bing images with different level of efficiency. Flickr might be calling its new feature as Similarity Search only because users do not upload or provide their desired image to look for. However the result of the services have much similarities.
Flickr’s similarity search, of course, doesn’t seem to be as robust as with Google, you can search for any photo across the web. Flickr currently limits you to perform the Similarity Search only on photos that are currently available on the community. Well that as well is a huge database of high quality photos from professional photographers.
Google also seems to use various approaches and not only semantic. Identity match can easily be seen if you upload a celeb photograph and check results. It even matches color, texture as well as semantic. On the other hand Bing images also doesn’t seem to be as good in identity match.
