Windows 10: Visual Search from Bing now lets you search what you see

Brink · Jun 21, 2018

Today we're launching new intelligent Visual Search capabilities that build upon the visual technology already in Bing so you can search the web using your camera. Now you can search, shop, and learn more about your world through the photos you take.

These new Visual Search capabilities are available today in the US on the Bing app for iOS and Android, and for Microsoft Launcher (Android only). They’ll also begin rolling out today for Microsoft Edge for Android, and will be coming soon to Microsoft Edge for iOS and Bing.com. Just click the camera button to get started:

For example, imagine you see a landmark or flower and want to learn more. Simply take a photo using one of the apps, or upload a picture from your camera roll. Bing will identify the object in question and give you more information by providing additional links to explore.

You can even shop from your photos for fashion and home furnishings. Let’s say you see a friend's jacket you like, but don't know its brand or where to purchase. Upload a pic into the app's search box and Bing will return visually-similar jackets, prices, and details for where to purchase.

[youtube]YAjUX-Px6Yc[/youtube]

We’ll be working hard over the coming months to add more capabilities to Visual Search, so your input on these features is greatly appreciated, as always. We hope you’re as excited by Visual Search as we are!

- The Bing Team
Click to expand...

Source: Visual Search from Bing now lets you search what you see | Bing Search Blog

:)

Brink · Jun 21, 2018

Introducing Bing Visual Search

Over the years people have come to expect search engines to automatically detect intent and provide great search results for text queries typed into a single search box. Now Bing takes the first step to achieve the same for images. The Bing Team sets out to connect your camera to a deep search experience.

We started several years ago by introducing ‘Search By’ Image capability. This allowed a user to specify an entire image to be used as a search query. But what if you only want to search for a certain object you saw in an internet image – or one you photographed? So far you'd have to jump through hoops to accomplish this - but not anymore. In this post we will share some recent work we’ve done to solve this challenge and some of the technical wizardry that made it possible.

Introducing Bing Visual Search

Let's say you are looking for kitchen decoration inspiration, and an image attracted your attention. You click on a thumbnail result to get to the ‘Detail View’. You really like the overall décor, but you are particularly interested in that nice-looking chandelier. Would it be possible to see where you can get one just like that? With Bing Visual Search, now you can!

In the Detail View, you will now see a magnifying glass symbol in the top left of the image. It is called the visual search button.

Clicking the visual search button displays a visual search box on the image. You can click and drag this box to adjust it to cover just the object of your interest. You can also simply draw a box around the chandelier if that’s more convenient.

Every time you adjust the visual search box, Bing instantly runs a visual search using the selected portion of the image as the query.

We realize that many Bing image search users may be shopping for items they see in the image or a similar product. We automatically detect the shopping intent and, in addition to regular image search, we also run a product search to find matching products:

Now you can simply click on the chandelier that is right for you, pick the best merchant on the detail page and finalize your purchase!

If you're not in a shopping mood after all, you can still click on “Related Images” to continue exploration of similar images. We are continuously working to detect more intents and bring the best information to the results to satisfy your search needs.

After you’ve found the perfect chandelier options for your project, it’s easy to conduct a visual search for other items. For example, in our example image you can select that beautiful bowl and find a similar one for your kitchen. You can search for any object you see in the image.

Visual search is in its infancy, and we are aware of cases where there is still room for improvement. For example, you may need to tweak your visual search box to fully capture the object of interest to get the best results.

All this goodness is available on your PC or mobile device by visiting Bing.com or in the Bing mobile app. Developers can build visual search into their app using Bing APIs as described here.

Please try out our visual search - just be careful as it can get quite addictive! Let us know what you think using Bing Listens or the Feedback button on Bing.

Under the hood

So how does it all work?

In text based search, the first step is query understanding. Similarly, in visual search, we first need to understand the query image. The major difference is that now instead of query terms we need another way to represent the query-image. Once the query understanding phase is complete, the subsequent steps in the execution of the query are similar:

As a first step in the query-image understanding process, we run Image Processing Service to perform object detection, extraction of various image features including DNN features, recognition features, and additional features used for duplicate detection.

Next comes text query inference step. Here we try to generate the best text query to represent the input image. This is based on a pipeline widely used in Bing image search to identify so called BRQs (for 'best representative query'). You can learn more about how this process works in our earlier post at: The Image Graph - Powering the Next Generation of Bing Image Search).

Subsequently we run a triggering model to identify different scenarios for search by image. Here we leverage the expertise already used in Bing answers triggering. For instance, if we detect that the query-image has the shopping intent, then we show rich segment specific experience.

Once the image understanding phase is complete, we enter the next step: matching. In order to implement search by image inside of existing Bing index serve stack, designed mostly for text search, we need to get text-like representation for the image feature vector. To accomplish this we employ a technique known in vision area as Visual Words. This technique allows us to quantize a dense feature vector into a set of discrete visual words, which are essentially a clustering of similar feature vectors into clusters, using the joint k-means algorithm. The visual words are then used to narrow down a set of candidates from billions to several millions.

After the matching step, we enter the stage of multilevel ranking. We need to rank millions of image candidates, and we’ll do that based on feature vector distances. To speed up the calculations, we use an innovative algorithm developed by Microsoft Research in collaboration with University of Science and Technology of China called Optimized Product Quantization (for more information see the paper: Optimized Product Quantization for Approximate Nearest Neighbor Search). The main idea is to decompose the original high-dimensional vector into many low-dimensional sub-vectors that are then quantized separately as pictured below.

After quantization is complete we need to calculate distances between the query-image and result-image vectors. Instead of using the usual Euclidean distance calculation, we perform a table lookup against a set of pre-calculated values to speed things up even further.

As a result of Optimized Product Quantization, we have reduced candidate set from millions to thousands. At this point we can perform a more expensive operation to rank the images more accurately. After multiple levels of ranking the de-duplication step is executed to remove any duplicate images from the results. The set of images we end up with after this step is the final result set that will be returned to the user.

What’s Next?

We’re constantly working to make the experience better. For example, you may soon notice that we will automatically help you pick objects without needing to draw a box, and provide other tools to help refine your search. We’re also continually focused on bringing the most comprehensive and highest quality visual search results.

Let us know what you think and stay tuned for more improvements!

- The Bing Team
Click to expand...

Source: Beyond text queries: Searching with Bing Visual Search | Search Quality Insights

rtwjunkie · Jun 21, 2018

What is your favorite search engine

Everyone knows how much I rail against MS. But I have to give them credit for Bing. It works the most relevant for me in my searches, and my search histories carry over to my Windowsphone.

Also, they actually list which search results were paid for to be in the top results (listing them as advertisements), unlike google, who is in litigation in EU for not being open about that shit.

Brink · Jun 21, 2018

Object Detection for Visual Search in Bing

In one of the recent blog entries we talked about how Bing Visual Search lets users search for images similar to an individual object manually marked in a given image (e.g. search for a purse shown in an image of your favorite celebrity). Since that post we took the functionality one step further: with our new Object Detection feature you don't have to draw the boxes manually anymore, Bing will do that for you.

Going back to our main scenario, imagine you're looking for outfit inspiration and you ended up in Bing Image Details Page looking at an interesting set. Bing automatically detects several objects and marks them, so you don't have to fiddle with the bounding box anymore. See below and note the 3 hotspots with light grey background and white boundary.

Now if you just click on the hotspot over an object of interest, Bing will automatically position the bounding box in the right place for that object and trigger a search, showing its results in Related Products and Related Images sections of the page.

To explore more, you can now adjust the box or try clicking on the other hotspots.

Here’s also a short video showing a live demo of Visual Search and Object Detection:

-w803it0LVU

We invite you to play around with this functionality but note that this is still an early version with limited coverage, currently only targeting certain categories of the fashion segment.
You are probably wondering how Object Detection was made possible, and that's what we cover in the next paragraphs.

Object Detection Model

The core of the underlying solution is the Object Detection Model. The goal of Object Detection is to find and identify objects in an image. We not only want to determine the category of the object that got detected but also its precise location and area occupied within the frame.

To get started we needed to define a set of object categories we would support. Based on user activity on Bing we noticed that fashion related searches were quite popular among our users. Thus, we decided to use that as an initial target. Having collected the training data, we got to the next step: training our models. Out of several DNN based object detection frameworks we evaluated for speed and accuracy balance, we selected the popular Faster R-CNN as the one best fitting our needs.

One crucial characteristic shared by most object detection algorithms is generation of category-independent region hypotheses for recognition, or “region proposals”. As compared to other frameworks where region proposals are generated offline, Faster R-CNN speeds the process up significantly enough for it to be done online. It achieves this by creatively sharing the full-image convolutional features between Region Proposal Network (RPN) and the detection network. Overall the structure of the Faster R-CNN network for object detection is as follows:

Product Deployment

Once we finalized the model structure we needed to work out the details of deployment. Given that Bing serves billions of users and that we show the results of object detection for every view of the detail page, providing smooth user experience under all conditions at sensible operational cost was no small challenge. Running the Faster R-CNN object detection using standard hardware was taking about 1.5 seconds per image. This was clearly not a workable solution.

The most obvious options to remedy the situation such as using asynchronous calls or offline pre-processing were not practical in this case: even with asynchronous initiation of the detection the user would still likely suffer a noticeable delay before the calculation completed. On the other hand pre-processing everything offline for the billions of images in Bing constantly changing index would take a really long time.

Luckily for us our partners at Azure were just testing new Azure NVIDIA GPU instances. We measured that the new Azure instances running NVIDIA cards accelerated the inference on detection network by 3x! Additionally, analyzing traffic patterns, we determined that a caching layer could help things even further. Our Microsoft Research friends had just the right tool for the job: a fast, scalable key-value store called ObjectStore. With the cache to store the results of object detection in place we were not only able to further decrease the latency but also save 75% of GPU cost.

Another way Azure helped us increase efficiency was through its elastic auto scaling. We always design our services to provide smooth experience even at peak loads. Such loads however don't occur very often. Traffic also fluctuates significantly between different hours of the day as well as between weekdays and weekends. Thanks to Azure Service Fabric, Microsoft's micro-service framework which we used to implement our Object Detection feature, we managed to make it reliable, scalable and cost efficient.

Ultimately with Azure, ObjectStore, and NVIDIA technology we built an object detection system which can handle full Bing index worth of data under peak traffic, provide great experience to users while making cost-effective use of resources.

What's Next?

We are constantly working on improving the precision and recall of the model by experimenting with even more sophisticated networks. We are also working on scaling up beyond the initial fashion categories, as well as expanding to other domains.

Please note that Object Detection is currently only available on the desktop with the mobile support still in the works. It is expected to be released for mobile in coming months.

Finally, note that Bing also supports Celebrity Recognition which to the user may look somewhat similar:

Celebrity Recognition however is based on a Face Detection Model which is different from the Object Detection Model discussed in this article, and it will be covered in one of the upcoming posts.

We hope that Bing Visual Search with Object Detection will make your visual explorations so much more enjoyable, streamlined and fruitful. Stay tuned for more improvements, and do let us know what you think using Bing Listens or the Feedback button on Bing!
Click to expand...

Source: Object Detection for Visual Search in Bing | Search Quality Insights

Windows 10: Visual Search from Bing now lets you search what you see

Visual Search from Bing now lets you search what you see

Visual Search from Bing now lets you search what you see

Visual Search from Bing now lets you search what you see

Visual Search from Bing now lets you search what you see - Similar Threads - Visual Search Bing

What color is you search?

Bing visual search has dissapeared from Windows 10

How do you disable Bing from the taskbar search?

Windows 10 search bar now with Bing Visual Search

Now you don't see me - and now you see

[REPORT] "Bing Is Pushing Malware When You Search for Chrome"

Now you see it, Now you don't

Introducing Bing Visual Search

Object Detection for Visual Search in Bing