Windows 10: Microsoft Bing now delivers more visually immersive experiences

Brink · Mar 4, 2021

At Microsoft Bing we're looking for ways to give you back time so you can focus on the things that really matter. Weâre also looking for ways to move thoughtfully beyond a list of links, to a world of search results that seamlessly combine information with visually rich imagery in a single beautiful view.

With that in mind, weâre excited to announce several new search experiences that quickly deliver information in a way thatâs intuitive and engaging. The result is a visually rich format that allows you to quickly find what you are looking for without having to sift through large blocks of text. These new features are just the beginning as we look to evolve the search experience across all of Microsoft Bing!

Get inspired with intuitive, interactive search results

One of our goals is to deliver a search experience that moves beyond a static set of results. One way weâre doing so is to update our search result pages so that they adapt to intuitively highlight the content youâre engaging with as you go.

For instance, with people cooking from home more than ever, weâve heard many mention that they are overwhelmed by the sheer amount of recipe options online, and frustrated by recipes hidden below long introductions and ads. To help with that, weâve released an updated recipes experience that extracts and aggregates the most relevant recipes content and presents it in a single view on the search results page.

This view shows recipe pictures along with high-level information like calories per serving and user reviews. If a recipe includes a how-to video, weâll show that as part of the answer, and it will play if you hover over. Seamlessly click from one recipe to the next and see them expand inline so you don't have to bounce from page to page. The expanded view shows more information such as: an ingredient list and possible substitutions for when you donât have everything on hand, a drop-down menu for you to scale the recipe to a certain number of servings, and nutritional information. You can see this feature for yourself with queries like âfish tacos recipesâ or âeasy mac and cheese recipesâ to see how it can help you quickly decide on a recipe and get right to cooking.

We applied similar principles to search results for those looking to catch up on home improvement or for creative projects to do with their kids. Search for things like âDIY coffee tableâ and âvegetable garden ideasâ, and youâll get a rich view of results that can serve as a starting point for your inspiration. From there you can click an idea that catches your eye to visit its home site for more details, save it for later, or find other similar suggestions or ideas.

While exploring these visual results, you might see something you like but is difficult to describe with words. Thatâs why weâve also integrated visual search into the experience, so you can quickly search using an image. For example, in the âDIY coffee tableâ result, you may see wicker baskets that fit the tableâs look and feel. With just a click, youâll get image results of similar-looking items, and can directly click off to retailer sites to purchase a particular basket if youâre sold on it.

Weâve also made a few updates to our carousel experiences, with intuitive hover-over experiences that can help you, for example, quickly organize your book or movie lists. To see it in action, search for things like âchef documentariesâ or âspace booksâ, and youâll get a rich carousel of answers that showcases just the resultsâ high-level information to avoid crowding the page. If you decide you want to learn more about a specific result, hover over it, and Microsoft Bing will then expand the result with more detailed information.

Quickly get informed with infographic-inspired experiences

Another part of our vision for modern search is to thoughtfully aggregate information for more detailed topics, and to design the page in such a way that you can quickly find what youâre looking for. Our goal is to provide both style AND substance.

For example, you can search for broad topics like âKenyaâ or âgiraffesâ. Results for these move beyond a dense summary of text results and blue links, and aggregate the top text and visual results into an infographic-inspired experience. Curiosity piqued? Click off to sites that contain more detail or explore other related topics. You can see the updated experience alongside the previous experience below:

Another feature that showcases this design approach is our local answers, which can help if, for example, youâre trying to stay close to home and want to be a tourist in your own hometown. Search for things like âthings to do in Seattleâ or âEiffel Towerâ, and youâll get aggregated information from a variety of sources. Instead of just a single carousel of images, or just a text summary of one aspect of what youâve searched for, you will now see a comprehensive overview of the topic â inclusive of Bing Maps, top images, visitor reviews, and more â so in a single view you can learn more about the area and focus on whatever is of interest to you. This experience is flighting and will be fully available in the US in coming weeks.

Weâve applied similar principles to more list-centric queries like âbest rivers to visitâ or âmost interesting volcanoesâ. Instead of getting just a text result, or information on a single answer, you get an explorable overview of relevant image-led content.

We hope youâre as excited about these new experiences as we are! Stay tuned in the coming months for more visually immersive experiences across the web. As always, you can provide feedback by clicking the feedback button on the lower right corner of the results page.
Click to expand...

Source: https://blogs.bing.com/search/2021_0...-save-you-time

:)

Brink · Mar 4, 2021

Bing delivers text-to-speech, intelligent answers, and visual search

At NVIDIA’s GPU Technology Conference this week, Bing demonstrated natural sounding text-to-speech AI, expanded intelligent answers, and the ability to quickly see multiple objects auto-detected within an image to search for visual matches. All these features help you find what you’re looking for faster and are powered via Azure virtual machines running on NVIDIA GPUs optimized with NVIDIA CUDA-X AI software libraries.

Text-to-speech

The updated Bing app can now change text to speech, meaning Bing can speak answers to your queries back to you with a voice that’s nearly indistinguishable from a human’s. This advance was made possible by breakthroughs in deep neural networks that give our AI human-like intonation and clear articulation of words. In addition to improvement to our conversational AI, this capability as a real-time service would not be possible without the higher processing power of NVIDIA GPUs.

The Bing app also supports speech as an input, meaning you can speak to your mobile device and Bing will change your spoken word to text and search that query for you. Simply press the microphone button on the app homepage, speak your question, and you’ll get search results.

Intelligent answers

Bing intelligent answers allow you to get comprehensive, summarized information aggregated across several sources in response to certain queries.

We’re now taking intelligent answers one step further by advancing our deep learning models. These models require a lot of processing power, so we’re leveraging recent advances in GPU technology that allow us to process entire web pages much faster and more efficiently than traditional models powered by CPUs. This advance allows us to provide answers for harder questions than ever before. For example, instead of the relatively simple answer to ‘what is the capital of Bangladesh’, Bing can now provide answers to more complex questions, such as ‘what are different types of lighting for a living room’, quicker than before.

Visual search

Visual search is another area in which recent developments have enabled huge strides in efficiency and coverage.

Visual search allows you to search using an image. For example, if you see an image of an accent light you like, Bing can show visually-similar decor and even show purchase options at different price points if the item is available online. To save you time, visual search also automatically detects and places clickable hotspots over important objects you may want to search for next.

Our advanced visual search capabilities such as object detection are quick and automatic using NVIDIA GPUs for inferencing, which have yielded dramatic processing efficiency compared to CPU-powered inference, hence unlocking this scenario for our customers.

New intelligent scenarios powered by Azure and NVIDIA

These new scenarios are all made possible by Bing powered by Azure N-series virtual machines running NVIDIA GPUs. Text-to-speech, speech-to-text, instant answers, and visual search are all part of the next great search frontier, and we’re very excited to see what our partnership continues to enable in the future.
Click to expand...

Source: https://blogs.bing.com/search/2019-0...-visual-search

Brink · Mar 4, 2021

Bing delivers text-to-speech, intelligent answers, and visual search

At NVIDIA’s GPU Technology Conference this week, Bing demonstrated natural sounding text-to-speech AI, expanded intelligent answers, and the ability to quickly see multiple objects auto-detected within an image to search for visual matches. All these features help you find what you’re looking for faster and are powered via Azure virtual machines running on NVIDIA GPUs optimized with NVIDIA CUDA-X AI software libraries.

Text-to-speech

The updated Bing app can now change text to speech, meaning Bing can speak answers to your queries back to you with a voice that’s nearly indistinguishable from a human’s. This advance was made possible by breakthroughs in deep neural networks that give our AI human-like intonation and clear articulation of words. In addition to improvement to our conversational AI, this capability as a real-time service would not be possible without the higher processing power of NVIDIA GPUs.

The Bing app also supports speech as an input, meaning you can speak to your mobile device and Bing will change your spoken word to text and search that query for you. Simply press the microphone button on the app homepage, speak your question, and you’ll get search results.

Intelligent answers

Bing intelligent answers allow you to get comprehensive, summarized information aggregated across several sources in response to certain queries.

We’re now taking intelligent answers one step further by advancing our deep learning models. These models require a lot of processing power, so we’re leveraging recent advances in GPU technology that allow us to process entire web pages much faster and more efficiently than traditional models powered by CPUs. This advance allows us to provide answers for harder questions than ever before. For example, instead of the relatively simple answer to ‘what is the capital of Bangladesh’, Bing can now provide answers to more complex questions, such as ‘what are different types of lighting for a living room’, quicker than before.

Visual search

Visual search is another area in which recent developments have enabled huge strides in efficiency and coverage.

Visual search allows you to search using an image. For example, if you see an image of an accent light you like, Bing can show visually-similar decor and even show purchase options at different price points if the item is available online. To save you time, visual search also automatically detects and places clickable hotspots over important objects you may want to search for next.

Our advanced visual search capabilities such as object detection are quick and automatic using NVIDIA GPUs for inferencing, which have yielded dramatic processing efficiency compared to CPU-powered inference, hence unlocking this scenario for our customers.

New intelligent scenarios powered by Azure and NVIDIA

These new scenarios are all made possible by Bing powered by Azure N-series virtual machines running NVIDIA GPUs. Text-to-speech, speech-to-text, instant answers, and visual search are all part of the next great search frontier, and we’re very excited to see what our partnership continues to enable in the future.
Click to expand...

Source: https://blogs.bing.com/search/2019-0...-visual-search

Brink · Mar 4, 2021

Introducing Bing Visual Search

Over the years people have come to expect search engines to automatically detect intent and provide great search results for text queries typed into a single search box. Now Bing takes the first step to achieve the same for images. The Bing Team sets out to connect your camera to a deep search experience.

We started several years ago by introducing ‘Search By’ Image capability. This allowed a user to specify an entire image to be used as a search query. But what if you only want to search for a certain object you saw in an internet image – or one you photographed? So far you'd have to jump through hoops to accomplish this - but not anymore. In this post we will share some recent work we’ve done to solve this challenge and some of the technical wizardry that made it possible.

Introducing Bing Visual Search

Let's say you are looking for kitchen decoration inspiration, and an image attracted your attention. You click on a thumbnail result to get to the ‘Detail View’. You really like the overall décor, but you are particularly interested in that nice-looking chandelier. Would it be possible to see where you can get one just like that? With Bing Visual Search, now you can!

In the Detail View, you will now see a magnifying glass symbol in the top left of the image. It is called the visual search button.

Clicking the visual search button displays a visual search box on the image. You can click and drag this box to adjust it to cover just the object of your interest. You can also simply draw a box around the chandelier if that’s more convenient.

Every time you adjust the visual search box, Bing instantly runs a visual search using the selected portion of the image as the query.

We realize that many Bing image search users may be shopping for items they see in the image or a similar product. We automatically detect the shopping intent and, in addition to regular image search, we also run a product search to find matching products:

Now you can simply click on the chandelier that is right for you, pick the best merchant on the detail page and finalize your purchase!

If you're not in a shopping mood after all, you can still click on “Related Images” to continue exploration of similar images. We are continuously working to detect more intents and bring the best information to the results to satisfy your search needs.

After you’ve found the perfect chandelier options for your project, it’s easy to conduct a visual search for other items. For example, in our example image you can select that beautiful bowl and find a similar one for your kitchen. You can search for any object you see in the image.

Visual search is in its infancy, and we are aware of cases where there is still room for improvement. For example, you may need to tweak your visual search box to fully capture the object of interest to get the best results.

All this goodness is available on your PC or mobile device by visiting Bing.com or in the Bing mobile app. Developers can build visual search into their app using Bing APIs as described here.

Please try out our visual search - just be careful as it can get quite addictive! Let us know what you think using Bing Listens or the Feedback button on Bing.

Under the hood

So how does it all work?

In text based search, the first step is query understanding. Similarly, in visual search, we first need to understand the query image. The major difference is that now instead of query terms we need another way to represent the query-image. Once the query understanding phase is complete, the subsequent steps in the execution of the query are similar:

As a first step in the query-image understanding process, we run Image Processing Service to perform object detection, extraction of various image features including DNN features, recognition features, and additional features used for duplicate detection.

Next comes text query inference step. Here we try to generate the best text query to represent the input image. This is based on a pipeline widely used in Bing image search to identify so called BRQs (for 'best representative query'). You can learn more about how this process works in our earlier post at: The Image Graph - Powering the Next Generation of Bing Image Search).

Subsequently we run a triggering model to identify different scenarios for search by image. Here we leverage the expertise already used in Bing answers triggering. For instance, if we detect that the query-image has the shopping intent, then we show rich segment specific experience.

Once the image understanding phase is complete, we enter the next step: matching. In order to implement search by image inside of existing Bing index serve stack, designed mostly for text search, we need to get text-like representation for the image feature vector. To accomplish this we employ a technique known in vision area as Visual Words. This technique allows us to quantize a dense feature vector into a set of discrete visual words, which are essentially a clustering of similar feature vectors into clusters, using the joint k-means algorithm. The visual words are then used to narrow down a set of candidates from billions to several millions.

After the matching step, we enter the stage of multilevel ranking. We need to rank millions of image candidates, and we’ll do that based on feature vector distances. To speed up the calculations, we use an innovative algorithm developed by Microsoft Research in collaboration with University of Science and Technology of China called Optimized Product Quantization (for more information see the paper: Optimized Product Quantization for Approximate Nearest Neighbor Search). The main idea is to decompose the original high-dimensional vector into many low-dimensional sub-vectors that are then quantized separately as pictured below.

After quantization is complete we need to calculate distances between the query-image and result-image vectors. Instead of using the usual Euclidean distance calculation, we perform a table lookup against a set of pre-calculated values to speed things up even further.

As a result of Optimized Product Quantization, we have reduced candidate set from millions to thousands. At this point we can perform a more expensive operation to rank the images more accurately. After multiple levels of ranking the de-duplication step is executed to remove any duplicate images from the results. The set of images we end up with after this step is the final result set that will be returned to the user.

What’s Next?

We’re constantly working to make the experience better. For example, you may soon notice that we will automatically help you pick objects without needing to draw a box, and provide other tools to help refine your search. We’re also continually focused on bringing the most comprehensive and highest quality visual search results.

Let us know what you think and stay tuned for more improvements!

- The Bing Team
Click to expand...

Source: Beyond text queries: Searching with Bing Visual Search | Search Quality Insights

Windows 10: Microsoft Bing now delivers more visually immersive experiences

Microsoft Bing now delivers more visually immersive experiences

Microsoft Bing now delivers more visually immersive experiences

Microsoft Bing now delivers more visually immersive experiences

Microsoft Bing now delivers more visually immersive experiences - Similar Threads - Microsoft Bing delivers

Bing is now officially called Microsoft Bing

Windows 10 search bar now with Bing Visual Search

New Twitter.com design experiment now available for more people

Bing Ads is now Microsoft Advertising

Bing delivers text-to-speech, intelligent answers, and visual search

Microsoft Teams delivers new experiences for the intelligent workplace

Delivering a faster and more responsive Outlook.com

Delivering a more beautiful OneDrive

Delivering a single unified Store experience in Windows 10