Windows 10: Bing delivers text-to-speech, intelligent answers, and visual search

Brink · Mar 20, 2019

At NVIDIA’s GPU Technology Conference this week, Bing demonstrated natural sounding text-to-speech AI, expanded intelligent answers, and the ability to quickly see multiple objects auto-detected within an image to search for visual matches. All these features help you find what you’re looking for faster and are powered via Azure virtual machines running on NVIDIA GPUs optimized with NVIDIA CUDA-X AI software libraries.

Text-to-speech

The updated Bing app can now change text to speech, meaning Bing can speak answers to your queries back to you with a voice that’s nearly indistinguishable from a human’s. This advance was made possible by breakthroughs in deep neural networks that give our AI human-like intonation and clear articulation of words. In addition to improvement to our conversational AI, this capability as a real-time service would not be possible without the higher processing power of NVIDIA GPUs.

The Bing app also supports speech as an input, meaning you can speak to your mobile device and Bing will change your spoken word to text and search that query for you. Simply press the microphone button on the app homepage, speak your question, and you’ll get search results.

Intelligent answers

Bing intelligent answers allow you to get comprehensive, summarized information aggregated across several sources in response to certain queries.

We’re now taking intelligent answers one step further by advancing our deep learning models. These models require a lot of processing power, so we’re leveraging recent advances in GPU technology that allow us to process entire web pages much faster and more efficiently than traditional models powered by CPUs. This advance allows us to provide answers for harder questions than ever before. For example, instead of the relatively simple answer to ‘what is the capital of Bangladesh’, Bing can now provide answers to more complex questions, such as ‘what are different types of lighting for a living room’, quicker than before.

Visual search

Visual search is another area in which recent developments have enabled huge strides in efficiency and coverage.

Visual search allows you to search using an image. For example, if you see an image of an accent light you like, Bing can show visually-similar decor and even show purchase options at different price points if the item is available online. To save you time, visual search also automatically detects and places clickable hotspots over important objects you may want to search for next.

Our advanced visual search capabilities such as object detection are quick and automatic using NVIDIA GPUs for inferencing, which have yielded dramatic processing efficiency compared to CPU-powered inference, hence unlocking this scenario for our customers.

New intelligent scenarios powered by Azure and NVIDIA

These new scenarios are all made possible by Bing powered by Azure N-series virtual machines running NVIDIA GPUs. Text-to-speech, speech-to-text, instant answers, and visual search are all part of the next great search frontier, and we’re very excited to see what our partnership continues to enable in the future.
Click to expand...

Source: https://blogs.bing.com/search/2019-0...-visual-search

:)

Brink · Mar 20, 2019

Introducing Bing Visual Search

Over the years people have come to expect search engines to automatically detect intent and provide great search results for text queries typed into a single search box. Now Bing takes the first step to achieve the same for images. The Bing Team sets out to connect your camera to a deep search experience.

We started several years ago by introducing ‘Search By’ Image capability. This allowed a user to specify an entire image to be used as a search query. But what if you only want to search for a certain object you saw in an internet image – or one you photographed? So far you'd have to jump through hoops to accomplish this - but not anymore. In this post we will share some recent work we’ve done to solve this challenge and some of the technical wizardry that made it possible.

Introducing Bing Visual Search

Let's say you are looking for kitchen decoration inspiration, and an image attracted your attention. You click on a thumbnail result to get to the ‘Detail View’. You really like the overall décor, but you are particularly interested in that nice-looking chandelier. Would it be possible to see where you can get one just like that? With Bing Visual Search, now you can!

In the Detail View, you will now see a magnifying glass symbol in the top left of the image. It is called the visual search button.

Clicking the visual search button displays a visual search box on the image. You can click and drag this box to adjust it to cover just the object of your interest. You can also simply draw a box around the chandelier if that’s more convenient.

Every time you adjust the visual search box, Bing instantly runs a visual search using the selected portion of the image as the query.

We realize that many Bing image search users may be shopping for items they see in the image or a similar product. We automatically detect the shopping intent and, in addition to regular image search, we also run a product search to find matching products:

Now you can simply click on the chandelier that is right for you, pick the best merchant on the detail page and finalize your purchase!

If you're not in a shopping mood after all, you can still click on “Related Images” to continue exploration of similar images. We are continuously working to detect more intents and bring the best information to the results to satisfy your search needs.

After you’ve found the perfect chandelier options for your project, it’s easy to conduct a visual search for other items. For example, in our example image you can select that beautiful bowl and find a similar one for your kitchen. You can search for any object you see in the image.

Visual search is in its infancy, and we are aware of cases where there is still room for improvement. For example, you may need to tweak your visual search box to fully capture the object of interest to get the best results.

All this goodness is available on your PC or mobile device by visiting Bing.com or in the Bing mobile app. Developers can build visual search into their app using Bing APIs as described here.

Please try out our visual search - just be careful as it can get quite addictive! Let us know what you think using Bing Listens or the Feedback button on Bing.

Under the hood

So how does it all work?

In text based search, the first step is query understanding. Similarly, in visual search, we first need to understand the query image. The major difference is that now instead of query terms we need another way to represent the query-image. Once the query understanding phase is complete, the subsequent steps in the execution of the query are similar:

As a first step in the query-image understanding process, we run Image Processing Service to perform object detection, extraction of various image features including DNN features, recognition features, and additional features used for duplicate detection.

Next comes text query inference step. Here we try to generate the best text query to represent the input image. This is based on a pipeline widely used in Bing image search to identify so called BRQs (for 'best representative query'). You can learn more about how this process works in our earlier post at: The Image Graph - Powering the Next Generation of Bing Image Search).

Subsequently we run a triggering model to identify different scenarios for search by image. Here we leverage the expertise already used in Bing answers triggering. For instance, if we detect that the query-image has the shopping intent, then we show rich segment specific experience.

Once the image understanding phase is complete, we enter the next step: matching. In order to implement search by image inside of existing Bing index serve stack, designed mostly for text search, we need to get text-like representation for the image feature vector. To accomplish this we employ a technique known in vision area as Visual Words. This technique allows us to quantize a dense feature vector into a set of discrete visual words, which are essentially a clustering of similar feature vectors into clusters, using the joint k-means algorithm. The visual words are then used to narrow down a set of candidates from billions to several millions.

After the matching step, we enter the stage of multilevel ranking. We need to rank millions of image candidates, and we’ll do that based on feature vector distances. To speed up the calculations, we use an innovative algorithm developed by Microsoft Research in collaboration with University of Science and Technology of China called Optimized Product Quantization (for more information see the paper: Optimized Product Quantization for Approximate Nearest Neighbor Search). The main idea is to decompose the original high-dimensional vector into many low-dimensional sub-vectors that are then quantized separately as pictured below.

After quantization is complete we need to calculate distances between the query-image and result-image vectors. Instead of using the usual Euclidean distance calculation, we perform a table lookup against a set of pre-calculated values to speed things up even further.

As a result of Optimized Product Quantization, we have reduced candidate set from millions to thousands. At this point we can perform a more expensive operation to rank the images more accurately. After multiple levels of ranking the de-duplication step is executed to remove any duplicate images from the results. The set of images we end up with after this step is the final result set that will be returned to the user.

What’s Next?

We’re constantly working to make the experience better. For example, you may soon notice that we will automatically help you pick objects without needing to draw a box, and provide other tools to help refine your search. We’re also continually focused on bringing the most comprehensive and highest quality visual search results.

Let us know what you think and stay tuned for more improvements!

- The Bing Team
Click to expand...

Source: Beyond text queries: Searching with Bing Visual Search | Search Quality Insights

Brink · Mar 20, 2019

Bing Launches More Intelligent Search Features

In December, we announced new intelligent search features which tap into advances in AI to provide people with more comprehensive answers, faster.

Today, we’re excited to announce improvements to our current features, and new scenarios that get you to your answer faster.

Intelligent Answers Updates

Since December we’ve received a lot of great feedback on our experiences; based on that, we’ve expanded many of our answers to the UK, improved our quality and coverage of existing answers, and added new scenarios.

More answers that include relevant information across multiple sources

Bing now aggregates facts for given topics across several sites for you, so you can save time by learning about a topic without having to check several sources yourself. For example, if you want to learn more about tundras, simply search for “tundra biome facts” and Bing will give you facts compiled from three different sources at the top of the results page.

Hover-over definitions for uncommon words

An enhancement to our intelligent answers we’re rolling out this week gives you insight into unfamiliar topics at a glance. When Bing recognizes a word that isn’t common knowledge, it will now show you its definition when you hover over with the cursor.

For example, imagine you are searching to find answers to a medical question. We've given you an answer, but there are some terms in the answer that you aren't familar with. Just hover over to get the definition without leaving the page.

Multiple answers for how-to questions

We received positive feedback from users including Bing Insiders who said they liked being able to view a variety of answer options in one place so they could easily decide which was best for them. We found that having multiple answers is especially helpful in situations where users struggle to write a specific enough query, such as when they have DIY questions but may not know the right words to ask. In the next few weeks, we’ll be shipping answers for how-to questions, so people can go one level deeper on their search and find the right information quickly.

More opportunities to search within an image

Another new feature we announced in December is intelligent image search, which allows you to search within an image to find similar images and products. With intelligent image search, users can manually select a desired object by manually cropping around them if they’d like, but our built-in object detection feature makes this easier by identifying these objects and highlighting them with clickable hotspots so all you have to do is click to get matching results.

When we first launched intelligent image search, the object detection feature was mostly focused on a few fashion items, like shirts and handbags. Since then we’ve expanded our object detection to cover all common top fashion categories, so you can find and shop what you see in more places than ever before.

Advancing our intelligent capabilities with Intel FPGA

Delivering intelligent search requires tasks like machine reading comprehension at scale, which require immense computational power. So, we built it on a deep learning acceleration platform, called Project Brainwave, which runs deep neural networks on Intel® Arria® and Stratix® Field Programmable Gate Arrays (FPGAs) on the order of milliseconds.

Intel’s FPGA chips allows Bing to quickly read and analyze billions of documents across the entire web and provide the best answer to your question in less than a fraction of a second. Intel’s FPGA devices not only provide Bing the real-time performance needed to keep our search fast for our users, but also the agility to continuously and quickly innovate using more and more advanced technology to bring you additional intelligent answers and better search results. In fact, Intel’s FPGAs have enabled us to decrease the latency of our models by more than 10x while also increasing our model size by 10x.

We’re excited to disclose the performance details of two of our deep neural networks running in production today. You can read more about the advances we’ve made in the white paper, "Serving DNNs in Real Time at Datacenter Scale with Project Brainwave", and working behind the scenes on the project:

V20DKWr0Ps0

We hope you’re as excited about these features as we are, and would love to hear your feedback!

- The Bing Team
Click to expand...

Source: Bing Launches More Intelligent Search Features | Bing Search Blog

See also: Intel FPGAs Accelerate Artificial Intelligence for Deep Learning in Microsoft’s Bing Intelligent Search

Brink · Mar 20, 2019

Bing Intelligent search: Coding answers at your fingertips

If you're a developer you probably read through a lot of documentation, forums and discussions to find solutions to your coding questions, learn new programming languages or master new tools. It always takes a great deal of time and energy to read through the many long threads discussing similar problems and potential solutions before finally finding that one answer.

Many of us are developers too, and we thought: what if Bing were intelligent enough to do this for us? What if it could save users’ time by automatically finding the exact piece of code containing the answer to the question? That is how Code Sample Answer was born.

You can see it live today on Bing by trying a query like “convert case using a function in R”, and we are sure you will appreciate how the code snippet is extracted for you from the article and surfaced right in the results in the requested programming language. If you need more context, the link below the code sample will take you directly to the corresponding article. And just to be clear - our support is not limited to the Microsoft family of programing languages.

The Challenge

According to Wikipedia, source code is any collection of computer instructions written using some human readable programming language, usually as plain text. Each programming language has a defined set of instructions, syntax and form that that is unique to that language.

We wanted our solution to support a broad range of programming languages. For Bing to be able to satisfy this requirement and find a corresponding code snippet answering a user’s query, it had to be able to parse and understand those instructions, the syntax and form of the many different programming languages. Would it be possible to do it without having to build a completely separate system for each language?
At the same time, developers prefer to search for solutions to their problems using natural language. Bing needs to be able to map the intent of the query (expressed in natural language) to the intent of a code sample (expressed in a programming language) in order to find the most relevant code sample for the query.

We were able to solve this challenge by leveraging Bing’s Natural Language Processing (NLP) technology and language agnostic code understanding capabilities, dramatically cutting the time and effort of getting from question to answer for many developers.

How does Code Sample Answer work?

When a developer issues a complex query such as “send html email with attachment in outlook using java” containing multiple coding related terms: “html”, “java”, “outlook”, it takes quite a bit of work to correctly tease apart the actual intent of the user. In this case the intent is to send HTML formatted email with an attachment via Outlook using Java programming language. Getting query intent right is critical to being able to extract the most relevant code sample later in the process from a range of options available.

The natural language processing pipeline of Bing accomplishes this by converting the query to equivalent ‘coding query key-phrase’. Bing’s language agnostic code understanding engine then ensures the results correctly reflect the intent based on holistic query understanding rather than on simple, individual keyword matches. The diagram below shows this process in a bit more detail.

When a query is issued on Bing.com, it is first classified based on its intent as code or non-code type of query. It is then processed by several of Bing’s basic query alteration pipelines including our NLP pipeline and converted to a query key-phrase that can be subsequently matched to relevant web-pages.

Each of these pipelines is specialized for coding queries as their semantics is frequently quite different from general queries e.g.: consider the queries: “Chai or Mocha” and “Chai or Mocha assert”. While they may have some overlapping intent, the primary intent clearly differs. In the former case, a user may very well be looking for differences between chai and mocha beverage types, whereas in the latter it is quite obvious that the main intent is definitely to find out more about corresponding JavaScript testing frameworks.

In ambiguous cases such as this one, web results will continue to honor all likely intents and the Code Answer may be suppressed. It is only when Bing intelligently detects the coding intent with high confidence that that Code Sample Answer will trigger:

To achieve this level of precision for query intent detection Bing’s natural language processing pipelines for developers leverages patterns found in training data from developer queries collected over the years containing commonly used terms and text structure typical for coding queries. The system also leverages a multitude of click signals to improve the precision even further.

Once the query is classified as code vs. non-code query and key-phrase terms are identified, Bing’s language agnostic code understanding engine intelligently interprets the developer intent. This understanding of the intent is built from signals such as specific syntax, any API, tool or language names used in the query that are currently popular in the development community.

Based on query understanding the system then extracts the best matched code samples from popular, authoritative and well moderated sites like Stackoverflow, Github, W3Schools, MSDN, Tutorialpoints, etc. taking into account such aspects as fidelity of API and programming language match, counts of up/down-votes, completeness of the solution and more.

One of the key challenges in extracting and surfacing the best matched code snippet from a web page is that many of these pages can have multiple intents. e.g. this post on Stackoverflow has a primary topic as PDF generation by dynamic tables using Nodejs, but the same page contains a reply from a developer suggesting that Phantomjs is a solution. Such mixed suggestions on the page may lead to ambiguous results. So, to keep our results precise we extract web-page content using explicit semantic analysis which can measure the semantic relevance between a query key-phrase and a given web page.

The semantic score produced by the model captures the quality of match of the document snippet to the query. During the next steps in the process the snippets are evaluated, ranked, and the best one is returned.

In cases where we are highly confident that our best snippet completely satisfies user’s query it will be shown at the top of the result page as in the example above. You may also want to try: "array concat in js", “c# string.substring”, “arraylist toarray java”, “php preg_match” to see more examples of high confidence answers.

In cases where our confidence is lower the Code Sample Answer will be rendered deeper in the body of the page ("c# native json", “indenting bullets in html”, “INVALID PARAMETER EXCEPTION”, “how to move on root web site in sharepoint in c#.net”)

Closing thoughts

The functionality of Code Sample Answer is not limited strictly to programming languages, it also covers many tools developers commonly use. Thus, for example, if you have problems memorizing all the useful git commands and their syntax you can easily get that information as well using Code Sample Answer.

As we receive more and more signals from web searches and continuous feedback from developers we will continue working on enhancing the overall developer help experience and coverage of Code Sample Answer.

We count on feedback from users like you to help us understand what future enhancements would be most helpful and valuable in your daily tasks. We encourage you to use Bing when looking for any code related help, and if you have any suggestions or feedback, share it directly using the Feedback link on the page.

Happy Coding!
- Bing Tech Team (developer help)
Click to expand...

Source: Intelligent search: Coding answers at your fingertips | Search Quality Insights

Windows 10: Bing delivers text-to-speech, intelligent answers, and visual search

Bing delivers text-to-speech, intelligent answers, and visual search

Bing delivers text-to-speech, intelligent answers, and visual search

Bing delivers text-to-speech, intelligent answers, and visual search

Bing delivers text-to-speech, intelligent answers, and visual search - Similar Threads - Bing delivers text

Speech to text

Microsoft Bing now delivers more visually immersive experiences

text to speech

Text to Speech

Bing Intelligent search: Coding answers at your fingertips

Bing Launches More Intelligent Search Features

Introducing Bing Visual Search

Bing New Multi-Perspective Answers for More Intelligent Search

Object Detection for Visual Search in Bing