Windows 10: Bringing Intelligence to Bing GIF Search - Finding the right GIF

Brink · Sep 20, 2018

The popularity of GIFs has risen, in large part because they afford users a unique way to express their emotions. That said, it can be challenging to find the perfect GIF to express yourself. You could be looking for a special expression from a favorite celebrity, an iconic moment from a movie/tv show or just a way to say ‘good morning’ that’s not mundane. To provide the best GIF search experience Bing employs techniques such as sentiment analysis, OCR, and even pose modeling of subjects appearing in GIF flicks to reflect subtle intent variations of your queries. Read on to find out more about how we made it work, and experience it yourself on Bing’s Image search by typing a query like angry panda. Here are some of the techniques we’ve used to make our GIF search more intelligent.

Vector Search and Word Embeddings for Images to Improve Recall

As we’ve previously done with Image Ranking, and Vector Based Search, we’ve gone beyond simple keyword matching techniques, and captured underlying semantic relationships. Using text and image embedding vectors, we first mapped the queries and images into a high-dimensional vector space and used similarity to improve recall. In simple words, vector-based search teaches the search engine that words like “amazing” “great”, “awesome”, and “fantastic” are semantically related. This allows us to retrieve documents not just by exact match, but also by match with semantically related terms.

GIF Summarization and OCR algorithms for improving precision

One reason GIF Search is more complicated than image search is that a GIF is composed of many images (frames), and therefore, you need to search through multiple images and not just the first, to check for relevance. For instance, a search for a cat GIF, a celebrity or a tv show or cartoon character GIF needs to ensure that the subject occurs in multiple frames of the GIF and not just the first. That’s complicated. Moreover, many users include phrases in their queries like “hello”, “good morning”, “Monday mornings” etc. – where we need to ensure that these textual messages are also included in the GIF. That’s complicated too, and that’s where our Optical Character Recognition (OCR) system comes into play. We use a deep-neural-network-based OCR system and we’ve added synthetic training data to better adapt to the GIF scenario.

The multi-frame nature of a GIF introduces additional complexities for OCR as well. For example, an OCR system would look at the images below, and detect four different pieces of text – “HA”, “HAVE”, “HAVE FU” and “HAVE FUN”. In fact there’s just one piece of text – “HAVE FUN”. We use text similarity combined with spatial and temporal information to disambiguate such cases.

Sentiment Analysis using text – to improve results quality

A common scenario for GIF search is emotion queries – where users are searching for GIFs that match a certain emotion (searches like “happy”, “sad”, “awesome, “great”, “angry” or “frustrated”). Here, we analyze the sentiment/emotion of the GIF query and try to provide GIF results that have matching sentiment. Query sentiment analysis is complicated because there are usually just 2-3 terms in queries, and they don’t always reflect emotions. To understand query sentiment, we’ve analyzed public community websites and learned billions of relationships between text and emojis.

To understand the sentiment for GIF documents, we analyze the text that surrounds the GIF documents on web pages. Having the sentiment for both the query and documents, we can match the sentiment of the user query and the results they see. For instance, if a user issues the query “good job”, and we’ve already detected text like “Good job ” on chat sites, we would infer that “Good job” is a query with positive sentiment and choose the GIFs documents with positive sentiment.

Expressiveness, Pose and Awesomeness using CNNs

Celebrities are a major area of GIF searches. Given users are utilizing GIFs to express their emotions, it is a basic requirement that the top ranked celebrity GIFs convey strong emotions. Selecting GIFs that contain the right celebrity is easy but identifying GIFs that contain strong emotions or messages is hard. We do this by using deep learning models to analyze poses, actions, and expressiveness.

Poses and actions
Poses can be modeled using the positions of skeleton points, such as head, shoulder, hand etc. Actions can be modeled using the motion of these points across frames. To extract features to depict human poses and actions, we estimate the skeleton point positions in each frame and estimate the motion across adjacent frames. A full convolutional network is deployed for estimating each skeleton point of the upper body. The motion vectors of these skeleton points are extracted to depict the motion information. A final model deduces the ‘awesomeness’ by examining the poses and the actions in the GIF.

Exaggerated expressions
Here we analyze the facial expression of the subject to select results with more exaggerated facial expressions. We extract the expressions from multiple frames of the GIF and compute a score that indicates the level of exaggeratedness. Our GIF search returns results that have more exaggerated facial expressions.

By pairing deep convolutional neural networks, expressiveness, poses, actions and exaggeratedness models with our huge celebrity database, we can return awesome results for celebrity searches.

Image graph and other techniques
In addition to helping understand semantic relationships, Image Graph also improves ranking quality. Image Graph is made up of several clusters of similar images (in this case, GIFs), and has historical data (for e.g. click through rate etc.) for images. As shown in the graph below, the images within the same cluster are visually similar (the distance between images denotes similarity), and the distance between the clusters denotes visual similarity of the main images within the clusters. Now, if we know that an image in cluster D was extremely popular, we can propagate that clickthrough rate data to all other GIFs in cluster D. This greatly improves ranking quality. We can also improve the diversity of the recommended GIFs using this technique.

Finally, we also consider source authority, virality and popularity while deciding which GIFs to show on top. And, we have a detrimental content classifier (based on images, and another based on text) to remove offensive content to ensure that all our top results are clean.

There you have it – did you really imagine that so many machine learning techniques are required to make GIF ranking work? Altogether, these components bring intelligence to Bing’s GIF search experience, making it easier for users to find what they’re looking for. Give it a try on Bing image search.
Click to expand...

Source: Intelligent GIF Search: Finding the right GIF to express your emotions | Search Quality Insights

:)

Brink · Sep 20, 2018

Office Sway: bringing more search and more intelligence

From the beginning, Sway has aimed to help people easily create and share fantastic content that looks great on any device. Today, we’re thrilled to start bringing the intelligence of Bing Image Search into Sway Preview. These features allow you to search for content, review any applicable license terms, as well as expand Sway’s intelligent assistance to help you with content searches. It’s now easier than ever for you to use Sway to express your ideas using a variety of multimedia.

More search

Currently in Sway Preview it is easy to collect your content from a variety of cloud-connected sources already built right into Sway. Simply navigate to a source, tap or click your content, and then drag and drop it into your Sway. It’s what being part of the ‘cloud’ really means. We’ve already seen a great response to the built-in sources we started with, which include OneDrive, Facebook, Twitter, YouTube and your device’s local storage.

We also promised to expand that list of built-in sources over time. We’ve been listening to your feedback from UserVoice and directly through Sway via the “+Add Source” survey option in the content sources drop down. In response to requests from users (particularly in the education space) about wanting to be able to search for Creative Commons licensed images directly from the Internet, we’ve integrated Bing Image Search functionality into Sway! Now you can search for images tagged with Creative Commons licenses within Sway.

The list of sources from which you can add content directly into Sway is growing with the rollout of Bing Image Search built directly into Sway and Suggested searches. It also includes a “+Add Source” option where you can give feedback on other services you’d like to see us add.
Simply click the source dropdown and select Bing. Type anything into the search bar, press Enter and get back the results for a Bing Image Search. A URL to the source of the image is provided, which allows you to navigate to the original image and review the license terms to make sure your use is permitted. You are responsible for respecting others’ rights, including copyright. Learn more here.

Just like with Sway’s other sources, you can simply drag and drop any image from the Bing Image Search results right into your Sway. No need to first download an image to your hard drive and then upload it to Sway.

You can expand your search by clicking the “Show all results” button, but you are still responsible for respecting the rights of others.

It’s easy to drag and drop images from Bing Image Search results directly into your Sway.
Once you’ve included an image from Bing Image Search, any viewer with whom you’ve shared your Sway can see the source of that image and applicable license terms. They can simply click or tap the image to view it in full-screen mode, and the attribution will appear as a URL in the lower right corner. As an author, while editing your Sway, you can still see what this viewing experience will look like when you enter Preview mode (click “…” and then “Preview”).

Integrating Bing Image Search is just the first step in our continuous journey to expand the value that Microsoft’s search platform can add to Sway. We’ll also bring more of Microsoft’s connected consumer and business services into Sway, along with other sources of content you are authorized to use (please continue to provide feedback on additional sources through UserVoice, or in the “+Add Source” option in the content sources dropdown in Sway). This way, Sway will continue to make it easier than ever for you to incorporate your content to bring your ideas to life.

More intelligence

With Bing integration, we are also expanding Sway’s ability to assist you in the creation process by anticipating what content you might want to incorporate. Sway continuously analyzes what you’ve written and suggests terms for related Bing image searches, which you can initiate with just a tap or click. After initiating a search, you’ll see results for images tagged with Creative Commons licenses that you may be able to use.

When you first open the Add Content pane, Sway defaults to the “Suggested” option. From the first word you add to your Sway, Suggested searches uses natural language processing to begin showing tags for related words and phrases you might be interested in, based on what you’ve already written. When you see a tag you like, tap or click it to perform the Bing Image Search based on these words or phrases. After you confirm your use is permitted, you can just drag and drop any desired image result right into your Sway! Your viewers are able to click the image to see its attribution (described earlier in this blog).

By continuously analyzing the content you’ve already included, Sway can now suggest related Bing image searches you can choose to perform even if you never use those specific terms in your text.

Just tap or click a word or phrase tag from Suggested to perform a Bing Image Search!
Then, just drag and drop any image result right onto Sway.
As you add or change your content, Sway continues to re-evaluate which search terms and topics you might find relevant. You’ll notice that in addition to picking out relevant topics from your text, Suggested is also using the power of Bing to find related concepts you may not have thought to search for. Additionally, Sway weights its suggested search terms towards the specific content in the card you’re currently working with in the Storyline. That way, you get suggested search terms that are particularly relevant to you in real time. For example, if you’re writing about Italy, but you’re working in a section about history and philosophy, Sway will tailor its suggested searches to related topics such as Venice, Thucydides, Baths of Caracalla, Giorgio Napolitano—even if you didn’t specifically mention them by name.

At any point, while Suggested is open, you can initiate a manual Bing search of Creative Commons images in the “Search Bing” box at the top of the Add Content pane. This saves you the extra clicks of selecting Bing from the dropdown menu and ensures that you’re only one step away from the great content you want.

When we unveiled Sway Preview, the built-in design and layout engine was the first example of the powerful machine learning and intelligence platform we’re building to make it easier than ever for you to bring your ideas to life. The addition of Suggested searches expands that platform, so Sway can now function not only as your digital design assistant, but your content gathering assistant as well. We’ll continue adding more intelligence capabilities that help you quickly and easily create awesome Sways. Using our machine learning platform, Sway will become an even more personalized content creation assistant: learning your preferences, anticipating your needs, and working for you to help you get more done, more quickly.

We hope you find these features to be a great new addition to Sway!

—Sway Team, @Sway
Click to expand...

Source...

Brink · Sep 20, 2018

Bing Launches More Intelligent Search Features

In December, we announced new intelligent search features which tap into advances in AI to provide people with more comprehensive answers, faster.

Today, we’re excited to announce improvements to our current features, and new scenarios that get you to your answer faster.

Intelligent Answers Updates

Since December we’ve received a lot of great feedback on our experiences; based on that, we’ve expanded many of our answers to the UK, improved our quality and coverage of existing answers, and added new scenarios.

More answers that include relevant information across multiple sources

Bing now aggregates facts for given topics across several sites for you, so you can save time by learning about a topic without having to check several sources yourself. For example, if you want to learn more about tundras, simply search for “tundra biome facts” and Bing will give you facts compiled from three different sources at the top of the results page.

Hover-over definitions for uncommon words

An enhancement to our intelligent answers we’re rolling out this week gives you insight into unfamiliar topics at a glance. When Bing recognizes a word that isn’t common knowledge, it will now show you its definition when you hover over with the cursor.

For example, imagine you are searching to find answers to a medical question. We've given you an answer, but there are some terms in the answer that you aren't familar with. Just hover over to get the definition without leaving the page.

Multiple answers for how-to questions

We received positive feedback from users including Bing Insiders who said they liked being able to view a variety of answer options in one place so they could easily decide which was best for them. We found that having multiple answers is especially helpful in situations where users struggle to write a specific enough query, such as when they have DIY questions but may not know the right words to ask. In the next few weeks, we’ll be shipping answers for how-to questions, so people can go one level deeper on their search and find the right information quickly.

More opportunities to search within an image

Another new feature we announced in December is intelligent image search, which allows you to search within an image to find similar images and products. With intelligent image search, users can manually select a desired object by manually cropping around them if they’d like, but our built-in object detection feature makes this easier by identifying these objects and highlighting them with clickable hotspots so all you have to do is click to get matching results.

When we first launched intelligent image search, the object detection feature was mostly focused on a few fashion items, like shirts and handbags. Since then we’ve expanded our object detection to cover all common top fashion categories, so you can find and shop what you see in more places than ever before.

Advancing our intelligent capabilities with Intel FPGA

Delivering intelligent search requires tasks like machine reading comprehension at scale, which require immense computational power. So, we built it on a deep learning acceleration platform, called Project Brainwave, which runs deep neural networks on Intel® Arria® and Stratix® Field Programmable Gate Arrays (FPGAs) on the order of milliseconds.

Intel’s FPGA chips allows Bing to quickly read and analyze billions of documents across the entire web and provide the best answer to your question in less than a fraction of a second. Intel’s FPGA devices not only provide Bing the real-time performance needed to keep our search fast for our users, but also the agility to continuously and quickly innovate using more and more advanced technology to bring you additional intelligent answers and better search results. In fact, Intel’s FPGAs have enabled us to decrease the latency of our models by more than 10x while also increasing our model size by 10x.

We’re excited to disclose the performance details of two of our deep neural networks running in production today. You can read more about the advances we’ve made in the white paper, "Serving DNNs in Real Time at Datacenter Scale with Project Brainwave", and working behind the scenes on the project:

V20DKWr0Ps0

We hope you’re as excited about these features as we are, and would love to hear your feedback!

- The Bing Team
Click to expand...

Source: Bing Launches More Intelligent Search Features | Bing Search Blog

See also: Intel FPGAs Accelerate Artificial Intelligence for Deep Learning in Microsoft’s Bing Intelligent Search

Brink · Sep 20, 2018

Bing launches new intelligent search features, powered by AI

Today we announced new Intelligent Search features for Bing, powered by AI, to give you answers faster, give you more comprehensive and complete information, and enable you to interact more naturally with your search engine.

Intelligent answers:

One of the Intelligent Search features announced today are Intelligent Answers. These answers leverage the latest sta te of the art in machine reading comprehension, backed by Project Brainwave running on Intel’s FPGAs, to read and analyze billions of documents to understand the web and help you more quickly and confidently get the answers you need.

Bing now uses deep neural networks to validate answers by aggregating across multiple reputable sources, rather than just one, so you can feel more confident about the answer you’re getting.

Many times, you have to click into multiple sources to get a comprehensive answer for your question. Bing now saves you time by bringing together content from across multiple sources.

Of course, not every question has just one answer. Sometimes you might be looking for expert opinions, different perspectives or collective knowledge. If there are different authoritative perspectives on a topic, such as benefits vs drawbacks, Bing will aggregate the two viewpoints from reputable sources and intelligently surface them to you on the top of the page to save you time.

If there are multiple ways to answer a question, you’ll get a carousel of intelligent answers, saving you time searching from one blue link to another.

We’re also expanding our comparison answers beyond just products, so you can get a snapshot of the key differences between two items or topics in an easy-to-read table. Bing’s comparison answers understand entities, their aspects, and using machine reading comprehension, reads the web to save you time combing through numerous dense documents to find what you are looking for.

Bing also leverages technology built in Microsoft’s research labs to help make sense of numbers we increasingly encounter in the digital world. Bing translates this data into simple concepts so it’s easier to understand what data like the population of another country means.

Many of these answers are available today and others will be rolling out to users over the next week in the US with expansion to other markets over time.

Reddit on Bing:

A key element of Intelligent Search is bringing together different sources of knowledge, like the wisdom of the crowd, to help people make decisions. Today, we’re launching a new partnership with Reddit, an online community of 330M monthly active users, to bring information from the Reddit community, which generates 2.8M comments daily, to Bing. We are launching with three initial experiences, which we’ll continue to develop and expand as we get feedback from users:

While already in Bing, when you search for a specific Reddit topic or subreddit, like “Reddit Aww”, Bing will surface a sneak peak of the topic with the top conversations for the day from Reddit.

When searching for a general topic that is best answered with relevant Reddit conversations, Bing will surface a snippet of those conversations at the top of the page so you can easily get perspectives from the millions of Reddit users.

Bing will be the place to go to search for Reddit AMAs, Q&As with celebrities and every day heroes hosted by the Reddit community. On Bing you can discover AMA schedules and see snapshots of AMAs that have already been completed. Simply search a person’s name to see their AMA snapshot or search for “Reddit AMAs” to see a carousel of popular AMAs.

More conversational search:

We often hear that search would be easier if only Bing could complete your sentences. Half the battle of searching is knowing the right words to query. Combining our expertise in web scale mining of billions of documents and with Conversational AI, we're creating a new way to search that is interactive and can build on your previous searches to get you the best answer. Now if you need help figuring out the right question to ask, Bing will help you with clarifying questions based on your query to better refine your search and get you the best answer the first time around. You’ll start to see this experience in health, tech and sports queries, and we will be adding more topic areas over time. And because we’ve built it with large-scale machine learning, the experience will get better over time as more users engage with it.

Intelligent image search:

Today, we also shared more detail on Bing’s advanced image search features. Bing Image Search leverages computer vision and object recognition to give you more ways to find what you’re looking for. Search any image or within images to shop for fashion or home furniture. Bing detects and highlights different products within images or you can click the magnifying glass icon on the top right of any image to search within an image and find related images or products. We also previewed a new feature that helps you better explore the world around you. If you find a landmark on Bing image search or use a photo from your camera roll, Bing will identify it and share interesting information about that landmark, such as the origins of the landmark and other relevant trivia. For instance, if you are looking at the India Gate, Bing can tell you why it was created and even what kind of stone it was made from. More to come on this feature in the future.

We’re excited for you to try out all of Bing’s new Intelligent Search features and are committed to delivering even more features that will help to save you time and money in the future. To learn more about Intelligent Search visit our site here.

-The Bing Team
Click to expand...

Source: Bing launches new intelligent search features, powered by AI | Bing Search Blog

Windows 10: Bringing Intelligence to Bing GIF Search - Finding the right GIF

Bringing Intelligence to Bing GIF Search - Finding the right GIF

Bringing Intelligence to Bing GIF Search - Finding the right GIF

Bringing Intelligence to Bing GIF Search - Finding the right GIF

Bringing Intelligence to Bing GIF Search - Finding the right GIF - Similar Threads - Bringing Intelligence Bing

GIF Screensaver

Truly Transparent GIFs?

Inserting gif

Windows not looping gifs

Gif on opening???

Saving GIF images...

GIFs on Lock Screen

Avatars (Animated gif)

Bing - In Time for Halloween: Find Spooky GIFs in a Jiff