Windows 10: Making mixed reality: a conversation with Alexandros and Sophia

Who uses Holo Graph?

Alexandros: In a nutshell, the end users for Holo Graph are computational biologists, clinicians, and oncologists. Instead of looking at “big data” in a two-dimensional structure, they immerse themselves and explore and focus on their areas of interest in 3D. There are two scenarios that we currently use with Holo Graph. One is for cancer research and genomics, and the other is metabolomics.

For cancer research, it’s for drug discovery. We want to find how specific drugs relate to specific genes. With our app, I can upload my network that has all of this correlating information, and I can explore it, manipulating and changing the ways I look at data. If I click on a hologram of a drug, I’ll see the drug’s most up-to-date information directly from the Food and Drug Administration (FDA) – that’s an API tool. If I click on the gene, gene cards will tell me more about that specific gene.

The other use case that we’re doing with our colleague Dr. Karsten Suhre from Weill Cornell Medicine in Qatar is metabolomics. Dr. Karsten Suhre has identified connections between metabolites, genes, and diseases such as Crohn’s disease and diabetes. Using Holo Graph he can browse and identify unexpected paths in the network. One of the latest videos that we shared was collaborative sharing and manipulation of this very network on Crohn’s disease to identify if there are unexpected connections to other diseases.




The Holo Graph team comes from the Englander Institute for Precision Medicine at Weill Cornell Medicine and the Weill Cornell Medicine Qatar. From left, Sophia Roshal, Dr. Karsten Suhre, Dr. Olivier Elemento, Dr. Andrea Sboner and Alexandros Sigaras. (Photo courtesy of the Englander Institute for Precision Medicine)

When did you get started building and designing for Windows Mixed Reality? Any tips for others just beginning?

Alexandros: We became interested on the platform about two years ago and were delighted to be included in the first wave of HoloLens devices that shipped. Whether on an online forum or a meetup, there are a lot of talented people happy to help you get there and share their experience. Don’t try to reinvent the wheel. You will be surprised how many questions have already been answered before! As far as tips go, download and try out apps from the Windows Store and make sure to reach out to the community for any questions.

Sophia: Fragments was the biggest inspiration for me because the virtual characters sit on actual chairs. We recently updated our app to include avatars and do the same thing, and it was really cool to see that! Our avatars follow the person’s movement, and we also use spatial mapping to find the floor because the only point of reference on HoloLens is the head. There are MixedRealityToolkit scripts that finds planes, where you can find the lowest one and that will be the floor. Then you calculate the height between the head and the floor and you can map the avatar from there.

Alexandros: Doing tutorials Holograms 240 with the avatars and Mixed Reality 250 with sharing across devices are excellent examples to see the capabilities here.

Sophia: For someone just beginning, Mixed Reality Academy is by far the easiest way to start building apps. MixedRealityToolkit is the main tool I use. I will write my own scripts most of the time, but if you’re just starting, you have to get it!

What inspires you?

Sophia: Impact on our patients’ lives. Seeing your code making a positive impact in someone else’s life battling with cancer is one of the most rewarding experiences ever. The Englander Institute for Precision Medicine is using cutting-edge technology to go through a sea of data and provide our care team and their patients better treatment options. We believe that AI and devices such as HoloLens are just beginning to show their true potential, and we look forward on what’s yet to come in the near future.

Alexandros: And it’s all about the power of people. The headset doesn’t save someone’s life; the clinician does. But mixed reality helps them see the patterns and get there. With HoloLens we want to answer, “If I were to show you this before you made the call, would that change anything? Would it make your decision and response faster? Would it give you more data?” And every person that we ask nods their head and says, “Yes, it’s right there. It’s almost like I can touch it.” Clinicians are used to reviewing genomic reports that can span up to hundreds of pages requiring significant time and effort. This doesn’t have to be the case though. HoloLens can act as a catalyst by significantly reducing the review time and make an impact at scale when combined with other tools such as AI, machine learning, and deep learning that we also do at the Institute.




Holo Graph can help researchers identify patterns in networks (photo courtesy of The Englander Institute for Precision Medicine)

How are you getting data into your application? Any tips for those who want to do data visualization with HoloLens?

Sophia: The easiest way to load dynamic data into an application is through a cloud integration app such as OneDrive or Dropbox. When you share data across the network to other users you need to consider secure transfer and adopt standard formats. Holo Graph currently supports .csv and XML/GraphML formats on OneDrive. We tend to share data across using JSON.

OneDrive loading has been a great surprise. We used to add files to the backend. With OneDrive, now anyone who wants to can load their data into the experience.

Alexandros: Data and data privacy are of utmost importance to the Institute. The real value of Holo Graph is not just about its looks; it’s about empowering researchers to get their real data in securely. As far as visualizing the data, my tip is to enable your users to break out of the 2D window and put their data on their environment on their terms.

I’m ending with a favorite quote from our conversation. Spoiler alert: mixed reality’s got game!

Alexandros: The way I explain mixed reality is this: Imagine you have a virtual basketball. If you’re throwing it onto the real ground, it bounces off because it knows where the ground is, and there’s “friction.” You can repeat this 100 times and it would happen the same way – you can expect it. It’s literally bringing digital content into real life, allowing you to bend the rules.

Sophia and Alexandros are seriously inspiring. You can connect with Sophia and Alex on Twitter @RoshalSophia and @AlexSigaras.

Want to get started #MakingMR? You can always find code examples, design guides, documentation, and more at Windows Mixed Reality Dev Center. Want more? Check out mixed reality design insights on Microsoft Design Medium. Inspiration abounds!



:)

 

mixed reality in 1703

Hey mate,

First off, make sure your PC is a Windows Mixed Reality-ready PC and an optional compatible mixed reality headset. Check the requirements here: Windows
Mixed Reality minimum PC hardware compatibility guidelines

And here's the how to: How to use the Windows Mixed Reality simulator in Windows 10 Creators Update

Cheers.

 

Dell Visor -- Mixed Reality Portal Never Showed Up?

We appreciate your response. You are correct, as of the moment, Windows S and N editions do not support Mixed Reality. For more information and future announcements about Mixed Reality, you can frequently visit
Windows Blogs. We hope you'll continue to enjoy using Mixed reality on your laptop.

We are currently making some actions to make sure that every Microsoft representative is informed about the features and limitations of Mixed Reality.

If further assistance is needed, we are here to help.