Windows 10: A Tale of Two Kernels

Together we go through various troubleshooting steps, we work with the user so that feedback gets filed, and we wait until the next build to see if it was an issue only for this user. About a week after 15002 was released, another build was shared out to the Fast ring for PC users, build 15007. Unfortunately for this user, it’s the same situation yet again:

Same user, same problem, but still only one report that landed in front of us.

A Blue Screen by Any Other Name

Fast forward a bit to early February. We’re doing the second #WinBugBash for the Windows 10 Creators Update and a new error appears from another Insider who is frustrated with not being able to update and participate.

Well this is interesting. Another single-user issue of being stuck on 14986, but a very different error message. We work through all the usual troubleshooting steps, we ensure that feedback gets filed, and we test through a few more releases on the chance that any potential changes in the OS end up solving this issue. These steps didn’t resolve the issue.

As we work on this issue over the course of a few weeks, the frustration begins to set in for these two users. On our side, we’re still pouring through the data. How do we fix this issue? Why is it so hard to diagnose? And why aren’t other users reporting this? If this was an OS issue, there should be more than one person hitting each of these errors. And if it was the same issue as the PTE_Misuse error from the first user, why aren’t both machines showing the same error? The one thing in common here is that both users are stuck on 14986 and haven’t been able to update to any new builds since then. Something is wrong and we need to tackle this from a different angle. It’s time to put a new strategy in place.

Comparing Notes

It’s March 1st, I have two unresolved issues, and I haven’t been able to make headway for these Insiders. Pulling out all the stops, I loop both users into a private conversation via Twitter DM. It’s time to crush these issues once and for all. I’ve talked extensively with the Deployment team here at Microsoft, but there’s no actionable data in the setup error logs. All that we know is there’s a failure and it’s early enough in the update process that no telemetry is being captured. All signs point to a kernel-level failure, but we don’t know exactly what is happening.

Sometimes taking drastic actions can have surprising results. After starting the private conversation, both users share their system information data and immediately we see a connection:





Now we have something to work with! Deeper investigation and additional hardware comparison shows that both users are running the near-exact laptops, both laptops being from the same manufacturer, and both with the same processor. The only notable difference is the amount of RAM in each machine (8GB and 12GB). At last we have a link! But even with this commonality, we can’t resolve the issue without seeing what’s going on. It’s time for kernel debugging, but neither user has done this before, and neither have I. There’s always time to learn something new though, and that’s part of what makes the Windows Insider Program fun.

Cracking the Issue

I could spend pages of writing detailing what happened over the month of March and early April, but I’ll sum it up in a few bullets:

• I shared instructions with Tony and he set up a kernel debugger. We noticed that the failure happened before the debugger could grab anything during the boot process. The effort dead-ended on his machine.
• I did some research and found another laptop from a different manufacturer with the same processor. I set up a kernel debugger and tried to reproduce the issue. The issue didn’t appear. This points us to viewing this bug as being specific to the OEM (manufacturer) of the laptops these two users have.
• We engage with the OEM and kick off a full investigation. The OEM tested the failure scenario on a variety of laptops and could reproduce the issue. This was reassuring that we were on the right path, but we still had to root cause the bug.
• Several weeks of investigation and hypothesis testing ensued. Nearly 50 emails and countless investigation hours later between the OEM and Microsoft, our combined triage team found the issue.

It turned out that a BIOS update released by the OEM in late November for this family of laptop models would fail to allow newer preview builds of Windows 10 to install. The failure was so early in the boot process that none of our log gathering tools would capture the failure, including a standard kernel debugger. Now that we knew the full scope of the problem, we could begin to work on a fix.

Once the OEM had prepared an updated BIOS for these machines, it was time to test. The OEM completed a full range of testing and validation on their side. But what about the two Insiders who had helped tirelessly to help us troubleshoot? It was time to thank them. Once again partnering with this OEM, the updated BIOS that had been prepared was delivered securely to these two Insiders. Both users installed the new BIOS and then attempted to install build 15063. At last… success! The BIOS update solved the issue and had fully resolved the update issue for these two users.

An Interesting Side-Note

From of all of this, it’s funny to look back and realize that we had accidentally found the issue back at the beginning of March and not even realized it.While the noted timing of the BIOS update was slightly off (Brent had updated in December), this bit of conversation captured the problem:





Build 15002 hadn’t been released when each of these users had taken the BIOS update. Both users had installed build 14986 and then installed the BIOS that was now causing problems. Sometimes it’s best to trust your instincts, and it is always important to pay attention to all the details, but I digress from our story.

Learning an Important Lesson or Two (Actually, Three)

After fully detailing the scope of the issue and identifying all possible laptop models from this OEM that could be affected by the issue, we put a block in place to prevent affected retail users across the globe from attempting to install the Windows 10 Creators Update and hitting this upgrade failure. Months of hard work and investigation had helped protect users from a failed OS upgrade experience. The work of two dedicated Windows Insiders helped prevent potentially hundreds of thousands of failed OS upgrades and frustrated users. These two users and their feedback helped others from across the world to keep from using their internet bandwidth (which can be extremely expensive depending on locale) to download an upgrade they wouldn’t be able to install. We listened to Insider feedback and understood there was something bigger here, even if there were the only two people reporting the issue.

Windows Insiders are an amazing bunch of people. They like to learn, they like to explore new features and functionality, and with just as much importance, they like to poke around and find bugs. It’s this nature of curiosity that helps us continually make Windows even better. Another important lesson we learned from this experience is in highlighting the various layers of software that are at play in making a PC work properly. The initial thoughts from the affected users was that we had regressed (broken) functionality at the OS-level. As it turned out in this scenario, it was the OEM firmware that was the culprit of the regression. The software creation process is complex and fraught with ways to create potentially unexpected results (a polite way of saying “bugs”!). Anyone who writes software wants to create a good product and a positive end-user experience. To help with the troubleshooting process in the future, we’ll be creating an additional series of documentation; some will highlight the various layers of software and how they work, as other docs will lay out the foundation of engaging in troubleshooting and some of the related best practices. We’re committed to helping Insiders learn just as much as we’re committed to learning from Insiders.

Lastly, this overall scenario was an important reminder to us on the Windows Insider Engineering Team. Seeking to increase the diversity of hardware and application usage will be an even-greater focal point of the Windows Insider Program as we move forward. Seeing two Insiders represent so many retail users was a profound moment. We’re already working through ways to identify additional scenarios such as this and working to give them the attention they so greatly deserve. It will be an ongoing effort, but we are committed to this program and to listening to our greatest fans who take the time to share their insights and feedback. Knowing how important it is for the millions (of Insiders) to represent the billions (of Windows users), we also know it is just as important for the two who represent the hundreds of thousands. You never know if you may be the one user, or one of a small handful of users, whose feedback and dedication ends up being responsible for helping so many others.

This tale of two kernels is an expression of my unending gratitude to the dedication of every Windows Insider who helps make a difference every day. Without your efforts, this program would not be the success it is.

– Jason



:)

 

A Tale of Two Computers

Hi,

I suggest you to post your query to TechNet forum for further assistance:

Technet forums - Platform Networking

Thank You.

 

A Tale of Two Computers

I have two computers running Windows 10 Pro connecting to my local network and they are nearly identical. S1 is a Lenovo T510 Core i5 560m, w/8GB memory and 500GB HD. S2 is a Lenovo W510 Core i7 820qm w/12GB memory and 750GB HD. S1 is wirelessly connected
to my LAN and S2 is connected by etherenet. Both S1 and S2 have fingerprint scanners which I use to logon. There are two other computers, both running W10 which house my network shares. S1 sees all of my network shows after initial logon with no problem.
S2 ALWAYS requires a second logon to see the network shares on only ONE of the servers; everything else shows up automatically on S2 as well as every where else. Networking is set up identically on all these computers; they are all in the same Homegroup;
network discovery and file sharing are on; they use a common workgroup name; all of them have password protection for network shares disabled, and all of them have the network identified as a private network.

I'm completely out of ideas why S2 ALWAYS requires a second logon to access the network shares on only ONE of the servers. It doesn't matter whether I log on with password, PIN or fingerprint. As fate would have it, it's the one with the most heavily
used and important shares. The odd thing is S2 always requires a 2nd provision of
S2's credentials. I wasted a lot of time figuring that out because I
thought it was asking me for credentials for the external shares. The logon prompt is ambiguous as to what credentials it is expecting me to supply. It didn't seem logical that it was asking for my S2 credentials again because I'd just supplied them
minutes, if not seconds prior. This sounds like a bug somewhere in Windows Hello or Networking. I'm not sure which. Anybody have a clue what's happening here?