Windows 10: Windows 10 v1903 - ntoskrnl.exe & ISR/DPC latency spikes

Hi,


After updating to 1903 LatencyMon associates ntoskrnl.exe with large latency spikes on the order of ~1,300µs. Same results with KB4497935.


I posted here too as I thought it could be related to the GPU driver:

https://forums.geforce.com/default/...-ver-1903-ntoskrnl-exe-large-latency-spikes-/


Others are also reporting similar symptoms. E.g. in some instances this results in noticeable audio clicks/pops.

https://www.reddit.com/r/Windows10/comments/bu34xv/update_1903_high_dpc_latency_issues/


Are you seeing similar results?


Does anyone at Microsoft acknowledge this?


It should be easily reproducible.


Thanks,

Howard

:)

 

High DPC/ ISR Latency

Hello community,

So by way of background, for a while now I have experienced issues regarding audio crackles and pops. This primarily occurs when using Cubase 9.5 but also exists outside of the software too, occasionally. After searching for solutions, I came across LatencyMon
and decided to start testing (the results of which can be found below). I should add that when trying it for the past few days, I have noticed that the 'Highest measured interrupt to process latency' changes between HDAudBus.sys and Wdf01000.sys.

Steps I have taken so far:

1) The first thing I did was simply to plug out all of the USB devices that were inserted in my computer one by one. One potentially notable piece of information is that I switched audio interface very recently - from the Focusrite 2i2 to the Audient iD4.
However, I experienced these issues with my last audio interface too. This step produced no differences.

2) I then, through device manager, disabled a huge variety of drivers one by one to see if there was a particular driver causing the issue. This primarily included network, audio and USB drivers. This step produced no differences.

3) Following this I downloaded Driver Booster and tried to update all outdated drives, of which there were many. This step produced no differences.

Pretty sure there were a few other minor things I have tried too such as disabling WiFi, antivirus and more. All produced no result.

More than happy to provide more details if necessary. Any help is greatly appreciated.

LatencyMon:

Code:

_________________________________________________________________________________________________________CONCLUSION_________________________________________________________________________________________________________Your system appears to be having trouble handling real-time audio and other tasks. You are likely to experience buffer underruns appearing as drop outs, clicks or pops. One or more DPC routines that belong to a driver running in
 your system appear to be executing for too long. One problem may be related to power management, disable CPU throttling settings in Control Panel and BIOS setup. Check for BIOS updates. LatencyMon has been analyzing your system for  0:04:26  (h:mm:ss) on all processors._________________________________________________________________________________________________________SYSTEM INFORMATION_________________________________________________________________________________________________________Computer name:                                        DESKTOP-9TGICJAOS version:                                           Windows 10 , 10.0, build: 17134 (x64)Hardware:                                             N552VX, ASUSTeK COMPUTER INC.CPU:                                                  GenuineIntel Intel(R) Core(TM) i5-6300HQ CPU @ 2.30GHzLogical processors:                                   4Processor groups:                                     1RAM:                                                  12156 MB total_________________________________________________________________________________________________________CPU SPEED_________________________________________________________________________________________________________Reported CPU speed:                                   2304 MHzNote: reported execution times may be calculated based on a fixed reported CPU speed. Disable variable speed settings like Intel Speed Step and AMD Cool N Quiet in the BIOS setup for more accurate results._________________________________________________________________________________________________________MEASURED INTERRUPT TO USER PROCESS LATENCIES_________________________________________________________________________________________________________The interrupt to process latency reflects the measured interval that a usermode process needed to respond to a hardware request from the moment the interrupt service routine started execution. This includes the scheduling and execution
 of a DPC routine, the signaling of an event and the waking up of a usermode thread from an idle wait state in response to that event.Highest measured interrupt to process latency (µs):   2250.220222Average measured interrupt to process latency (µs):   5.684522Highest measured interrupt to DPC latency (µs):       2190.220275Average measured interrupt to DPC latency (µs):       2.232872_________________________________________________________________________________________________________ REPORTED ISRs_________________________________________________________________________________________________________Interrupt service routines are routines installed by the OS and device drivers that execute in response to a hardware interrupt signal.Highest ISR routine execution time (µs):              149.830729Driver with highest ISR routine execution time:       HDAudBus.sys - High Definition Audio Bus Driver, Microsoft CorporationHighest reported total ISR routine time (%):          0.021921Driver with highest ISR total time:                   Wdf01000.sys - Kernel Mode Driver Framework Runtime, Microsoft CorporationTotal time spent in ISRs (%)                          0.023631ISR count (execution time <250 µs):                   82832ISR count (execution time 250-500 µs):                0ISR count (execution time 500-999 µs):                0ISR count (execution time 1000-1999 µs):              0ISR count (execution time 2000-3999 µs):              0ISR count (execution time >=4000 µs):                 0_________________________________________________________________________________________________________REPORTED DPCs_________________________________________________________________________________________________________DPC routines are part of the interrupt servicing dispatch mechanism and disable the possibility for a process to utilize the CPU while it is interrupted until the DPC has finished execution.Highest DPC routine execution time (µs):              1638.547743Driver with highest DPC routine execution time:       Wdf01000.sys - Kernel Mode Driver Framework Runtime, Microsoft CorporationHighest reported total DPC routine time (%):          0.562742Driver with highest DPC total execution time:         Wdf01000.sys - Kernel Mode Driver Framework Runtime, Microsoft CorporationTotal time spent in DPCs (%)                          0.710236DPC count (execution time <250 µs):                   944181DPC count (execution time 250-500 µs):                0DPC count (execution time 500-999 µs):                162DPC count (execution time 1000-1999 µs):              4DPC count (execution time 2000-3999 µs):              0DPC count (execution time >=4000 µs):                 0_________________________________________________________________________________________________________ REPORTED HARD PAGEFAULTS_________________________________________________________________________________________________________Hard pagefaults are events that get triggered by making use of virtual memory that is not resident in RAM but backed by a memory mapped file on disk. The process of resolving the hard pagefault requires reading in the memory from
 disk while the process is interrupted and blocked from execution.NOTE: some processes were hit by hard pagefaults. If these were programs producing audio, they are likely to interrupt the audio stream resulting in dropouts, clicks and pops. Check the Processes tab to see which programs were
 hit.Process with highest pagefault count:                 winword.exeTotal number of hard pagefaults                       643Hard pagefault count of hardest hit process:          323Number of processes hit:                              18_________________________________________________________________________________________________________ PER CPU DATA_________________________________________________________________________________________________________CPU 0 Interrupt cycle time (s):                       17.051465CPU 0 ISR highest execution time (µs):                149.830729CPU 0 ISR total execution time (s):                   0.251434CPU 0 ISR count:                                      82832CPU 0 DPC highest execution time (µs):                1638.547743CPU 0 DPC total execution time (s):                   7.498538CPU 0 DPC count:                                      938734_________________________________________________________________________________________________________CPU 1 Interrupt cycle time (s):                       6.104686CPU 1 ISR highest execution time (µs):                0.0CPU 1 ISR total execution time (s):                   0.0CPU 1 ISR count:                                      0CPU 1 DPC highest execution time (µs):                227.875868CPU 1 DPC total execution time (s):                   0.026960CPU 1 DPC count:                                      2310_________________________________________________________________________________________________________CPU 2 Interrupt cycle time (s):                       5.310433CPU 2 ISR highest execution time (µs):                0.0CPU 2 ISR total execution time (s):                   0.0CPU 2 ISR count:                                      0CPU 2 DPC highest execution time (µs):                203.476563CPU 2 DPC total execution time (s):                   0.009951CPU 2 DPC count:                                      1081_________________________________________________________________________________________________________CPU 3 Interrupt cycle time (s):                       5.152465CPU 3 ISR highest execution time (µs):                0.0CPU 3 ISR total execution time (s):                   0.0CPU 3 ISR count:                                      0CPU 3 DPC highest execution time (µs):                279.683160CPU 3 DPC total execution time (s):                   0.021462CPU 3 DPC count:                                      2222_________________________________________________________________________________________________________

 

Another High Latency Thread / High ISR & DPC

Hello!

The issue:

So, as the thread subject suggests, I'm having some serious hardware latency issues with my graphics and audio kernels. The problem manifests as pops and clicks in audio, usually accompanied with a massive drop in overall PC responsiveness--it's gotten to
the point of taking in excess of 10 seconds to even update my mouse position--or if the responsiveness is low enough, my computer just shuts off altogether. The popping and cracking happens even after a fresh boot with nothing open but a tab on YouTube. It
seems to occur more with web browsers open while gaming, and notably less with no browsers open. ( Yes, they are all up to date. I typically use FireFox, but I've been poking at Chrome to see how things run with it, and the issue does indeed still persist.
) The issue is realized while playing World of Warcraft as a significant frame rate drop. I usually hang around 80-100 FPS doing nothing, and have dropped to less than 1 FPS--sounds impossible, but believe me....it is. Interestingly enough, if I simply reload
my user interface in game, it sometimes goes away until a further reload or enough time passes. I've had this issue for over a year now, and it's only gotten worse as time has passed, and it's beginning to become detrimental to even being able to use my computer
for less C/GPU intensive tasks. That said, here are my stats.

Computer name: DESKTOP-NRNR1U1

OS version: Windows 10 , 10.0, build: 17763 (x64)

Hardware: MS-7850, MSI, Z97 PC Mate(MS-7850)

CPU: GenuineIntel Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz

GPU: NVIDIA Geforce GTX 970

Logical processors: 4

Processor groups: 1

RAM: 16332 MB total

What I've tried:

I've sifted through over half a decade of posts regarding related issues and from what I can see it's basically a trial and error process for figuring out what's the root cause. I've followed most of the instructions posted in threads here and elsewhere
to no avail. I've uninstalled and reinstalled all of my audio and video drivers, disabled advanced features on those drivers, turned off NVIDIA enhancements to sound, rolled back the very same
drivers, disabled physical ports not in use--e.g. some USB ports, extra a/v lines in/out--disabled all real-time applications running, updated literally everything from BIOS to Windows itself. Nothing
has worked and the error persists. Any push in the right direction would be greatly appreciated.

LatencyMon Dump:

Below are the results from LatencyMon.

MEASURED INTERRUPT TO USER PROCESS LATENCIES

_________________________________________________________________________________________________________

The interrupt to process latency reflects the measured interval that a usermode process needed to respond to a hardware request from the moment the interrupt service routine started execution. This includes the scheduling and execution of a DPC routine,
the signaling of an event and the waking up of a usermode thread from an idle wait state in response to that event.





Highest measured interrupt to process latency (µs): 14340.80

Average measured interrupt to process latency (µs): 5.417743





Highest measured interrupt to DPC latency (µs): 14335.20

Average measured interrupt to DPC latency (µs): 1.592265









_________________________________________________________________________________________________________

REPORTED ISRs

_________________________________________________________________________________________________________

Interrupt service routines are routines installed by the OS and device drivers that execute in response to a hardware interrupt signal.





Highest ISR routine execution time (µs): 10012.01750

Driver with highest ISR routine execution time: dxgkrnl.sys - DirectX Graphics Kernel, Microsoft Corporation





Highest reported total ISR routine time (%): 0.464963

Driver with highest ISR total time: dxgkrnl.sys - DirectX Graphics Kernel, Microsoft Corporation





Total time spent in ISRs (%) 0.583767





ISR count (execution time <250 µs): 1258484

ISR count (execution time 250-500 µs): 0

ISR count (execution time 500-999 µs): 354

ISR count (execution time 1000-1999 µs): 511

ISR count (execution time 2000-3999 µs): 1187

ISR count (execution time >=4000 µs): 0









_________________________________________________________________________________________________________

REPORTED DPCs

_________________________________________________________________________________________________________

DPC routines are part of the interrupt servicing dispatch mechanism and disable the possibility for a process to utilize the CPU while it is interrupted until the DPC has finished execution.





Highest DPC routine execution time (µs): 17283.671250

Driver with highest DPC routine execution time: nvlddmkm.sys - NVIDIA Windows Kernel Mode Driver, Version 419.35 , NVIDIA Corporation





Highest reported total DPC routine time (%): 0.216383

Driver with highest DPC total execution time: dxgkrnl.sys - DirectX Graphics Kernel, Microsoft Corporation





Total time spent in DPCs (%) 0.554121





DPC count (execution time <250 µs): 3810173

DPC count (execution time 250-500 µs): 0

DPC count (execution time 500-999 µs): 565

DPC count (execution time 1000-1999 µs): 81

DPC count (execution time 2000-3999 µs): 517

DPC count (execution time >=4000 µs): 0









_________________________________________________________________________________________________________

REPORTED HARD PAGEFAULTS

_________________________________________________________________________________________________________

Hard pagefaults are events that get triggered by making use of virtual memory that is not resident in RAM but backed by a memory mapped file on disk. The process of resolving the hard pagefault requires reading in the memory from disk while the process
is interrupted and blocked from execution.





NOTE: some processes were hit by hard pagefaults. If these were programs producing audio, they are likely to interrupt the audio stream resulting in dropouts, clicks and pops. Check the Processes tab to see which programs were hit.





Process with highest pagefault count: system





Total number of hard pagefaults 16454

Hard pagefault count of hardest hit process: 8248

Number of processes hit: 94









_________________________________________________________________________________________________________

PER CPU DATA

_________________________________________________________________________________________________________

CPU 0 Interrupt cycle time (s): 66.984845

CPU 0 ISR highest execution time (µs): 10012.01750

CPU 0 ISR total execution time (s): 21.322807

CPU 0 ISR count: 1201137

CPU 0 DPC highest execution time (µs): 17283.671250

CPU 0 DPC total execution time (s): 18.901094

CPU 0 DPC count: 3526731

_________________________________________________________________________________________________________

CPU 1 Interrupt cycle time (s): 17.843233

CPU 1 ISR highest execution time (µs): 7667.93750

CPU 1 ISR total execution time (s): 1.550123

CPU 1 ISR count: 55937

CPU 1 DPC highest execution time (µs): 4938.701250

CPU 1 DPC total execution time (s): 1.649164

CPU 1 DPC count: 137761

_________________________________________________________________________________________________________

CPU 2 Interrupt cycle time (s): 13.564233

CPU 2 ISR highest execution time (µs): 3711.1450

CPU 2 ISR total execution time (s): 0.149511

CPU 2 ISR count: 3340

CPU 2 DPC highest execution time (µs): 5009.251250

CPU 2 DPC total execution time (s): 0.70990

CPU 2 DPC count: 78497

_________________________________________________________________________________________________________

CPU 3 Interrupt cycle time (s): 12.177555

CPU 3 ISR highest execution time (µs): 1027.076250

CPU 3 ISR total execution time (s): 0.002039

CPU 3 ISR count: 347

CPU 3 DPC highest execution time (µs): 3099.746250

CPU 3 DPC total execution time (s): 0.595057

CPU 3 DPC count: 68445

 

storport.sys and ndis.sys DPC latency fix?

So I built my PC back during the summer after getting a summer job, and since building, I have diagnosed and fixed a few issues, but one keeps occurring and I am unable to fix myself. I am not completely sure
if it is a hardware issue or purely software based.

Anyway, I've been getting some latency issues that have caused stuttering and sometimes lag while streaming videos from YouTube and Netflix, playing games, and sometimes even while browsing websites. I downloaded
LatencyMon to help find whats causing these issues, and I have pointed out two major .sys driver files that are causing my stuttering. Those being "storport.sys", which is the major cause, and "ndis.sys" being the smaller, but yet
significant cause.

storport.sys definition in LatencyMon is "Microsoft Storage Port Driver", and ndis.sys description is "Network Driver Interface Specifications".

storport.sys gives me upwards of 100,000 in LatencyMon, and I believe ndis.sys gives me about 20,000.

I've read somewhere that storport.sys latency is mostly caused by the Samsung m.2 SSD's but I'm not sure how to fix this

I keep all my drivers updated using DriverEasy

Specs:

MB: Aorus X470 Ultra Gaming ATX AM4 Socket

CPU: AMD Ryzen 2600

GPU: Gigabyte Geforce GTX 1060 6GB Windforce OC Edition

Memory: G.Skill Ripjaw V-Series 16GB DDR4-3200 - F4-3200C16D-16GVKB (Possible latency cause?)

Storage: Samsung 970 EVO 250GB (Boot)

Storage: Seagate Barracuda 2TB HDD (Mass)

Storage: Western Digital "Blue?" 3TB HDD (Backup) (Not 100% sure which drive this is, I think it's blue)

Network Card: TP-Link TL-WDN4800 PCIe x1 802.11a/b/g/n (Possibly cause of ndis.sys latency?)

Operating System: Windows 10 Pro 64bit

Here's the link to the PcPartPicker parts list if its needed: System Builder - Ryzen 5 2600 3.4 GHz 6-Core, GeForce GTX 1060 6GB 6 GB WINDFORCE OC 6G, Meshify C ATX Mid Tower - PCPartPicker

Here's what LatencyMon comes up with after experiencing a storport.sys latency spike:

_________________________________________________________________________________________________________

CONCLUSION

_________________________________________________________________________________________________________

Your system appears to be having trouble handling real-time audio and other tasks. You are likely to experience buffer underruns appearing as drop outs, clicks or pops.
One or more DPC routines that belong to a driver running in your system appear to be executing for too long. At least one detected problem appears to be network related. In case you are using a WLAN adapter, try disabling it to get better results. One problem
may be related to power management, disable CPU throttling settings in Control Panel and BIOS setup. Check for BIOS updates.

LatencyMon has been analyzing your system for 0:10:19 (h:mm:ss) on all processors.

_________________________________________________________________________________________________________

SYSTEM INFORMATION

_________________________________________________________________________________________________________

Computer name: TYS-PC

OS version: Windows 10 , 10.0, build: 17763 (x64)

Hardware: X470 AORUS ULTRA GAMING, Gigabyte Technology Co., Ltd., X470 AORUS ULTRA GAMING-CF

CPU: AuthenticAMD AMD Ryzen 5 2600 Six-Core Processor

Logical processors: 6

Processor groups: 1

RAM: 16331 MB total

_________________________________________________________________________________________________________

CPU SPEED

_________________________________________________________________________________________________________

Reported CPU speed: 340 MHz

Note: reported execution times may be calculated based on a fixed reported CPU speed. Disable variable speed settings like Intel Speed Step and AMD Cool N Quiet in the
BIOS setup for more accurate results.

WARNING: the CPU speed that was measured is only a fraction of the CPU speed reported. Your CPUs may be throttled back due to variable speed settings and thermal issues.
It is suggested that you run a utility which reports your actual CPU frequency and temperature.

_________________________________________________________________________________________________________

MEASURED INTERRUPT TO USER PROCESS LATENCIES

_________________________________________________________________________________________________________

The interrupt to process latency reflects the measured interval that a usermode process needed to respond to a hardware request from the moment the interrupt service
routine started execution. This includes the scheduling and execution of a DPC routine, the signaling of an event and the waking up of a usermode thread from an idle wait state in response to that event.

Highest measured interrupt to process latency (µs): 100830.20

Average measured interrupt to process latency (µs): 3.116927

Highest measured interrupt to DPC latency (µs): 100827.60

Average measured interrupt to DPC latency (µs): 1.344504

_________________________________________________________________________________________________________

REPORTED ISRs

_________________________________________________________________________________________________________

Interrupt service routines are routines installed by the OS and device drivers that execute in response to a hardware interrupt signal.

Highest ISR routine execution time (µs): 148.150

Driver with highest ISR routine execution time: dxgkrnl.sys - DirectX Graphics Kernel, Microsoft Corporation

Highest reported total ISR routine time (%): 0.095580

Driver with highest ISR total time: dxgkrnl.sys - DirectX Graphics Kernel, Microsoft Corporation

Total time spent in ISRs (%) 0.105838

ISR count (execution time <250 µs): 228176

ISR count (execution time 250-500 µs): 0

ISR count (execution time 500-999 µs): 0

ISR count (execution time 1000-1999 µs): 0

ISR count (execution time 2000-3999 µs): 0

ISR count (execution time >=4000 µs): 0

_________________________________________________________________________________________________________

REPORTED DPCs

_________________________________________________________________________________________________________

DPC routines are part of the interrupt servicing dispatch mechanism and disable the possibility for a process to utilize the CPU while it is interrupted until the DPC
has finished execution.

Highest DPC routine execution time (µs): 100834.210

Driver with highest DPC routine execution time: storport.sys - Microsoft Storage Port Driver, Microsoft Corporation

Highest reported total DPC routine time (%): 0.062271

Driver with highest DPC total execution time: nvlddmkm.sys - NVIDIA Windows Kernel Mode Driver, Version 416.94 , NVIDIA Corporation

Total time spent in DPCs (%) 0.169238

DPC count (execution time <250 µs): 1872845

DPC count (execution time 250-500 µs): 0

DPC count (execution time 500-999 µs): 227

DPC count (execution time 1000-1999 µs): 0

DPC count (execution time 2000-3999 µs): 0

DPC count (execution time >=4000 µs): 0

_________________________________________________________________________________________________________

REPORTED HARD PAGEFAULTS

_________________________________________________________________________________________________________

Hard pagefaults are events that get triggered by making use of virtual memory that is not resident in RAM but backed by a memory mapped file on disk. The process of
resolving the hard pagefault requires reading in the memory from disk while the process is interrupted and blocked from execution.

NOTE: some processes were hit by hard pagefaults. If these were programs producing audio, they are likely to interrupt the audio stream resulting in dropouts, clicks
and pops. Check the Processes tab to see which programs were hit.

Process with highest pagefault count: chrome.exe

Total number of hard pagefaults 33063

Hard pagefault count of hardest hit process: 8281

Number of processes hit: 84

_________________________________________________________________________________________________________

PER CPU DATA

_________________________________________________________________________________________________________

CPU 0 Interrupt cycle time (s): 14.904984

CPU 0 ISR highest execution time (µs): 148.150

CPU 0 ISR total execution time (s): 3.907992

CPU 0 ISR count: 224044

CPU 0 DPC highest execution time (µs): 100834.210

CPU 0 DPC total execution time (s): 6.092103

CPU 0 DPC count: 1833174

_________________________________________________________________________________________________________

CPU 1 Interrupt cycle time (s): 1.837817

CPU 1 ISR highest execution time (µs): 72.60

CPU 1 ISR total execution time (s): 0.017628

CPU 1 ISR count: 1600

CPU 1 DPC highest execution time (µs): 533.150

CPU 1 DPC total execution time (s): 0.092153

CPU 1 DPC count: 13765

_________________________________________________________________________________________________________

CPU 2 Interrupt cycle time (s): 1.719505

CPU 2 ISR highest execution time (µs): 59.50

CPU 2 ISR total execution time (s): 0.003827

CPU 2 ISR count: 517

CPU 2 DPC highest execution time (µs): 110.490

CPU 2 DPC total execution time (s): 0.042660

CPU 2 DPC count: 11228

_________________________________________________________________________________________________________

CPU 3 Interrupt cycle time (s): 1.836355

CPU 3 ISR highest execution time (µs): 2.460

CPU 3 ISR total execution time (s): 0.000320

CPU 3 ISR count: 427

CPU 3 DPC highest execution time (µs): 85.570

CPU 3 DPC total execution time (s): 0.018822

CPU 3 DPC count: 4977

_________________________________________________________________________________________________________

CPU 4 Interrupt cycle time (s): 1.813737

CPU 4 ISR highest execution time (µs): 1.820

CPU 4 ISR total execution time (s): 0.000497

CPU 4 ISR count: 701

CPU 4 DPC highest execution time (µs): 228.950

CPU 4 DPC total execution time (s): 0.019418

CPU 4 DPC count: 4982

_________________________________________________________________________________________________________

CPU 5 Interrupt cycle time (s): 1.796899

CPU 5 ISR highest execution time (µs): 4.290

CPU 5 ISR total execution time (s): 0.000660

CPU 5 ISR count: 887

CPU 5 DPC highest execution time (µs): 108.190

CPU 5 DPC total execution time (s): 0.020504

CPU 5 DPC count: 4948

_________________________________________________________________________________________________________

Thanks to anyone that takes a look into this. You're really helping me out. The stutters are getting quite annoying.