Windows 10: slow access to hd with millions of files

We have our security cameras recording to a 3 TB hard drive and it tends to get loaded with LOTS of files, well into the millions, loaded into subfolders based on camera, year, month, whether photos or videos (ex: f:\hikvision2\2017\01\photos). Access to this hard drive has become cumbersome - it takes minutes just to get a directory listing.

I've already gone down the "change to general file type instead of photos" route, still slow access.

I'd be interested in getting input on how to best address this issue and be able to maintain a history of files from the security cameras.

:)

 

Copying files VERY slow, speed drops to zero, even on USB 3 -- no solutions have worked

I was having around 1 million small files to copy on my USB3 archive drive and it was slowing down drasticaly at around 100 000 files every time.

The ONLY solution that worked for me was to pause the copy process 5 minutes each time that it was slowing down, then to resume it.

 

Hard drive became super slow after upgrading to Windows 10

Hi,

I've got the same problem, seem that the OS has hard coded the HD read at 10MB/s. I edit videos and this is way too slow. I did not have this degree of render problems with windows 8.1. With windows 10, render crawls with HD read being the culprit. File
transfer (to NAS) is also crazy slow, with HD read maxing (100% @ 10MB/s) before network coming close to max.

I have turned off automatic management of virtual memory (was trying to keep a +20G page file) and set the min and max to Microsoft calculated values. This made the system much more responsive, but did not help with rendering (tons of reading of files off
the HD) as it still maxed at 10MB/s.

Thanks

 

Possibly switching to a NAS RAID server or USB 3.0 RAID server with at least dual hard drives? If you went all out for a 4 drive servers, there are different raid modes that would offer both drive failure recovery capability and spanning across more than one drive to increase performance.

USB 3.0 raid enclosure - Newegg.com

 

Windows Explorer has always had problems with large numbers of files/folder. You can run Process Monitor - Windows Sysinternals | Microsoft Docs to see what files Explorer is accessing just to confirm it's not doing anything weird. But from personal experience once you get into the 10s of thousands of files there's not much you can do.

You can try a different file explorer app or do things from the command line. There may be existing solutions out there for security systems like this. But a custom front-end wouldn't be hard to write if the folders are well-organized.

Edit: This is assuming you have thousands of files/folders within a single folder level (e.g. \folder\00001, \folder\00002, ... , \folder\99999).

 

The hard drive is actually in a Synology NAS, so, don't know if that's got anything to do with the access performance. But, there are thousands of jpg files in each directory - when a camera triggers the system sends jpg snapshots to the hard drive, at least 5 pics each time it triggers. At night when we have spider webs waving across the camera, we get triggers about each 10 sec, all night long. Same when fog rolls across all the cameras: boom a lot of cameras trigger and send snapshots do the hard drive.

So, the hard drive is chock full of jpg files and even getting to the drive to delete files is a pita since you have to navigate down the directory tree and each move takes several minutes.

 

Not sure if this helps you but I've seen my response times improve when I did this:

Slow Folder Loading when sorting by Date Solved - Page 2 - Windows 10 Forums

 

You said that you have tried "general file type instead of photos" without success.
I, too, was unsuccessful when choosing "photos".

Instead I find that the best choice is "documents".
Try that. It worked for me with both Windows 10 and Windows 7.

 

No change, I've tried that route several times with no change.