Windows 10: File explorer --Very Slow for moving large number of small files

Hi there

If you have Loads (and I mean Loads) of small files - e-books, excel data. word docs, emails etc etc then File explorer runs REALLY slow on even fast devices

I gave up moving this lot after about 4 hours. (W10 machine).

Switched to Linux system -- did it in 45 mins similar hardware to W10 machine - both Real machines --not VM's and direct transfer (not via LAN).

What gives !!!!!

I know it's only a 1 time move but similar hardware with identical HDD should not have a HUGE difference in the move --note in fact Linux actually has hard a harder job since the file system was NTFS which isn't a native Linux file system either. Source system external HDD on USB 3 device,, target systm larger HDD device on USB3 device -- no hubs - USB 3 transfer working OK on both W10 and Linux so reasonable comparison.

IMO the NTFS file system has serious problems if you start messing around with this type of data. !!!! Perhaps Ms need to look at a better file system for Mass storage devices. 5 and 6 TB HDD's are quite common now even for consumer sector and not expensive either.

I know this is an exceptional job but it does show that something in File explorer isn't very efficient once you start manipulating large volumes of data - especially in small files. (Approx 290 TB of data consisting of 128,000 files in 55,000 odd directories)





I'm usually quite happy with Windows - but there is definitely a problem here. Fortunately it was a 1 off job though !!!!.

Cheers

jimbo

:)

 

windows 10 file deletion is extremely slow

I am using windows 10 Enterprise fully updated with all Service packs.

When I delete / move or copy files it is very slow, even deleting 1 small txt file brings up the deletion progressbar and gives me a status.

Copying / moving to is very slow. copying like 5 files from one dir to another (even on the same SSD or HD) is slow

Any idea how to fix this ?

 

New and Disruptive Files Icon Behavior

Please see

Elephant size drag and drop 'icon'/image

Windows 10 - Very large icon when copying or moving files in File Explorer......

I'm not the only person with this problem.

 

In my quest to find out what my network transfer speeds were with a program called iPerf3 I found several posts on Stack Exchange that came to the same conclusion as you, In most cases there was literally a x2 increase in speed using Linux.

I transfer quite a few VM's once or twice a month and I find that it is faster to zip the whole 15 to 25 Gig folder and then transfer it.

 

I've found that using USB 3.0 devices is faster than USB 2.0 as it should be but the key is in having USB 3.0 ports, cables and drives. To that end I've added USB 3.0 Add-in PCIe X1 cards with 4 ports on it and for convenience a USB 3.0 Hub in the front of the case that takes power from the computer's power supply, not from the USB ports. Amazing what investing about $50 can help.

 

Hi there

I decided for this type of stuff In future to use a dedicated USB3 external 2 bay mains powered enclosure with (optionally switchable) RAID 0 / 1 for the HDD's. Using RAID 0 on a USB 3 enclosure might be considered "Bonkers" by some but it speeded up the job no end. The RAID card is internal in the USB enclosure (can disable if I want to use the HDD's as separate HDD's --easier to have one logical HDD !!! I wasn't worried about data integrity with the RAID 0 - just wanted fastest possible transfer speed.

On Windows the RAID 0 also speeded up transfer but was a real Slouch compared to Linux.

Decided to use XFS (Linux) file system and use the Linux machine for this type of transfer in future.

Windows NTFS needs a beef up certainly for large volumes of data transfer.

I suppose that's why Linux still rules for large servers out in the commercial world. Don't get me wrong I LIKE windows too -- but seems for some jobs it's just not man enough for the task in hand.

BTW type in my original post --Data transfer amount of course approx 290 GB -- not 290 TB !!!!

Cheers
jimbo