Windows 10: 800 Identical Files in downloads but I cannot see or do anything about them.

Hi! I've been getting notifications on my PC that there are around 800 identical files in my Downloads, but when I click on it nothing happens. I've had an insane storage issue for around a year now and the notifications started coming in recently. I get them around once a week. If I check my downloads, everything looks normal. If I check my storage settings, its saying that 100 GB of storage is being taken up by Installed Apps, but the largest thing shown is 100 MB. None of it adds up to anywhere near 100 gigabytes. Is my computer having a stroke? Is it a virus that my antiviruses can't see?

:)

 

Identical File Finder

@Ford
I don't think that step #3 is even needed. The probabiliy of two MD5 hashes from differing files colliding is what, 2^-128 ?
Even if your were hashing a million files the odds of a hash collision on differing files is astronomically small.

 

Identical File Finder

Neat idea but...I would recommend doing a 1:1 compare...

1. If filesize is equal, move on to step two; else, not identical.
2. Read one chunk (4096 bytes) from file A and file B into separate buffers.
3. Use a for loop to make sure all bytes are equal: fileAbuffer == fileBbuffer
4. If an inequality is ever discovered, break the loop, close the file streams, and output non-match. If they are equal, rince and repeat until End of File is reached.

Using this method, you could even have a percentage match option where you count the differences and use the filesize as the denominator in creating the percent. If not above a user specified percentage (100% if not given), reject it.


The method described above would likely be faster than md5.ComputeHash(file) on large files because it could stop on the first byte.

 

Identical File Finder

Makes sense to me. Sequential reading of a pair of arrays is fast (and of course, doesn't have to do an encryption algorithm even if the whole file is read).

In the last scenario, it probably would be even faster as the odds of two differing files being identical for the 1MiB is unlikely also and it would find the difference sooner.

Now get in there and optimize it more to make it even faster. *Big Grin

In a worse case scenario (let's say 5 files with a differing last byte) you would have to do full file reads 10 times for the array method, but only 5 hash runs and then do a hash List compare.
Seems in this case they would be a lot closer.