Windows 10: How to safely remove an external drive?

Sorry to be so long getting back. This took a long time to write (too long), and I ended up doing some research during the writing. Then when I was about 2/3 done, the forum software somehow lost my reply content in the edit box during a preview, so I had to rewrite a lot of it. *Sad

There almost certainly errors and typos, but I'm too wasted to proof it any more carefully right now.

*Doh I knew (and almost said in previous post) the answer was going to be obvious! *Redface ... .... ...But thanks!

Part of my problem was that the image doesn't display full size in my browser, and I was having a hard time reading the relevant P: lines in question. By the time I downloaded the image from imgur.com so I could display it full size to read the P: lines, I had already given up trying to find the column in Process Explorer. ...So I didn't even look at the top left corner where I can now clearly see "Process Monitor". Worse, I'm already familiar with Process Monitor; but to be fair, I rarely use it, or the 'Time of Day' column (which I wondered about but didn't search for), the registry keys, etc. in the image would have tipped me off.

Ahhh, thanks! That seems like a good additional precaution. Although I always just make sure I've waited more than long enough before unplugging, I will incorporate that into my practices. Should especially be relevant if/when there is a need to unplug in a hurry.

To the contrary, although I rarely use it, I think it is very powerful and reliable... ...but just not the right tool for the job of finding open drive handles.

When I thought the screenshot was from Process Explorer I was saying those P:\ line entries would not need to be closed before removing drive (because probably have nothing to do with handles on the P:\ drive). Now that I realize the screenshot is from Process Monitor , I would point out that the P:\ line entries cannot be forced closed.

Process Monitor displays a running list (log) of the filtered process events that occur on the system between when you start and stop monitoring, so each of those lines simply represents a single process event that occurred at the time listed in the 'Time of Day' column.

I don't know enough about it, but it seems likely to me that there are process events when a P:\ drive handle is opened or closed (and that might even be part of what the CreateFile and CloseFile P:\ line entries in the screenshot indicate ???). But if so, the resulting line entry in Process Monitor line simply shows when that event occurred. I guess if you knew the right kind of event to filter for, you could conceivably monitor for only handle open and close events, and then would be able to view when those occurred on the P:\ drive and check for a close following each open. But that would be an extremely inefficient way to observe handles, and you still wouldn't be able to force the handles closed from within Process Monitor.

Yes <- - - - - - - [Gee, I can almost never answer that briefly.] *Tongue

Not completely sure I'm understanding you here, but if you have already verified in Process Explorer that all other handles are closed on the external drive, that particular transfer should be the only thing left to complete. The journaling crucial to NTFS drive recovery will have already occurred first, and there shouldn't be any other processes besides completing the operations of writing of any file changes to the drive (flushing any remaining data from the cache and writing it to the drive). No further journaling required, at least for drive recovery.

Per NTFS Transaction Journal:NTFS log provides file system recoverability by logging, or recording, the operations required for any transaction that alters important file system data structures. This is done before these operations are carried through on the disk. This process ensures that if the system crashes, partially completed transactions can be redone or undone when the system comes back online
As I understand it, the idea behind journaling is that any NTFS structural changes are logged to the drive first, before the file data changes (held in a cache) are actually written to/deleted from the drive. This protects the NTFS data structures so recovery to a stable state is possible in the event of system crashes, power failure, etc. (...including unplugging a USB NTFS drive). But obviously it doesn't protect the actual data to be stored in those structures if the data isn't yet written to the drive.

So as I understand it, even if the file is quite large, the NTFS journaling includes logging of files names, logging where on the drive data is to be written/deleted, etc. Logging these events involves a trivial amount of metadata that can be written to the drive almost instantly/immediately from a human perspective, and is written before the writing of relevant data, so the NTFS logging events are not really a meaningful source of delays the user has to be concerned about. I could be wrong, but I'm assuming the OS won't represent to the user that the operation is completed before this crucial NTFS logging occurs. Then the remaining operations necessary to write the actual data to the drive occur (with each step in the process is also logged almost instantly in NTFS journal). What could take longer is flushing data from the cache and writing it to the drive, transferring large amounts of data over a limited USB transfer rate connection, etc. But unless another change is made to the file, no further NTFS logging is needed to protect drive file system recoverability, even if data is still being written (although still unwritten file data won't be recoverable).

Again per NTFS Transaction Journal:After a transaction is logged, the NTFS performs the sub-operations on the volume itself, in the cache. Finally, after the cache has been updated, NTFS writes another record to the log file, recording that the transaction has been completed.

When a system is being recovered after a system failure, the NTFS reads through the log file and redoes each committed transaction. After that, the NTFS locates all of the transactions in the log file that weren't committed at the failure and rolls back.
As I understand it, after the data has been flushed form cache and written to disk, the final NTFS logging event occurs to mark the process as complete. Again this involves a trivial amount of metadata that can be written to the drive almost instantly. If the file data itself hasn't yet been completely written to the drive, the final NTFS log file record showing the file is complete won't be there, and NTFS rollback will occur upon recovery. But once the data writing is complete, the NTFS log entry should occur almost instantly, and the drive can then be safely removed. (I'm pretty sure, but not 100%, this also means that if a very large file is only partly written, most of the written data will be recoverable. Depending on the type of data it may or may not be usable: i.e., a plain text file, an .mp3 file, or some types of video files that can be split anywhere might work as expected up to the point where the transfer ended.)

I assume probably the same re flushing the cache. But if the ...$Extend\$RmMetadata\$Txf handles exist for NTFS journaling (per our hypothesis) and are the only problem (and I think we can be confident they are once enough time has passed), the error message is popping up not because of uncompleted data writing, but because those handles are "locking" the drive. And the OS apparently does not have any means to automatically close them when no longer needed, and the user UI has no manual option to unlock/close them.

See NTFS journaling info above. The NTFS logging is written almost instantly to the drive before, during, and immediately after the data is written, so there is no reason for NTFS to 'break'. At worst, MTFS will rollback to the last completed/logged transaction and the file system will be stable (not broken) going forward from there. At worst, any unwritten data in the one file that did not finish writing will be missing from that file, but the file metadata will be intact. Maybe an analogy would be a paper file folder (NTFS data structure) being placed in a file cabinet (HDD), but not all the relevant sheets of paper (data) get added to the file folder before someone closes the drawer and locks the file cabinet (system crash, power loss, unplug USB). When you unlock and open the file cabinet (reboot), the file folders are all intact, complete, and usable, except the one folder that was incomplete when the drawer got closed and locked. But even it's contents are in the proper folder and usable, although incomplete.

If you allow enough time (or flush the cache), it seems even that last file written to the drive should be complete and sound. I fact after researching this a bit more and trying to write it up, I think I've convinced myself that I have been being way overly cautious in waiting at least 10-15 minutes (and often hours) before removing a drive. Unless I have been writing a hug file to the drive, if only the ...$Extend\$RmMetadata\$Txf handles exist, I think I've convinced myself that 1-2 minutes should be more than sufficient

Might be helpful for those drives that have a blinking LED, but not all do (and most USB flash drives don't, but still could be formatted NTFS), and I don't know if they all behave the same way. I think I'd have more confidence in checking the handles in Process Explorer and adding in the additional step of flushing the cache with Sync or FFB (see below).

That's what I meant when I said,

To make that a bit clearer, it seems like Microsoft should build in a way to detect when ...$Extend\$RmMetadata\$Txf handles are the only issue preventing drive removal, and then either automatically close the handles when the user clicks "Safely Remove Hardware..." or provide a UI checkbox or button for user to manually close those handles (maybe with brief description of the issue and guidance to help user decide whether to use the checkbox/button).

*Wink Yeah, I love Everything Search! I'll try to go back and check those Indexing discussions as soon as I get a chance. Do you use the same username there? (I'm there as well)

Are you saying that is the way it now works? ...or that settings can now be tweaked to do that? I haven't checked the site for a while, and I'm running v1.4.0.713b, which I know is not the most recent version. I'll try to check for updates as soon as I get a chance.

Ahhh, thanks for the translation. Helps me better understand the whole $Recycle.bin issue.

If that is easy to replicate, or if you run into a similar situation, it would be nice to know for sure, as it might imply Recycle Bins are keeping the ...$Extend\$RmMetadata\$Txf handles open. Apparently it is established that a corrupted Recycle Bin is one cause for the handles not closing, I would then want to get some idea of how often that is the cause. What if it is the main cause, or even the only cause? I don't think that is the case as I don't think I'm likely to have corrupted Recycle Bins because I nearly always delete files completely (Shift+Delete) rather than sending them to the Recycle Bin

So would I, but if it is corrupted Recycle Bins that cause the handles to not be released, that is an important finding, and if deleting the corrupted Recycle Bin closes the handles (and results in them not staying open during future attempts to "Safely Remove Hardware...", that would be even more important.

Agreed, both Sync & FFB just flush the data in the cache to the drive, but theoretically that should stop any NTFS journaling as well, as that logging is done almost instantly before data is written, updated almost instantly while data is being written, and finalized almost instantly when the data-writing is completed (see discussion above). But what Sync and FFB don't do is close the NTFS ...$Extend\$RmMetadata\$Txf handles on the drive.

To stay with the admittedly imperfect paper file folder analogy, the file folder (NTFS) is labeled (NTFS) and slotted into the correct section (NTFS) of the correct drawer (NTFS) of the filing cabinet (HDD), and all the relevant sheets of paper (data) are filed in the file folder, but the filing secretary (OS) is still holding onto the file folder (...$Extend\$RmMetadata\$Txf handles) inside the drawer, so the drawer cannot be closed and locked ("Safely Remove Hardware").

Ahhh, I see you have also found USB expert, Uwe Sieber (although you probably use his German page). I don't think I've visited his site since I was researching the ...$Extend\$RmMetadata\$Txf handles a few years ago, and I don't think I grasped the issues well enough at the time to have realized FFB might be useful. (...Might have even been before FFB was first introduced in April 2013 but probably since then.) ...But I had been thinking while drafting each of my posts in this thread that I would go back and check his site for anything new on the issue as soon as I get a chance.

According to Google, there is no mention of "$Extend\$RmMetadata\$Txf" (or "$Extend" or "$RmMetadata" or "$Txf") anywhere on his site, but he's the only one I've run across on the internet who I'm guessing could probably authoritatively clarify the ...$Extend\$RmMetadata\$Txf handles issue. I remember thinking about trying to email him about it a few years ago, but never got around to it.

I notice he mentions a couple of limitations of Sysinternals Sync that FFB overcomes:"The Sysinternals sync tool can flush volumes only which have a drive letter. And it forgets drive Z: when called without parameters"
It would be interesting to know if the FFB -f switch simply does the same thing as closing handles in Process Explorer:-f force dismount (open handles become invalid)If so, the switch would presumably force close all open handles on the drive. That, along with detection of what is preventing 'Safely Remove Hardware", would be the kind of function I think Microsoft should build in to automatically use (or manually selectable by user), if Windows detects that only the four ...$Extend\$RmMetadata\$Txf handles are preventing "Safely Remove Hardware".

 

Yes, that is really bad. Very sorry for that. But glad you are back.

Yes, that's right, I would say, if one watches the processes / handles - in this case - P: there should not be other ones, I would say, than the ones you see. But if one does not watch them - and usually it should be like that - one may be never knows, if there still is a process running, may be re-indexing the NTFS, rebuilding it or whatever.

Yes, but sometimes it does not work, a recovery is not possible, so both the NTFS and the back up are corrupted obviously. May be the back up, recording, journaling work in another way (worse performance) when the option "Quick removal" is activated.

Yes, and / or sometimes it obviously fails completely, the drive is shown as "raw".

Yes, that sounds plausible, may be one thought was, that, if the data in the cache is written to the drive Win updates the journal, NTFS and then there is no need anymore to keep the handle / process running.

That is the reason why I do not move files generally but copy and after I delete the source files or use programs like FastCopy (doing it that way, removing the source files - when moving - after all of the files are transferred correctly)

No, I do not, it is Biff.

Yes, if I remember it right (and have understood it right) it is what the developer says.

Yes, the new version is super. Keeps offline indexes automatically, indexes offline drives automatically when plugged in.

I am pretty sure the handle / process were / was released after that bin error was removed. I would not know how to reproduce such behavior, if such situation, respectively when it occurs again I will let you know what happens.

May be flushing sometimes causes Win to stop journaling / NTFS actions. May be it depends on the state the NTFS / journal is.

Yes. But it occurred that after wating for about 15, 20 minutes or so suddenly a drive could be removed safely after several tries. I would think it depends on what / how much Win writes, indexes, (re)builds, does regarding the journaling, NTFS.

Yes, there are some programs connecting to safely removing / USB and such. It is worth a visit (in this context).

Yes, indeed, a good idea to ask him.

Yes, I remember to have read that as well. And FFB shows "OK" for a successfull execution, I am not sure anymore, but there shall be circumstances which let that sync tool fail flushing external / USB drives (without displaying it).

Yes, would be difficult to test, I assume.

Yes, sounds like a good approach.

Thank you very much for the explanations sounding very plausible.