Windows 10: Win10 NTFS file copy/backup utility that handles permissions correctly

IIS = "aye-ayeeeeeeeee-esssssssssssssssssss!" LOL *Wink

" @slicendice " , just wondering, have you tested things with the root Administrator account lately? If you've got the latest updates, you may very well find yourself unpleasantly surprised.

As far as privileges and permissions go, it's actually been our own experience to keep things simple for the Administrators group in that the following are to be done:

1 - We Enable the Internal (ROOT) Admin Account, and also set it so that it cannot be used remotely.

2 - We setup a Second Admin Account, throw in several layers of security that include our own TLD and CA servers with RADIUS and Directory Services via x.500 with Windows 10 in a Standalone WorkStation Configuration, that is a WorkGroup Member so that both Machines and Users have to compartmentally ID to be authorized on our Private Cloud. This approach has worked for well over 10 years without incident.

3 - Obviously the Second Admin account is the most used by Machine Administrators, so assigning rights specifically to either the Administrators Group or the Accounts directly has worked fine in the past without any problems, and performance on the latest builds is still the same from our own testing and observations.

4 - The ShadowCopy service can still work in our configuration, but we don't really consider that to be a "Syncing" service because ShadowCopy as a service was originally intended solely for Data Backup to be allowed while Windows is in live operation, and even though we've got it locked down very heavily on both software and even our hardware firewalls, ShadowCopy will still run as we need it to, but in no other cases such as "Syncing" will ShadowCopy operate.

NOTE ON ITEMS 2, 3, and 4: The implementation we use is supported by proxying Internet Access; essentially making us our own ISP while optimizing portability for authorized equipment. This method ensures Machines are Registered to our infrastructure, and for primary support of the infrastructure, we utilize Novell Linux Enterprise Server (a.k.a. NLES). A purely OpenSource variant of NLES is of course OpenSUSE that can work equally as well, we just prefer NLES because it's a bit more concrete for us and easier to register to our Enterprise.

SLI - IT-IS Team

 

There are lots of reasons for tree-copying/tree-syncing the entire file system, such as cloning the c: drive so that one can compare the "saved" system stuff against the current running stuff, or having a second set-up to use for "experiments" (such as a DVD-based re-install). "sfc /scannow" is supposed to be able to restore system files, but it's broken on my Win10 (wasn't so for previous Win versions).

There are two issues in doing verbatim tree-copying. If the source tree is a "moving" target that can't be reliably snapshot in a consistent state, Windows has traditionally used volume shadow copy to make a duplicate of the source as necessary. If the copying mechanism has to over-write the destination tree (as in an incremental sync copy), then it can't be locked out of modifying the destination tree (as it is now the case with Win10). For example, there is a "really hidden" directory at c:\Windows\Winapps that's owned by TrustedInstaller. If a "first" copy (by an utility running under Administrator) of c:\Windows\Winapps is made to f:\Windows\Winapps, then the copy/sync will succeed. However, some subsequent copy of some stuff in c:\Windows\Winapps to f:\Windows\Winapps could likely fail because the destination Winapps is owned by TrustedInstaller, and can't be modified by Administrator.

Linux doesn't have this problem, as root can overwrite anything. Windows obviously instituted TrustedInstaller to plug some security hole where some app that manages to run under Administrator wreaks havoc on some system files. Win10's explorer apparently uses some new "smart copy" mechanism to deal with some of the legitimate needs to copy/modify/delete system files, but it's not clear smart copy works with command-line tools.

Xcopy/shadowcopy/robocopy probably can't deal with the destination-tree permission lock problem, and can only do copy anyway (verses sync, where only modified files are copied). Syncing is often much more preferred over dumb copying, since there may be 500MB file system out of which only 20MB needs to be updated at the destination. Linux's rsync utility does this very nicely (can even re-start the sync properly, in case of interruptions).

With these inherent "design flaws," it's not clear to me that doing a Win10 re-install with _not_ saving user files and apps would do me any good, as the underlying problem will continue to exist, and there is a good chance my apps may not be able to re-install properly.

A related, hysterical, problem I'm having is that Windows update is being forced upon us. Since my Windows Update is broken, I see my eventvwr repeatedly download and try installing update packages. While many are shown as succeeding (which I'm suspicious about, given that these packages keep getting updated), the big bad one is KB4088776, which is the critical cumulative update for 1709, which keeps failing to install. To add insult upon injury, Microsoft decided that even if an update fails, if it required a restart after installation (whether succeeding or failing), the system is restarted. So, my machine gets rebooted almost once a day.
Brilliant. If some company is so arrogant to force things on its customers, then it better get the thing right. Prior to Win10, one can mark an update for no re-try (although the cumulative update should really be installed)..

 

Our issue with Syncing is that in some cases Syncing can be exploited, and then just hooking up a high capacity drive and using the sync services can cause a serious data breach. Eliminating use of Syncing is the right choice for us, but we certainly recognize its not the right choice for everyone.

The issues @Win10 User has brought up, definitely require a little R&D work. A community effort would certainly be helpful in figuring out the best approaches.

SLI - IT-IS Team

 

Actually I haven't. I will experiment with this for the next couple of days/weeks and see what works and what not.

Shadow Service should still work as intended and thus being able to copy the whole running system. I could be wrong though. Only testing will prove one or the other. *Smile

 

@slicendice - We've noticed Microsoft's slipped a few past the goalie using updates in the recent past, but as long as things are regularly checked, it's pretty easy to retrace and re-implement any settings that become lost. Existing permissions can be helpful in preserving settings as well, so if you've hit any golden combinations, please let us all know.

Looking forward to hearing more. *Smile

Our Best Always,

SLI - IT-IS Team

"Let us proclaim the mysteries of IT!" *Wink

 

The saying "there is no security where physical isolation/security is missing" always rings true. If some hacker/insider hooks up a USB drive and boots off some live-DVD, then the (then off-line) system (or critical data) drive could easily be cloned. Locking down sync tools on an on-line system just makes it a little more difficult to breach critical data (when the system has to be left on-line), but the physical media where any sync is destined still needs to be physically retrieved.

In a lab environment, I've used Linux's rsync to easily clone one Unix system to another, as that's often a desired mode of operation for doing experiments and code development.

The Win10 server I'm currently trying to fix is my home system. I don't have any of the concerns with multi-user use, insider hacking, etc., although I have to deal with hardware failure, power failure, etc., that could corrupt things.

I really just want to get my Windows Update working correctly again. The fact that "sfc /scannow," "dism," "Windows troubleshooting," and a miriad of other Win10-designated ways of fixing the system don't work is a manifestation of how Microsoft dropped the ball on this critical aspect. It doesn't help to have all these fancy schemes to plug up security holes when the code to implement these schemes can't even be applied (as the case of the cumulative update not installing).

Using the big-hammer approach of a full re-install is not a viable answer for many home users (or tiny businesses) as we have lots of old apps that work fine currently (but may not be re-installable). In a large work environment, the IT department would deal with these things, and have funds/manpower to re-install from scratch.

 

OK!

Let's start with this WU CU issue. What have you done in order to resolve this issue, except for running sfc, dism and system check?

Have you tried to disable fast startup, temporarily disable AV and finally, temporarily stopped all WU related services and deleted everything in C:\Windows\SoftwareDistribution\Download folder and rebooted system at least 2 times (first time will end up in WU error message if it was already trying to install something)?

Your WU issue could simply be an issue with AV corrupting your WU downloads or interfering with the configuration step.

What about Hypervisor support? Have you tried booting without hypervisor support? I've solved many issues in the past by doing this in Admin CMD:
Code: bcdedit /set hypervisorlaunchtype off shutdown /r /t 0 [/quote] To re-enable hypervisor type:
Code: bcdedit /set hypervisorlaunchtype on[/quote] OR
Code: bcdedit /set hypervisorlaunchtype auto[/quote]

 

@Win10 User , You're completely right about that, and all too often Microsoft goes script crazy and just recites "Run SFC /Scannow" & "dism.exe /online /cleanup-image /restorehealth" to a degree that it's almost a cultesque mantra now. Unfortunately it's a mantra that's been catching on in the industry, and keeps people from doing any real troubleshooting because they're constantly having to reset their equipment followed by addressing even more issues related to any corruption that occurs after the reset.

Are you using Group Policy at all to manage Windows Updates?

The reason we ask is because we often retest a very focused configuration for Windows Updates within Group Policy where everything is setup accordingly in a very specific manner for each setting; however for the past 2 years, it's never worked properly despite the settings being spot on. We do use a minimal configuration to prevent Windows Update from meddling with drivers, but obviously in our configuration, we're just using the straight up Windows Update, and we don't support use of WSUS because we use a Linux-based backend for everything else.

If you think it would help, it'll take us some time, but we could post the Windows Updates settings that we do keep configured, and then just bear in mind that all other settings for Windows Updates in our Machine Group Policy are set to non-configured. This approach has kept Windows Updates working for us without any meddling from Microsoft's proprietary and often very mediocre drivers that don't work nearly as well as the hardware component's manufacturers drivers often do. The only exception we've ever found to this is actually using things like Memory Card Reader-Writer assemblies, but that's about all Microsoft Drivers have been good for in our own experience.

We're all too aware, you've probably already done this, but instead of our assuming, we have to ask just to cover it; Have you tried re-reregistering services, repairing the Visual C Runtimes, and repairing the Microsoft .NET runtimes?

Please let us know if you think any of our offerings would be helpful.

Best Always,

SLI - IT-IS Team

"IT personnel using scripts are like actors, 'Oh, I'm not really a technician, but I play one at work.'"

 

An additional note: By any chance, have you been fiddling with C:\Program Files\WindowsApps folder permissions? I don't recommend touching the StoreApp folder permission, since the whole thing is very fragile and can break the whole system. I've done this out of curiosity and MAN was I in trouble for a while. Maybe I locked out the correct user/application by mistake, but got it sorted out in the end though. *Smile

 

Both RoboCopy and Xcopy have a /m option...

...which I use for backups in a batch file I wrote. I use it to do incremental backups.

But RoboCopy is more sophisticated than the old DOS Xcopy command. It has many more options available - including mirroring, for example. More info from TechNet here...
https://social.technet.microsoft.com...s.aspxit's

https://technet.microsoft.com/en-us/.../ee851678.aspx

 

Hi folks
Is the target destination for the copies a NAS / Linux server.

If it is there's an absolute peach of a program to do all this stuff and a huge amount more . It's a GUI front end to the RSYNC Linux program. Install GRSYNC on your server -- great thing also in running these backup things from a NAS is that you can schedule these backup jobs to run whenever you want (CRONTAB etc).

For Windows only systems ROBOCOPY is probably your best option but I am always reading that people have problems with it.

Here's a typical screen shot of GRSYNC - note there's zillions of options too - if you have a NAS type server really have a look at this program. Showing standard basic options but loads of advanced stuff too. Simply connect / mount your Windows share to the server and execute. !!





Cheers
jimbo

 

I've tried pretty much everything under the sun found doing Web searches for solutions.

But regardless, the _one_ thing that should have worked, booting off the Win10 1709 DVD and using its "sfc /scannow" and live-CD based tools should have fixed the off-line NTFS file system. But, I was totally dismayed when the live-DVD sfc exhibited exactly the same behavior as the on-line sfc, with the "Windows Protection System could not start the repair service" error. This _is_ totally bizarre, as a live-DVD (or alternate running OS) solution should _always_ work fixing an off-line file system. Microsoft engineering failed to understand this simple, basic, concept. And yes, I did explicit-path execution of the sfc.exe from the installation DVD's repair cmd shell to ensure I ran the correct executable.

Yup, tried it all.

Not running any AV other than Microsoft Windows Defender.

Perhaps WU downloads are being corrupted, but this is a small likelihood. A failing "sfc /scannow" indicates a much deeper problem.

To re-enable hypervisor type:
Code: bcdedit /set hypervisorlaunchtype on[/quote] OR
Code: bcdedit /set hypervisorlaunchtype auto[/quote] [/quote] I may try this, but am not hopeful it'll help.

I'm more inclined to give up on running (on-line) file system fixing because all the ownership/permission issues are inherent road blocks to updating/modifying corrupted files. But, the Windows install DVD repair tools are no help.

Someone may have to create a Linux live-DVD with appropriate NTFS tools to fix all the broken Windows file system repair mechanisms. Sigh.

 

Nope.

I'm a software engineer with several decades of experience.

 

Cool! *Smile

One more thing that I forgot to ask earlier and could be relevant. Which Windows 10 Edition, Build and Revision are you running and having issues with? Also would it be possible to get a detailed hardware list of your troublesome PC?


Windows Defender has caused a lot of havoc every now and then on my systems, so disabling this until problem is fixed, eliminates a lot of layers that could go wrong. I think you should try it too.

There are some builds where sfc online and offline commands were broken for some reason. I don't remember which builds though.

 

@Win10 User ; Well, guess that also answers the question of if you've re-registered services and run the Visual C & .NET Library repairs to a definitive "Yes!" *Wink

What about Group Policy management of Windows Updates? We're all too aware our own end that our full set of desired defaults for management of Windows Updates in Group Policy don't work because there are now more bugs than a bait store present in some areas of the Group Policy Manager, and have been for some time. Any possibility you're having a similar issue?

Also, @slicendice is right, Windows Defender actually has some modules in need of recall because they weren't ready for market and frequently corrupt data as follows:

[SOURCE URL: https://answers.microsoft.com/en-us/...=1521661574323 ]
Windows Defender Function C - Network Inspection Service (This service is loaded with bugs, and it is suggested to Microsoft's developers that it be removed from ALL versions of Windows until it is retooled and is ready for market because at present all it does is corrupt data)
Windows Defender Function D - Anti-Exploit Service (This service is also loaded with bugs, and it is suggested to Microsoft's developers that it be removed from ALL versions of Windows until it is retooled and is ready for market because at present all it does is corrupt data)

The above defective services can be disabled in Group Policy, and with a little extra effort in the Registry can also be permanently disabled as well.

SLI - IT-IS Team