How Windows backup backs up files

Hi. I've noticed that Windows Backup checks to see IF files on the hard drive have changed since the last backup, and only backs up the files that have changed, but what about sections WITHIN the file?

I have several large VHD files which I am backing up, and it seems that it backs up each ENTIRE VHD file each time, not just the sections that have changed. This ends up taking a ridiculous amount of time to run a backup (and greatlly slows down disc I/O throughput during the backup). Is Windows Backup designed to re-copy an ENTIRE file? Wouldn't it be more efficient if it could somehow only copy the bytes in the file that have changed?

It's a trade-off. While seeing if a file has changed is (ostensibly) as simply as looking at its last modified date, to find out which sections have changed you would need to change the entire original file and the entire modified file. Needless to say, this could take a lot of I/O and time, and is only really worth it if only a small percentage of the file has changed.

The designers of Windows Backup apparently took the view that, on average, it wasn't worth the overhead, but of course you can find corner cases (like VHD files) where it would've been.

You could mount the VHD and then run backup on it. This seems a lot more complicated than setting the archive attribute, but such is life.

-Josh

@JoshRoss:I have lots of VHD files which I am managing for other users, so starting up the user's VM could be an invasion of their privacy, and would be a violation of my agreement with my customers. I run a website which allows users to host virtual machines using my servers. Privacy is something that me and my company take very seriously. It just seems like there should be a better way in the Windows 7/Windows Server 2008 R2 OS to back up files more efficiently.

@IDWMaster: Would it matter if the tools were automated? How about offering an agent to run on the client's vm, that used a repository that you host?

I think you need VHD backup utility like how SQL has their own backup utility? As far as OS concern, both VHD and DB are a single file. Thus, it is changed and will be backup as a file.

I think it is better to back them up seperately, just like DB.

13 hours ago, IDWMaster wrote

@JoshRoss:I have lots of VHD files which I am managing for other users, so starting up the user's VM could be an invasion of their privacy, and would be a violation of my agreement with my customers. I run a website which allows users to host virtual machines using my servers. Privacy is something that me and my company take very seriously. It just seems like there should be a better way in the Windows 7/Windows Server 2008 R2 OS to back up files more efficiently.

There are drive solutions that can take immediate snapshots of currently working disks and they've been around for at least a decade.  They are expensive, though.  If you want the perfect solution, don't go with the one that comes free with the OS

I am thinking ....  Seems like you need a good storage system with some kind of SAN

and a backup system that uses that.

I ended up splitting the VHDs into multiple parts using a SegmentedStream (that way, one VHD spans multiple files). Then the stream of the VHD is loaded into the virtualization software and executed.

what is a segmented stream?

This might be a stupid question, but why not just use difference disks? If the drives didn't have much churn, you could save yourself a lot of time and tape.

-Josh

23 hours ago, JoshRoss wrote

This might be a stupid question, but why not just use difference disks? If the drives didn't have much churn, you could save yourself a lot of time and tape.

 

-Josh

The virtualization software I'm using doesn't support differencing disks.

1 day ago, SteveRichter wrote

what is a segmented stream?

A SegmentedStream is a stream which automatically creates multiple files with a specified maximum size, instead of one large file.

For example:

SegmentedStream sampleStream = new SegmentedStream(104857600);
sampleStream.Write(ahugebytearray,0,ahugebytearray.Length);

//104857600 is the maximum length per file. If more data than this is saved to the stream, it will automatically create multiple files.