Our website runs on a pair of colo-ed web servers that get syncronised together by Application Center. Let's call the servers Andy and Betty (names have been changed to protect the innocent). Both sit behind a server iron that does load balancing.
Website updates get published to Andy via an automatic FTP service. That works fine, although it can be a little sluggish.
After the files appear on Andy, Application Center pops up and says 'Hey, time to copy some files and permissions over to Betty!'. And lo, the files are copied and Betty looks the same as Andy.
However, we have noticed recently that Betty has not been looking the same as Andy. Specifically, a number of zero-byte files have appeared on Betty, where their equivalent back on Andy are perfectly okay. Mostly these are GIFs and PDFs, but we have had a few important pages get zeroed out.
Unfortunately there's no evidence in the logs about why this is happening. It's not consistent, and we have not been able to replicate it. At the moment we're just letting the world see Andy, and Betty is invisible to the public. Andy is man enough for the job but we'd really like them both to work in partnership.
While we try to work out what's going on (with help from Microsoft), I've written a Windows service app that detects zero-length files. It monitors the file system and if any files become zero-length and stay that way for more than a few minutes, the administrators are alerted. It's a simple little app; if anyone wants to see it or have a copy, they're more than welcome. It's in C#.
If anyone can shed any light on our little problemette described above, we'll be very grateful.