FileSystemWatcher Events and incomplete file errors

This is a question that has popped up numerous times, and I myself have had to deal with it:

You get a FileSystemWatcher event that a new file was created in a folder you are monitoring. So you try to process the file, but problems arise because the process that has written the file isn't finished writing it yet.

One way to handle this is to have the process that is writing the file send a second small text file consisting of something like the name of the file just written and the datetime that it was completed. You would get an event that this new file had been written, and your code logic would tell you what to do at this point. Of course, this assumes that you have sufficient control over the file - writing process to make this change.

One newsgroup poster commented, "We're running a windows service which contains several filesystemwatchers. Sometimes we hit spikes of 20-50.000 files in a matter of seconds and each file can potentially take some time to process. There's two problems we had to solve because of this:

1. If you spend too long time processing each event, and events arrive quickly, the internal Win32 filesystemwatcher buffer will fill up and start dropping events. See the InternalBufferSize property in FileSystemWatcher docs for more information.

2. Files not being closed when the event is raised."

The solution for #1 was to spend as little time as possible in the event handler. They simply post the data to a queue and let a separate worker thread process it.

In the worker thread they check each file before processing it. Some files must be handled in the order they arrive, so the queue will stall until the file at the head of the queue is closed.

Other files can be handled in any order, and they tag them with the worker thread iteration id and push them back to the end of the queue. To prevent endless processing of an opened file in a tight loop, they use the iteration id to check if they should Thread.Sleep or not before processing the next item in the queue.

To check if a file is available/closed, they use the following code:

bool IsFileClosed( FileInfo fi )
{
try
{
FileStream fs = fi.Open( FileMode.Open, FileAccess.ReadWrite,
FileShare.None );
fs.Close();
fs.Dispose();
return true;
}
catch
{
return false;
}
}

Seems like a workable solution to me. It's unfortunate to have to sink to the level of using exceptions for business logic, but I can't think of a better way.