There actually is a way that you can make a good guess that the browser is handling a file download. I asked this same question and TJ Crowder had a great idea:
What are techniques to get around the IE file download security rules?
I'm actually using the idea now and it works perfectly. The trick is to have the page send a "nonce" parameter back with a random string of characters in it. The page then starts polling document.cookie
every 100 milliseconds or so, checking to see whether that string of characters is in the cookie yet.
The server, in turn, sets a chosen cookie (doesn't really matter what it's called) to the value sent by the form in the "nonce" parameter. It then ships out the file download as it normally would.
When the HTTP response gets back to the browser, the cookie will be set. The Javascript that's polling for the cookie value will see that, and it will then know that the HTTP response is being processed. Now, it won't know of course that the user didn't hit "Cancel" on the file download.
If the server decides that the original request is in error (say, if the file download involves a form and the user supplied bad input or missed a field), then it won't set the cookie and can just respond with HTML for the error (or whatever).