1

I've read about this deadlock condition that I'm pretty certain is affecting my code (below). What I don't understand is: this code worked perfectly well running on Windows Server 2003 (.net 2.0) for the past 12 years. Now we've been trying to move it to Windows Server 2012, where it always deadlocks.

While my DLLs are built for "anyCPU" (still targeting .net 2.0), the executable process being run is absolutely 32-bit, and the move from Server 2003 to Server 2012 goes from a 32-bit to 64-bit OS.

I think I understand what to do to resolve the issue, but does anyone know why this behavior would have changed from Server 2003 to Server 2012?

public string DoMyProcess(string filenameAndPath, string arguments)
{
    string stdout="";
    int exitCode = 0;               

    try 
    {               
        ProcessStartInfo procStartInfo = new ProcessStartInfo();
        procStartInfo.FileName = filenameAndPath;
        procStartInfo.CreateNoWindow = true;
        procStartInfo.Arguments = arguments;            
        procStartInfo.RedirectStandardOutput = true;
        procStartInfo.UseShellExecute = false;

        System.Diagnostics.Process theProcess = null;
        try
        {
            theProcess = Process.Start(procStartInfo);
            theProcess.WaitForExit();

            exitCode = theProcess.ExitCode;

            // moving this ABOVE WaitForExit should eliminate deadlocks
            // But why did it always work on Server 2003 but not on Server 2012?
            stdout = theProcess.StandardOutput.ReadToEnd();

        }
        catch (System.Exception e)
        {
            string errMsg = e.Message;
            log_the_error("threw an exception: " + e.Message);
        }                   
    }           

    return stdout;
}

UPDATE:

Mystery deadlock still exists, even after changing the above code as recommended:

        try
        {
            theProcess = Process.Start(procStartInfo);
            stdout = theProcess.StandardOutput.ReadToEnd();                     
        }
        catch (System.Exception e)
        {
            string errMsg = e.Message;
            log_the_error("threw an exception: " + e.Message);
        }                   
    }

What other conditions could cause that deadlock? If I were to examine StandardError, would it reveal anything useful?

UPDATE #2:

FWIW, we have provisioned another Windows Server 2003 (32-bit), which runs IIS 6. That was the original machine configuration this code ran on for 12 years (with only an occasional deadlock). Our same code that deadlocks on Server 2012 IIS 8 DOES NOT DEADLOCK on this Server 2003.

We now have our own minimal and complete code that reproduces the issue. However, the .exe we've licensed that is being executed by the process has confidentiality clauses that prevent us from posting. I realize that doesn't help the experts here.

THE ONE HINT we've encountered is that when run via the Visual Studio 2013 debugger installed on the actual server, the process doesn't deadlock/hang, while invoking the process from a browser OUTSIDE the server does. And oddly -- from a browser ON THE 2012 SERVER we can't connect to that test page -- the browser just says "connecting" and eventually times out (however, other sites hosted by the same server / same IIS 8 CAN BE REACHED from a browser on the server!)

Since the same command line parameters manually run from either an admin command shell or a non-admin command shell works perfectly, it's hard to believe it's a 64-bit / WOW64 problem with this 32-bit executable or it's required DLLs. We continue to search for places our permissions may be causing problems (the process needs to write to a temp folder, which we've placed at c:\temp, for now).

SMGreenfield
  • 1,680
  • 19
  • 35
  • 1
    I know that faulty code pattern is prevalent on the internet, but try using the example MS provides for the [Process.OutputDataReceived Event](https://msdn.microsoft.com/en-us/library/system.diagnostics.process.outputdatareceived(v=vs.110).aspx?cs-save-lang=1&cs-lang=csharp#code-snippet-2) as a starting point. Be cognizant that that event and the other process events arrive on a secondary thread, so plan accordingly. – TnTinMn Sep 04 '16 at 03:28
  • re: your edited update, there's nothing left in the code that should cause deadlock. That doesn't mean there isn't any...you haven't provided a [mcve] so it's impossible to know what other code might be at work here. Reading `StandardError` may or may not show anything useful; you may be misinterpreting some sort of error or user prompt in the external process as a deadlock, when in fact it's just waiting for you or something else to respond. Note that if you _do_ redirect `StandardError`, then you introduce the possibility of deadlock and will want to make sure you avoid that. – Peter Duniho Sep 05 '16 at 23:07
  • 1
    I have a few different SO answers that, while answering questions different from the one you have here, do include examples of redirecting both `StandardOutput` and `StandardError` without deadlocking the code. You might find one or more of them useful: http://stackoverflow.com/a/33508142, http://stackoverflow.com/a/26722542, and http://stackoverflow.com/a/38881345 – Peter Duniho Sep 05 '16 at 23:10
  • 1
    To clarify this earlier comment I made: _"Reading StandardError may or may not show anything useful"_. I mean that it is possible the external process has written some useful information to `StandardError` and that if you read and display that, it could give you information on why the process isn't exiting. Or it might not. Impossible to say from where I am sitting. :) – Peter Duniho Sep 05 '16 at 23:11
  • @PeterDuniho -- The current implementation calls special command line executables that I can't publish their parameters for. I may be able to reproduce this deadlock using some standard, commonly available 32-bit .exe. I've simulated running the exact process from an admin CMD line, and get no errors. I've even tested with RedirectStandardOutput=false and no attempt to read from StandardOutput -- and it still deadlocks. Are there IIS execution privileges that might cause deadlocks? Anything other than waiting for I/O? – SMGreenfield Sep 05 '16 at 23:18
  • 1
    Sorry, I don't know enough about IIS to address anything that would be specific to that. I wouldn't think there should be anything unusual about that context that would change the basic diagnostic approach, but I can't be sure. I would suggest starting from the basics: you know that deadlock means two different threads of execution are waiting on each other; so, start by identifying what's waiting on what, being very careful to distinguish between true _waiting_ and just something that's taking a long time. – Peter Duniho Sep 05 '16 at 23:29
  • @PeterDuniho -- What about if, after I were to start the process, I immediately dumped logging data into an SQL table BEFORE the process completed? Might that cause a deadlock? – SMGreenfield Sep 05 '16 at 23:50
  • I guess that depends on where the "logging data" came from and whether it has to wait on your call to `ReadToEnd()` in some way, as well as whether the external process somehow also affects the access to your SQL server. Lacking a [mcve], there are just too many different possibilities for me to be able to say anything even remotely conclusive. – Peter Duniho Sep 05 '16 at 23:57
  • @PeterDuniho -- In the excellent example you outlined in http://stackoverflow.com/a/26722542, is there any other method to declare / implement async static Task ConsumeReader(TextReader reader)? This code is Visual Studio 2008 (.net 2.0)... – SMGreenfield Sep 06 '16 at 05:57
  • 1
    Sorry, no. That version requires at a minimum VS2010/.NET 4.0 for the Task Parallel Library features and even then you wouldn't get `async`/`await` (which is only in C# 5 and later). If you're stuck on .NET 2.0, you'll want to follow the example in my first link (http://stackoverflow.com/a/33508142). Note that that example is incomplete; you'll need to look at the original code in the question also to get a complete picture. Of course, no need to copy their `Console.SetOut()`, etc. The important part is the `Output...` and `ErrorDataReceived` events; you'll handle them specific to your needs. – Peter Duniho Sep 06 '16 at 06:01

1 Answers1

1

Without a good Minimal, Complete, and Verifiable code example, it's impossible to answer completely.

What I can tell you is that your code was always broken, and always had the potential for deadlock. You fail to read anything from the process until the process exits, but the process may not be able to exit, if it writes so much data to stdout that the buffer fills and blocks the process.

If you haven't recompiled anything, but have found that you see deadlock now when you didn't before, then the most likely explanation is that the process you're starting writes more to stdout than it used to. I.e. all of the output used to fit in the buffer before, but now it doesn't. (I guess it's also possible the buffer size was reduced in the newer OS, but that seems unlikely to me.)

You should go ahead and move the call to ReadToEnd(). In fact, you should do away with the WaitForExit() altogether. If you are calling ReadToEnd(), that won't complete until the process has in fact exited anyway, so calling WaitForExit() afterwards would be pointless.

Community
  • 1
  • 1
Peter Duniho
  • 68,759
  • 7
  • 102
  • 136
  • I believe you are correct. The code was part of an example from a vendor that was experienced enough that they should have caught this -- our engineers were newbs with .net at the time. – SMGreenfield Sep 04 '16 at 18:39