0

I have a class method that i am trying to execute concurrently in almost 100 threads. This method is basically about HTTPWebRequests and database R/W My application is a console application which spins up threads to call this method.

Now when i dont thread and run 35-45 copies or instance of my console application, i get a faster/better performance, however when i run 100 threads, the performance drops by as much as 50%.

I have tried threadpool, array of background workers, threads and even tasks, but something is not working the way it should. This is how i am invoking a static method in main class that in turn class the other class method.

            while (flagProcess)
        {
            if (RAMPRead.CheckJobQueue())
            {


                bool allThreadsinUse = true;
                for (int i = 0; i < ts.Length; i++)
                {
                    if (ts[i] == null || ts[i].ThreadState != ThreadState.Running)
                    {
                        ts[i] = new Thread(() =>
                        {
                            RAMPit(session);
                        });
                        ts[i].Start();
                        ts[i].IsBackground = true;
                        allThreadsinUse = false;
                        while (!ts[i].IsAlive) { }
                        GC.Collect();
                        GC.WaitForPendingFinalizers();
                        GC.Collect();
                    }
                }
                if (!allThreadsinUse)
                {
                    Thread.Sleep(10000);
                    MemoryManagement.FlushMemory();
                }
            }
            else
            {
                Thread.Sleep(10000);
                MemoryManagement.FlushMemory();
            }
        }

        private static void RAMPit(object s)
    {
        string result = string.Empty;
        Session outsession = new Session();
        _sem.Wait();
        using (z z = new z("user", "pass"))
        {
            outsession = z.ProcessNext((Session)s, "user", "pass");
            session = outsession;
        }
        _sem.Release();
        MemoryManagement.FlushMemory();
        //return result;
    }

What am i doing wrong here OR how can i get to run many threads concurrently for faster output.

user1548923
  • 81
  • 2
  • 5
  • 1
    Do you have 100 cores in your CPU? – Blorgbeard Mar 09 '16 at 03:22
  • 3
    You shouldn't be using *anywhere near* 100 threads unless you're on a massively expensive machine. A good rule of thumb is the number of cores on in your CPU. Alternatively, you can use `Parallel.ForEach()` which will do a very good job at managing the number of threads needed – Rob Mar 09 '16 at 03:22
  • Thank you @Rob, If i dont multithread i am able to easily run 30-35 instance s of my console app, but switching to multithreading and trying to do the same thing with 40 threads doesnt work with same efficiency. I get your point about max threads relative to core. i appreciate your help – user1548923 Mar 09 '16 at 03:26
  • @user1548923 It depends entirely on how you're managing threads. Things like `Thread.Sleep(10000);` and forcing the garbage collection don't seem ideal (especially the sleep). Try using `Parallel.ForEach`. If you were to have a list of URLs, for example, you'd do something like this: `Parallel.ForEach(urls, url => { Process(url); })` with nothing else. Give that a whirl and let us know how it goes – Rob Mar 09 '16 at 03:29
  • It is not a coding solution, because it will not be solve your problem. Overclock your CPU. Lol. – Aizen Mar 09 '16 at 03:29
  • @Rob Thanks, I am trying Parallel.ForEach now, testing with it. – user1548923 Mar 09 '16 at 03:37
  • Parallel.ForEach is giving the same kind of results that threads threadpool and background workers did – user1548923 Mar 09 '16 at 03:51
  • If you still use `GC` in loop and `Thread.Sleep` your app can not be improved – NoName Mar 09 '16 at 03:53
  • 2
    If your CPU is pegged at 100%, you have enough threads. Any more is just adding overhead for no gain. – Blorgbeard Mar 09 '16 at 04:07
  • You need to determine what is throttling throughput. Is it I/O bound waiting for the HTTP traffic, is it database contention, CPU, memory, ... . Once you can measure the problem you can work on fixing it and tell what progress you're making. And there's something that just nags about the "spin wait while it's dead" (`while (!ts[i].IsAlive) { }`). – HABO Mar 09 '16 at 04:17

0 Answers0