[Mono-list] Easy way to deadlock Mono with concurrent HttpWebRequests

[Rubén de Alba]

Hi,

We can deadlock Mono launching a few HttpWebRequests. We can 
consistently reproduce in our app, but only sometimes happens in the 
attached program.

We initially detected this on a server pulling data from Azure. It 
deadlocks and it is unable to handle a single new request (since we rely 
on Socket.BeginReceive for that, and the callback is never invoked once 
the deadlock happens).

We suspect it could still be related to this ancient issue: 
http://www.mono-project.com/archived/articlethreadpool_deadlocks/

I have attached a small sample application to reproduce the issue. It 
occasionally deadlocks on *Mono 4.6.2*.*7* (and older) no matter whether 
the machine has 1 core or 8. What is consistent is that it performs 
worse than expected and fails. Not sure if this is a hint.

What the app does is:

* It launches 40 HttpWebRequest.GetResponse() in parallel.
* Most of them fail ("The authentication or decryption has failed.", 
"The request timed out") .
* The program retries if they fail - and retries also fail.
* CPU is very high (90% on my test box).
* After a while sometimes the entire program hangs.
* We also tried the HttpClient class using async calls and no threadpool 
but result wasn't any better.

Some interesting notes:
* The same code works well in .NET/Windows.
* And launching the 40 concurrent requests is super fast there compared 
to super slow using Mono.


Any hints or workarounds will be greatly appreciated :-P

Thanks!

Rubén,

www.plasticscm.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dot.net/pipermail/mono-list/attachments/20161223/75082055/attachment.html>
-------------- next part --------------
using System;
using System.IO;
using System.Net;
using System.Threading;

namespace threadpool_httpwebrequests
{
    class Program
    {
        static long mCounter = 0;

        static void Main(string[] args)
        {
            ServicePointManager.UseNagleAlgorithm = false;
            ServicePointManager.Expect100Continue = false;
            ServicePointManager.DefaultConnectionLimit = 500;

            int requestsCount = 40;

            if (args.Length == 1)
                requestsCount = int.Parse(args[0]);

            int ini = Environment.TickCount;

            Console.WriteLine("Going to launch {0} concurrent https requests...", requestsCount);

            LaunchClientRequest(requestsCount);

            WaitForRequestsToComplete(requestsCount);

            Console.WriteLine("The {0} requests finished {1} ms",
                requestsCount, Environment.TickCount - ini);
            Console.WriteLine("Test finished!");
        }

        static void LaunchClientRequest(int requestsCount)
        {
            for (int i = 0; i < requestsCount; i++)
                ThreadPool.QueueUserWorkItem(Read, null);
        }

        static void WaitForRequestsToComplete(int requestsCount)
        {
            while (Interlocked.Read(ref mCounter) != requestsCount)
                System.Threading.Thread.Sleep(500);
        }

        static void Read(object state)
        {
            for (int i = 0; i < 3; i++)
            {
                if (Read())
                    break;
            }
            Interlocked.Increment(ref mCounter);
        }

        static bool Read()
        {
            try
            {
                HttpWebRequest request = HttpWebRequest.Create("https://www.google.com") as HttpWebRequest;
                request.Method = "GET";
                request.ContentLength = 0;
                request.ContentType = "text/HTML";

                using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
                using (Stream stream = response.GetResponseStream())
                using (BinaryReader reader = new BinaryReader(stream))
                {
                    Console.WriteLine(reader.ReadString());
                    return true;
                }
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
                return false;
            }
        }
    }
}