[Mono-dev] Potential issue with async sockets in Raspberry ARM – Mono 4.4.0

[Greg Young]

We dealt with an issue similar to this long ago that was due to
threadpool requests being dropped occasionally. How reproducible is
your issue?

On Mon, Sep 12, 2016 at 11:45 PM, psantosl at codicesoftware.com
<psantosl at codicesoftware.com> wrote:
> Hi Alan,
>
> I'll try to create a small test case. We are experiencing this running a
> Plastic SCM server on raspberry, and a client on a different machine. It
> will take me a few hours to isolate a test case taking just the required
> parts from the plastic network protocol.
>
> Meanwhile: do you have any suggestion to try to check whether my assumption
> is correct? I mean, I suspect somehow the BeginReceive never awakes. I guess
> it is using epoll down the stairs. Any log or something I can build to make
> sure the issue is on my code and not on Mono?
>
> Thanks,
>
> pablo
>
>
> On 9/12/2016 11:37, Alan McGovern wrote:
>
> Can you share the code to repro the issue?
>
> Sent from my iPhone
>
> On 12 Sep 2016, at 09:39, "psantosl at codicesoftware.com"
> <psantosl at codicesoftware.com> wrote:
>
> Thanks for the hint Chris,
>
> We are actually enqueuing the request and doing the actual read on a
> different threadpool.
>
> That being said, we are reproducing this issue with a single client, I mean,
> single thread attending on the server. Not sure what can go wrong, tcp
> connection is still there, client can send, it simply looks like the server
> never wakes up to attend it, randomly.
>
> Any other hints would be appreciated.
>
> Thanks,
>
> pablo
>
> On 9/9/2016 18:11, Chris Swiedler wrote:
>
> From what I understand it’s dangerous to do blocking reads on the callback
> from something like BeginReceive. The callback will happen on a threadpool
> thread, and the blocking reads could then cause the threadpool to be
> exhausted. I don’t know if that’s causing your specific problem (it could,
> if the reads really do block and you have enough of them) but it’s something
> to watch out for.
>
>
>
> chris
>
>
>
> From: Mono-devel-list [mailto:mono-devel-list-bounces at lists.dot.net] On
> Behalf Of psantosl at codicesoftware.com
> Sent: Friday, September 09, 2016 2:42 AM
> To: mono-devel-list at lists.dot.net
> Cc: dude <rdealba at codicesoftware.com>
> Subject: [Mono-dev] Potential issue with async sockets in Raspberry ARM –
> Mono 4.4.0
>
>
>
> Hi,
>
> I’m posting here hoping someone can throw some light into the problem. I'm a
> little bit lost now.
>
> Our server code (Plastic SCM) running on Raspberry on Mono 4.4.0 (exact
> version below) ends up not awaking from socket.BeginReceive after a while.
>
> I mean, client connects and requests data in 4MB chunks, and depending on
> the run, it can transfer a few GB but it ends up not awaking. The client
> just sits waiting on a “socket recv” but the server doesn’t answer.
> Connection is established (can be checked at OS level).
>
> The code could be simplified as follows:
>
>             mSocket.BeginReceive(buffer, 0, 0, SocketFlags.None,
>                 OnMessageReceived, null);
>
> OnMessageReceived => does the EndReceive and then reads data and enqueues
> the request for a threadpool to attend it. Once the request is attented and
> the response sent, BeginReceive is invoked again. Important: all
> “BeginReceive()” calls are done from the same thread which NEVER dies.
>
> All we use the BeginReceive for is to decouple socket from thread, so we do
> not have a 1-1. You see we do pass “zero” as bytes to read, because all we
> want to achieve is to get awakened when new data is received, then just read
> using blocking calls (no async).
>
> I’m asking if there could be something about Mono because I read 4.6 release
> notes (http://www.mono-project.com/docs/about-mono/releases/4.6.0/) and the
> “atomic fixes for ARM64”. Could it be related somehow.
>
> The same code runs on Windows and Linux PCs (even Macs) without issues. We
> use the same code on all our production servers and even our Cloud roles,
> and we are not aware of issues.
> Now we are testing a new faster storage and using Raspberry to check how
> fast it goes on slower hardware.
>
> Complete Mono version:
> Mono JIT compiler version 4.4.0 (tarball Tue Jun 14 13:44:06 UTC 2016)
> Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors.
> www.mono-project.com
>         TLS:           __thread
>         SIGSEGV:       normal
>         Notifications: epoll
>         Architecture:  armel,vfp+hard
>         Disabled:      none
>         Misc:          softdebug
>         LLVM:          supported, not enabled.
>         GC:            sgen
> .
>
> Thanks!
>
> Pablo Santos
> www.plasticscm.com
>
>
> _______________________________________________
> Mono-devel-list mailing list
> Mono-devel-list at lists.dot.net
> http://lists.dot.net/mailman/listinfo/mono-devel-list
>
>
>
> _______________________________________________
> Mono-devel-list mailing list
> Mono-devel-list at lists.dot.net
> http://lists.dot.net/mailman/listinfo/mono-devel-list
>



-- 
Studying for the Turing test