[Mono-dev] Potential bug in NetworkStream in Raspberry ARM – Mono 4.4.0

Wed Sep 21 15:09:13 UTC 2016

I created a PR for this:

https://github.com/mono/mono/pull/3603 <https://github.com/mono/mono/pull/3603>

I submitted the PR for the 4.6 branch, although I would hope we can also put this in the current GA 4.4 branch (and master).  I assume multiple PR’s would be necessary.

> On Sep 21, 2016, at 4:08 AM, psantosl at codicesoftware.com wrote:
> 
> Thanks Dave,
> 
> I was only able to reproduce it (even on LAN) on slow hw like the raspberry and doing parallel requests. Single thread was unable to reproduce the error, I sent several GBs this way. But with 4 threads, it fails in like 8 seconds.
> 
> pablo
> 
> On 9/20/2016 14:30, David Curylo wrote:
>> Pablo,
>> 
>> I couldn’t find evidence of anyone already working on this issue, so I started on a PR yesterday.  I was a little blind after 8 hours of reading through tcpdump .pcap files :) but will submit it today with as good of test coverage as I can get without a remote system.  I’ll post here when I submit it.
>> 
>> -Dave
>> 
>>> On Sep 20, 2016, at 3:44 AM,  <mailto:psantosl at codicesoftware.com>psantosl at codicesoftware.com <mailto:psantosl at codicesoftware.com> wrote:
>>> 
>>> Hi David,
>>> 
>>> Great.
>>> 
>>> So, will you make a pull request? Or is there one already in progress?
>>> 
>>> Thanks,
>>> 
>>> pablo
>>> 
>>> On 9/20/2016 3:47, David Curylo wrote:
>>>> I’m glad it helps.  AFAICT a slow network actually _helps_ trigger the issue, guessing it allows the kernel TCP buffers to fill and leads to this issue.
>>>> 
>>>> Here is an older mono NetworkStream.Write, which was making up for the issue by continuing to push data through when the buffers filled:
>>>> https://github.com/mono/mono/blob/mono-3.12.1/mcs/class/System/System.Net.Sockets/NetworkStream.cs#L414 <https://github.com/mono/mono/blob/mono-3.12.1/mcs/class/System/System.Net.Sockets/NetworkStream.cs#L414>
>>>> 
>>>> So I expect that will work around your issue, although it needs to be fixed in Socket.cs like the comments <https://bugzilla.xamarin.com/show_bug.cgi?id=43902#c3> in the bug.
>>>> 
>>>>> On Sep 19, 2016, at 6:12 PM,  <mailto:psantosl at codicesoftware.com>psantosl at codicesoftware.com <mailto:psantosl at codicesoftware.com> wrote:
>>>>> 
>>>>> Hi David,
>>>>> 
>>>>> Thanks for the info!!
>>>>> 
>>>>> So, does this fix the issue?
>>>>> 
>>>>>             while (sent < count)
>>>>>             {
>>>>>                 sent += mSocket.Send(buffer, offset + sent, count - sent, SocketFlags.None);
>>>>>             }
>>>>> 
>>>>> Or am I simply having good luck??
>>>>> 
>>>>> I'm testing through VPN myself, so maybe the slow network helps. But, we are able to see the same issue on LAN... always using the raspberry as server.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> pablo
>>>>> 
>>>>> On 9/19/2016 23:38, David Curylo wrote:
>>>>>> Pablo,
>>>>>> 
>>>>>> I believe I’m experiencing a similar issue, also reported by others here:  <https://bugzilla.xamarin.com/show_bug.cgi?id=43902>https://bugzilla.xamarin.com/show_bug.cgi?id=43902 <https://bugzilla.xamarin.com/show_bug.cgi?id=43902>
>>>>>> 
>>>>>> Mono’s Socket.Send doesn’t block until all data is sent.  The old NetworkStream in mono used to account for this with a while loop to continue sending, but the NetworkStream from MS Reference Source expects the Socket to block until all data is sent.  When the reference source was pulled in, the Socket.Send could sometimes fail to send all the data.  I can’t ever make this failure occur on localhost, but it will occur for me communicating with remote hosts.
>>>>>> 
>>>>>> I’m working on a PR now to have Socket.Send_internal loop until all data is sent so Socket.Send will block as the reference source expects it to, unless anyone else has any suggestions.
>>>>>> 
>>>>>> -Dave
>>>>>> 
>>>>>>> On Sep 19, 2016, at 9:33 AM,  <mailto:psantosl at codicesoftware.com>psantosl at codicesoftware.com <mailto:psantosl at codicesoftware.com> wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I have found a potential issue with NetworkStream on ARM (Raspberry).
>>>>>>> 
>>>>>>> It might be a stupid bug on my code too, but I don't see it.
>>>>>>> 
>>>>>>> 
>>>>>>> In short: a simple multi-threaded server running on Raspberry (same behavior on Mono 4.4.0 as Mono 4.6.0), client on W10:
>>>>>>> 
>>>>>>> 1. Fails to correctly send data using NetworkStream (server seems to send the data, but it never arrives to client).  <https://github.com/psantosl/mono-raspberry-socket-issue/blob/master/Program.cs>https://github.com/psantosl/mono-raspberry-socket-issue/blob/master/Program.cs <https://github.com/psantosl/mono-raspberry-socket-issue/blob/master/Program.cs>
>>>>>>> 1.1. Fails *only* using NetworkStream (no BufferedStream or BinaryReader/Writer around, and it still fails):  <https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-only-network-stream/Program.cs>https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-only-network-stream/Program.cs <https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-only-network-stream/Program.cs>
>>>>>>> 
>>>>>>> 2. Works correctly using Sockets (no streams on top).  <https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-use-socket-only/Program.cs>https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-use-socket-only/Program.cs <https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-use-socket-only/Program.cs>
>>>>>>> 
>>>>>>> 3. Works correctly using a custom NetworkStream:
>>>>>>> 3.1.   <https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-use-alternative-network-stream/Program.cs>https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-use-alternative-network-stream/Program.cs <https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-use-alternative-network-stream/Program.cs>
>>>>>>> 3.2.   <https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-use-alternative-network-stream/MyNetworkStream.cs>https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-use-alternative-network-stream/MyNetworkStream.cs <https://github.com/psantosl/mono-raspberry-socket-issue/blob/master-use-alternative-network-stream/MyNetworkStream.cs>
>>>>>>> 3.3. Although I just saw it timing out after transferring several hundred MB on each connection (I'm on a slow VPN connected to the Raspberry, although we reproduced it on LAN too). Interestingly, using the regular NetworkStream it fails inmediately.
>>>>>>> 
>>>>>>> 
>>>>>>> When it fails, what happens is that at a given point in time (in my tests like in a few seconds) one of the client threads (or more than one) NEVER exits the Read, data never arrives although the server log says the data was written, but it doesn't. Since I've set a timeout for the read operation, you see how the program stops working.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> == Additional tests performed ==
>>>>>>> I tried with different buffer sizes, it always fails with the NetworkStream.
>>>>>>> 
>>>>>>> I ran the same server code on other setups:
>>>>>>> 
>>>>>>> * Window 10/.NET: OK.
>>>>>>> * OpenSuse 13.2 + mono 4.0.4 (Stable 4.0.4.1/5ab4c0d): OK.
>>>>>>> * Raspberry: failed both with mono 4.4.0 and mono-4.6.0.245 (built from GitHub on the 746756c commit.
>>>>>>> 
>>>>>>> I originally discovered the issue running a Plastic SCM server on BananaPi <http://www.bananapi.org/> (like a Raspberry + SATA).
>>>>>>> 
>>>>>>> == Extra notes ==
>>>>>>> * I know the multi-threaded server is not optimal and thread-per-connection is really bad, this is just a test case.
>>>>>>> * Initially I thought we had an issue with async sockets, but after a few days I just isolated the problem in NetworkStream.
>>>>>>> 
>>>>>>> 
>>>>>>> Any hints would be appreciated.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> pablo
>>>>>>> 
>>>>>>> www.plasticscm.com <http://www.plasticscm.com/>
>>>>>>> _______________________________________________
>>>>>>> Mono-devel-list mailing list
>>>>>>> Mono-devel-list at lists.dot.net <mailto:Mono-devel-list at lists.dot.net>
>>>>>>> http://lists.dot.net/mailman/listinfo/mono-devel-list <http://lists.dot.net/mailman/listinfo/mono-devel-list>
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dot.net/pipermail/mono-devel-list/attachments/20160921/20022f5a/attachment-0001.html>