[Mono-dev] Anonymous Pipe Stream Implementation

Atsushi Eno atsushieno at veritas-vos-liberabit.com
Mon Jul 8 02:42:30 UTC 2013


Hello,

SmokingRope wrote:
> Hi, I'm poking through the pieces of the System.Core/System.IO.Pipes 
> code looking to build out anonymous pipes. I have at least a basic 
> working Unix implementation now. Though i imagine Win32 and Named 
> pipes have been broken (if they ever worked) in the process, and there 
> is undoubtedly some cleanup work required. I'd like to get some 
> feedback from people more knowledgeable than myself about mono at this 
> point so i can try and wrap this project up into a pull request.

That's great! Thanks for taking care of this.

I'm skipping some parts of your questions and writing for some parts 
that I can answer...


>
> My initial test case is this example from MSDN: 
> http://msdn.microsoft.com/en-us/library/bb546102.aspx
>
> Handle Inheritance
> -------------------------
> Presently, Process.Start() arbitrarily closes all handles in the 
> fork()'d process, except stdin / stdout / stderr. There exists a todo 
> comment in this same code (io-layer/processes.c) pointing out that 
> (when inherit_handles is TRUE, which it is) that something should be 
> done differently than when not...
>
> if (/inherit_handles/ != TRUE) {
>
> /* FIXME: do something here */
> }
>
> Without a far more expansive knowledge of mono i can't make a very 
> well informed decision about the liberties i can take in addressing this.
> The cleanest solution might be to use FD_CLOEXEC on all the existing 
> fd's and (at least in process.Start case) the handles that should be 
> cleaned up would be closed automatically / inherited automatically. 
> Are there compatibility reasons this cannot be done? Has there been 
> any consideration for this that could be drawn upon?
> >From information i do have, it seems like these pipe handles could 
> also be flagged as 'inheritable' in the internal handle registry and 
> then after fork() mono could leave those open, while closing 
> everything else.
>
> In addition to this basic test case, i'm not currently aware of how 
> anonymous pipes should behave when you invoke Process.Start() several 
> times with an anonymous pipe. Without using FD_CLOEXEC it would be 
> difficult and messy to (in a 'execd' child process) to identify all 
> the pipe handles at startup and register them in the mono handle 
> registry as inheritable again (so that Process.Start in that child 
> process would pass these handles on again to its children). If 
> anonymous pipes are supposed to support this, it lends further support 
> for the FD_CLOEXEC solution.
>
> API Methods Difficult To Implement In Unix
> -------------------------
> There doesn't seem to be a clear specification for the anonymous pipe 
> feature and i've so far had to make some little assumptions / 
> inferences based on the documentation on MSDN. That said, the posix 
> anonymous pipe standard lacks a few features which are pertinent to 
> the API for anonymous pipe streams. In particular there are two 
> conflicting features:
>
>      * WaitForPipeDrain 
> <http://msdn.microsoft.com/en-us/library/system.io.pipes.pipestream.waitforpipedrain.aspx>() 
> defined on the AnonymousPipeStream requires some ability to peak into 
> an anonymous pipe and wait for the pipe's buffer to become empty
>      * Write / Read / etc... is supposed to throw an IOException if 
> the other end of the pipe stream has been closed (i have interpreted 
> the wording 'pipe is broken' from the msdn docs to mean this, although 
> there is a chance i am mis-interpreting this too)
>
> These two features conflict when trying to implement them in unix 
> through minor hacks / workarounds.
>
> I can implement WaitForPipeDrain() by leaving the read end of the pipe 
> handle open in the writer process, and calling poll(3) on the read end 
> repeatedly until the POLL_IN flag is gone.
>
> I can detect that the pipe is broken by closing the read end on the 
> writer, (and the write end on the reader) and calling poll(3), 
> throwing this "pipe broken" error when a POLLERR /. POLLHUP is issued. 
> One of these two flags will be set if all handles to the other end of 
> the pipe have been closed.
>
> In order to implement both of these conflicting features i actually 
> need to create two pipes (one where i can leave the read end open on 
> the writer process, and one where i can close it). This is not very 
> nice in the case that system resources might be at a premium, however 
> i haven't been able to devise any other way to get both features 
> either. Would there be any major objection to implementing it like 
> this (which as i have noted seems to be the only way)?

I would say that opening two pipes is of less concern. If people need 
very efficient named pipes then they should rather use platform-specific 
stuff.
>
> Using two separate pipes like this, the GetClientHandleAsString() 
> method is going to be significantly different from the win32 version. 
> When the client is supposed to be the writer, the client handle as 
> string must contain three separate fds. When the client is the reader 
> two fd's must be passed. I don't see any reason that this would bother 
> anybody, but if it's relevant let me know!

That method can return totally different string than what .NET returns. 
It's weird that if an application totally depends on certain format of 
the return values of it and I believe we can ignore them.

>
> Compatibility
> -------------------------
> I can create pipes and manipulate them without adding any new library 
> references but i do depend on using the poll(3) function more heavily 
> than it seems to be presently (it's only exposed for Network Sockets 
> presently). Are there any portability / compatibility considerations 
> with poll(3) that might need to be addressed?
>

I guess Mono.Posix.dll has a lot more bound API that you want to use (it 
is already referenced by System.Core.dll for desktop CLR_PROFILE in 
System.Core/Makefile).

If that still needs to touch our runtime (io-layer) then it is of our 
concern yes, as we are building runtime not only for desktop. Yet we 
have couple of defs (PLATFORM_ANDROID etc.), so as long as things can be 
easily conditioned out it would work.

> Versioning
> -------------------------
> I'm not familiar with how versioning between .NET 1/2/3/etc... is 
> being handled. Anonymous pipes are introduced in .NET 3.5. Do these 
> classes need to be hidden to older versions, etc... Maybe there is a 
> wiki page i haven't found discussing some of this?
>

Our least supported Framework API is 2.0 runtime, which covers 2.0, 3.0, 
3.5 and 3.5 SP1 API.  So, you don't need to add any condition. For 4.0 
specific types and members, use "#if NET_4_0" for all of them (you can 
hide them by changing them as internal for !NET_4_0).

> I have added several new internal calls to the MonoIO utility, 
> specifically for poll(3) and pipe handle manipulation. Does adding 
> internal calls require any versioning considerations?
>
> Testing
> -------------------------
> In order to write a test suite for anonymous pipes i need to start new 
> processes and i will need to use some sort of IPC mechanism to 
> communicate certain errors back to the test process. This seems like a 
> rather unique challenge compared to most of the testing needed for mono.
>
> Has any of this multi-process unit testing been done before in mono, 
> or are there any considerations that might be relevant?
>

Not everything has to be in our existing NUnit test style. We have a 
couple of standalone tests (e.g. in System.Web.Services or System.Xml). 
Probably you can create your own test runner to deal with multiple 
processes.

Please do not hesitate to show us your ongoing changes. I'm sure that 
almost no one cared this API (at least no one contributed to the 
implementation), so you're most likely the one at best knowledge here :) 
And with code people can talk better.

Atsushi Eno



More information about the Mono-devel-list mailing list