CC @spevans/@millenomi

@swift-ci create

I dont think this is a bug.

From the documentation of FileHandle.readabilityHandler @ https://developer.apple.com/documentation/foundation/filehandle/1412413-readabilityhandler
it states thats the block is executed when there is data to read. However it doesnt say that the block will be executed on EOF condition.

You may sometimes see an extra call of the block when there is no data left to read but it is not guaranteed to happen.

Infact, if the test is modified slightly with some extra debugging:

import Foundation

let process = Process()
process.executableURL = URL(fileURLWithPath: "/usr/bin/env")
let g = DispatchGroup()

let stdoutPipe = Pipe()
let stderrPipe = Pipe()
process.standardOutput = stdoutPipe
process.standardError = stderrPipe
g.enter()
g.enter()
var numberOfCallsOut = 0
var numberOfCallsErr = 0
stderrPipe.fileHandleForReading.readabilityHandler = { handle in
    let data = handle.availableData
    if data.isEmpty {
        print("stderr EOF")
        numberOfCallsOut += 1
        if numberOfCallsOut == 1 { g.leave() }
    } else {
        print("stderr: \(data)")
    }
}
stdoutPipe.fileHandleForReading.readabilityHandler = { handle in
    let data = handle.availableData
    if data.isEmpty {
        print("stdout EOF")
        numberOfCallsErr += 1
        if numberOfCallsErr == 1 { g.leave() }
    } else {
        print("stdout: \(data)")
    }
}

try! process.run()
fputs("!", stdout)
fflush(stdout)

g.wait()
print("numberOfCallsOut:", numberOfCallsOut)
print("numberOfCallsErr:", numberOfCallsErr)

On macOS it gives the following output:

 swift ./sr-12080.swift
!stderr EOF
stderr EOF
stderr EOF
stderr EOF
stderr EOF
stderr EOF
stderr EOF
stderr EOF
stderr EOF
stderr EOF
stderr EOF
stdout: 1104 bytes
stderr EOF
stderr EOF
stderr EOF
stderr EOF
stderr EOF
stderr EOF
stdout EOF
stderr EOF
stdout EOF
stderr EOF
stdout EOF
stderr EOF
stdout EOF
stderr EOF
stdout EOF
stderr EOF
stdout EOF
stderr EOF
stdout EOF
numberOfCallsOut: 17
numberOfCallsErr: 7
stderr EOF
stdout EOF
stderr EOF
stderr EOF
stdout EOF
stderr EOF
stdout EOF

[...repeats until contexts returns to the main thread and it exits...]

Where it sees an EOF straight away. If anything is a bug here its probably that the handler gets called too many times on macOS when there is nothing to read. I think it only 'works' on macOS becuase there are so many events one of them is going to have no data available.

I think the correct solution is to use process.waitUntilExit() to know that the sub-process has finished and there is no more data.

On Linux the output is:

swift-DEVELOPMENT-SNAPSHOT-2020-01-30-a-ubuntu18.04/usr/bin/swift swift-corelibs-foundation/sr-12080.swift
!stderr EOF
stdout: 2551 bytes
[...hangs...]
^C
 

So it hangs as it doesnt see an EOF on stdout but I think this is correct and trying to detect EOF this was isnt correct.

Comment by Raul Ignatus (JIRA)

Hi Simon,

You are right that this is the expected behaviour. We have assumed that because the docs for `availableData` say that it will be empty on EOF (https://developer.apple.com/documentation/foundation/filehandle/1411463-availabledata), the readabilityHandler will be called on EOF too. I think this can now be closed.

Thanks

Agreed! Pinging @millenomi to confirm that the readabilityHandler is not necessarily expected to fire on EOF.

Just ran into this myself, and I think I've discovered a bit of a pattern, at least for my case.

In my case the problem with the lack of a zero-length callback happened on `stdout`, and in particular it was because the subprocess closed its `stdout` before exiting. When I changed that to leave it open, then the zero-length callback consistently occurred. Not sure if that sheds some light but might be another data point.

Could it be that in the original example, `stderr` was closed before the process exit but `stdout` wasn't and that's why it would always hang?

> I think the correct solution is to use process.waitUntilExit() to know that the sub-process has finished and there is no more data.

Unfortunately this doesn't work as far as I can tell, since the last data on the pipe can arrive after this call returns. There is no guarantee in the documentation that this method waits for all data to be received.

So I thought the standard pattern here was to have a DispatchGroup with an `enter()` call for each asynchronous file handle handler and also for the termination of the process, and then a matching `leave()` call for each EOF condition seen on a file handle as well as when the termination handler is called.

If there is no guarantee about seeing a zero-length data as an indicator of EOF, then what's the best pattern for ensuring that all data has been received in addition to the process exiting?

Indeed, there is a race between Process termination and the data to be delivered through its pipes. The behaviour clearly differs on macOS and Linux: on macOS there is a final call with empty data, on Linux there isn't.

I double the last question for advice.