[SR-4754] [XCTest] Non-zero exit without further output #5028

Bouke · 2017-05-01T05:54:53Z


Previous ID	SR-4754
Radar	rdar://problem/31940383
Original Reporter	@Bouke
Type	Bug
Status	Resolved
Resolution	Done

Attachment: Download

Environment

Swift 3.1.1 / Ubuntu 16.04

Additional Detail from JIRA


Votes	0
Component/s	Package Manager
Labels	Bug, 3.1Regression
Assignee	None
Priority	Medium

md5: ca5cf60ed18982f3cf303205e081abf2

Issue Description:

A regression from 3.1 has caused a weird test problem on Linux. I have a single XCTestcase with a single test function. The output from `swift test` is the following. There's a few lines of summary missing and the exit code is 1. I'm not sure how to resolve this.

bouke@ubuntu-imac:~/iCloud/NetService$ swift test && echo "yay"
Test Suite 'All tests' started at 06:49:45.224
Test Suite 'debug.xctest' started at 06:49:45.225
Test Suite 'BrowserTests' started at 06:49:45.225
Test Case 'BrowserTests.testBrowse' started at 06:49:45.225
Test Case 'BrowserTests.testBrowse' passed (0.009 seconds)
Test Suite 'BrowserTests' passed at 06:49:45.235
     Executed 1 test, with 0 failures (0 unexpected) in 0.009 (0.009) seconds
bouke@ubuntu-imac:~/iCloud/NetService$ echo $?
1
bouke@ubuntu-imac:~/iCloud/NetService$

belkadan · 2017-05-02T17:41:17Z

Just in case, can you provide the content of the test?

belkadan · 2017-05-02T17:41:21Z

@swift-ci create

Bouke · 2017-05-02T21:23:08Z

Sure, see the full repo here: Bouke/NetService@4c20bd9

ankitspd · 2017-05-04T20:19:08Z

I get preconditional failure when using a recent snapshot

root@c760168d188f:/swiftpm# swift test
Compile Swift Module 'DNS' (5 sources)
Compile Cifaddrs Cifaddrs.c
Compile Swift Module 'Socket' (3 sources)
/swiftpm/.build/checkouts/DNS.git--8045046305985412307/Sources/DNS/Types.swift:165:35: warning: redundant conformance constraint 'IPType': 'IP'
    public static func ==<IPType: IP> (lhs: HostRecord<IPType>, rhs: HostRecord<IPType>) -> Bool {
                                  ^
/swiftpm/.build/checkouts/DNS.git--8045046305985412307/Sources/DNS/Types.swift:165:45: note: conformance constraint 'IPType': 'IP' inferred from type here
    public static func ==<IPType: IP> (lhs: HostRecord<IPType>, rhs: HostRecord<IPType>) -> Bool {
                                            ^
/swiftpm/.build/checkouts/DNS.git--8045046305985412307/Sources/DNS/Types.swift:165:35: warning: redundant conformance constraint 'IPType': 'IP'
    public static func ==<IPType: IP> (lhs: HostRecord<IPType>, rhs: HostRecord<IPType>) -> Bool {
                                  ^
/swiftpm/.build/checkouts/DNS.git--8045046305985412307/Sources/DNS/Types.swift:165:45: note: conformance constraint 'IPType': 'IP' inferred from type here
    public static func ==<IPType: IP> (lhs: HostRecord<IPType>, rhs: HostRecord<IPType>) -> Bool {
                                            ^
/swiftpm/.build/checkouts/DNS.git--8045046305985412307/Sources/DNS/Types.swift:165:35: warning: redundant conformance constraint 'IPType': 'IP'
    public static func ==<IPType: IP> (lhs: HostRecord<IPType>, rhs: HostRecord<IPType>) -> Bool {
                                  ^
/swiftpm/.build/checkouts/DNS.git--8045046305985412307/Sources/DNS/Types.swift:165:45: note: conformance constraint 'IPType': 'IP' inferred from type here
    public static func ==<IPType: IP> (lhs: HostRecord<IPType>, rhs: HostRecord<IPType>) -> Bool {
                                            ^
/swiftpm/.build/checkouts/DNS.git--8045046305985412307/Sources/DNS/Types.swift:165:35: warning: redundant conformance constraint 'IPType': 'IP'
    public static func ==<IPType: IP> (lhs: HostRecord<IPType>, rhs: HostRecord<IPType>) -> Bool {
                                  ^
/swiftpm/.build/checkouts/DNS.git--8045046305985412307/Sources/DNS/Types.swift:165:45: note: conformance constraint 'IPType': 'IP' inferred from type here
    public static func ==<IPType: IP> (lhs: HostRecord<IPType>, rhs: HostRecord<IPType>) -> Bool {
                                            ^
/swiftpm/.build/checkouts/DNS.git--8045046305985412307/Sources/DNS/Types.swift:165:35: warning: redundant conformance constraint 'IPType': 'IP'
    public static func ==<IPType: IP> (lhs: HostRecord<IPType>, rhs: HostRecord<IPType>) -> Bool {
                                  ^
/swiftpm/.build/checkouts/DNS.git--8045046305985412307/Sources/DNS/Types.swift:165:45: note: conformance constraint 'IPType': 'IP' inferred from type here
    public static func ==<IPType: IP> (lhs: HostRecord<IPType>, rhs: HostRecord<IPType>) -> Bool {
                                            ^
Compile Swift Module 'NetService' (10 sources)
Compile Swift Module 'NetServiceTests' (1 sources)
Compile Swift Module 'NetServicePackageTests' (1 sources)
Linking ./.build/debug/NetServicePackageTests.xctest
Test Suite 'All tests' started at 19:17:21.959
Test Suite 'debug.xctest' started at 19:17:21.960
Test Suite 'BrowserTests' started at 19:17:21.960
Test Case 'BrowserTests.testBrowse' started at 19:17:21.960
precondition failed: host name c760168d188f. should have suffix .local: file /swiftpm/Sources/NetService/Responder.swift, line 62
Current stack trace:
0    libswiftCore.so                    0x00007f3e55b35a78 <unavailable> + 3431032
1    libswiftCore.so                    0x00007f3e55b6c160 _swift_stdlib_reportFatalErrorInFile + 99
2    libswiftCore.so                    0x00007f3e5591ef40 closure #&#8203;1 in closure #&#8203;1 in closure #&#8203;1 in _assertionFailure(_:_:file:line:flags:) + 284
3    libswiftCore.so                    0x00007f3e55b1ceb1 <unavailable> + 3329713
4    libswiftCore.so                    0x00007f3e55b2ff99 <unavailable> + 3407769
5    libswiftCore.so                    0x00007f3e5591b8e4 <unavailable> + 1226980
6    libswiftCore.so                    0x00007f3e55a99030 specialized closure #&#8203;1 in _assertionFailure(_:_:file:line:flags:) + 172
7    libswiftCore.so                    0x00007f3e55ae280f <unavailable> + 3090447
8    libswiftCore.so                    0x00007f3e5591b8e4 <unavailable> + 1226980
9    libswiftCore.so                    0x00007f3e55a513f0 specialized _assertionFailure(_:_:file:line:flags:) + 144
10   NetServicePackageTests.xctest      0x000000000041b46e <unavailable> + 111726
11   NetServicePackageTests.xctest      0x000000000041a70c <unavailable> + 108300
12   NetServicePackageTests.xctest      0x000000000041a5c9 <unavailable> + 107977
13   NetServicePackageTests.xctest      0x0000000000428db1 <unavailable> + 167345
14   NetServicePackageTests.xctest      0x0000000000428c2a <unavailable> + 166954
15   NetServicePackageTests.xctest      0x000000000040b84b <unavailable> + 47179
16   NetServicePackageTests.xctest      0x000000000040bcb8 <unavailable> + 48312
17   NetServicePackageTests.xctest      0x000000000040b7e6 <unavailable> + 47078
18   NetServicePackageTests.xctest      0x000000000040bc4b <unavailable> + 48203
19   libXCTest.so                       0x00007f3e55e372df <unavailable> + 238303
20   libXCTest.so                       0x00007f3e55e2c851 <unavailable> + 194641
21   libXCTest.so                       0x00007f3e55e371ff <unavailable> + 238079
22   libXCTest.so                       0x00007f3e55e37f29 <unavailable> + 241449
23   libXCTest.so                       0x00007f3e55e3715b <unavailable> + 237915
24   libXCTest.so                       0x00007f3e55e37f79 <unavailable> + 241529
25   libXCTest.so                       0x00007f3e55e373f0 <unavailable> + 238576
26   libXCTest.so                       0x00007f3e55e154f0 XCTestCase.invokeTest() + 102
27   libXCTest.so                       0x00007f3e55e2b080 specialized XCTestCase.perform(_:) + 306
28   libXCTest.so                       0x00007f3e55e15430 XCTestCase.perform(_:) + 14
29   libXCTest.so                       0x00007f3e55e13520 XCTest.run() + 549
30   libXCTest.so                       0x00007f3e55e2b3a0 specialized XCTestSuite.perform(_:) + 766
31   libXCTest.so                       0x00007f3e55e2b3a0 specialized XCTestSuite.perform(_:) + 1745
32   libXCTest.so                       0x00007f3e55e2b3a0 specialized XCTestSuite.perform(_:) + 1745
33   libXCTest.so                       0x00007f3e55e15a10 XCTMain(_:) + 3552
34   NetServicePackageTests.xctest      0x00000000004bace4 <unavailable> + 765156
35   libc.so.6                          0x00007f3e53aae740 __libc_start_main + 240
36   NetServicePackageTests.xctest      0x000000000040b2e9 <unavailable> + 45801

Bouke · 2017-05-04T20:24:54Z

The code requires you to have a `.local` hostname:

sudo hostname my-hostname.local

ankitspd · 2017-05-04T20:29:31Z

Thanks, tests worked but I get code zero as output. Can you check if this happens for you on the latest trunk snapshot? https://swift.org/download/#snapshots

root@ankit:/swiftpm/NetService# swift test && echo "yay"
Compile Swift Module 'DNS' (5 sources)
Compile Swift Module 'NetService' (10 sources)
Compile Swift Module 'NetServiceTests' (1 sources)
Compile Swift Module 'NetServicePackageTests' (1 sources)
Linking ./.build/debug/NetServicePackageTests.xctest
Test Suite 'All tests' started at 19:27:11.667
Test Suite 'debug.xctest' started at 19:27:11.668
Test Suite 'BrowserTests' started at 19:27:11.668
Test Case 'BrowserTests.testBrowse' started at 19:27:11.668
Test Case 'BrowserTests.testBrowse' passed (0.0 seconds)
Test Suite 'BrowserTests' passed at 19:27:11.669
     Executed 1 test, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
Test Suite 'debug.xctest' passed at 19:27:11.669
     Executed 1 test, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
Test Suite 'All tests' passed at 19:27:11.669
     Executed 1 test, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
yay
root@ankit:/swiftpm/NetService# echo $?
0

Bouke · 2017-05-04T20:35:50Z

It still stops midway the tests on the last trunk snapshot:

bouke@ubuntu-imac:~/iCloud/NetService$ swiftenv local
DEVELOPMENT-SNAPSHOT-2017-04-24-a
bouke@ubuntu-imac:~/iCloud/NetService$ swift test && echo "yay"
Compile Swift Module 'Socket' (3 sources)
Compile Swift Module 'DNS' (5 sources)
Compile Cifaddrs Cifaddrs.c
Compile Swift Module 'NetService' (10 sources)
Compile Swift Module 'NetServiceTests' (1 sources)
Compile Swift Module 'NetServicePackageTests' (1 sources)
Linking ./.build/debug/NetServicePackageTests.xctest
Test Suite 'All tests' started at 21:33:24.307
Test Suite 'debug.xctest' started at 21:33:24.308
Test Suite 'BrowserTests' started at 21:33:24.308
Test Case 'BrowserTests.testBrowse' started at 21:33:24.308
Test Case 'BrowserTests.testBrowse' passed (0.01 seconds)
Test Suite 'BrowserTests' passed at 21:33:24.319
     Executed 1 test, with 0 failures (0 unexpected) in 0.01 (0.01) seconds
Test Suite 'debug.xctest' passed at 21:33:24.320
     Executed 1 test, with 0 failures (0 unexpected) in 0.01 (0.01) seconds
bouke@ubuntu-imac:~/iCloud/NetService$ echo $?
1

ankitspd · 2017-05-04T21:06:13Z

Could it be swiftenv that is causing the problem?

Bouke · 2017-05-04T22:50:20Z

Same result with a manual download:

bouke@ubuntu-imac:~/iCloud/NetService$ ~/swift-DEVELOPMENT-SNAPSHOT-2017-04-24-a-ubuntu16.04/usr/bin/swift test
Compile Swift Module 'DNS' (5 sources)
Compile Swift Module 'Socket' (3 sources)
Compile Swift Module 'NetService' (10 sources)
Compile Swift Module 'NetServiceTests' (1 sources)
Compile Swift Module 'NetServicePackageTests' (1 sources)
Linking ./.build/debug/NetServicePackageTests.xctest
Test Suite 'All tests' started at 23:48:48.992
Test Suite 'debug.xctest' started at 23:48:49.051
Test Suite 'BrowserTests' started at 23:48:49.052
Test Case 'BrowserTests.testBrowse' started at 23:48:49.052
Test Case 'BrowserTests.testBrowse' passed (0.013 seconds)
Test Suite 'BrowserTests' passed at 23:48:49.065
     Executed 1 test, with 0 failures (0 unexpected) in 0.013 (0.013) seconds
Test Suite 'debug.xctest' passed at 23:48:49.066
     Executed 1 test, with 0 failures (0 unexpected) in 0.013 (0.013) seconds
bouke@ubuntu-imac:~/iCloud/NetService$ echo $?
1

ankitspd · 2017-05-05T04:07:37Z

Can you try these commands:

$ swift build && ./.build/debug/NetServicePackageTests.xctest && echo $?

$ cd /tmp && mkdir hello && cd hello && swift package init && swift test && echo $?

Bouke · 2017-05-05T07:06:29Z

bouke@ubuntu-imac:~/iCloud/NetService$ swiftenv which swift
/home/bouke/.swiftenv/versions/3.1.1/usr/bin/swift
bouke@ubuntu-imac:~/iCloud/NetService$ .build/debug/NetServicePackageTests.xctest
Test Suite 'All tests' started at 07:53:24.981
Test Suite 'debug.xctest' started at 07:53:24.982
Test Suite 'BrowserTests' started at 07:53:24.983
Test Case 'BrowserTests.testBrowse' started at 07:53:24.983
Test Case 'BrowserTests.testBrowse' passed (0.009 seconds)
Test Suite 'BrowserTests' passed at 07:53:24.992
  Executed 1 test, with 0 failures (0 unexpected) in 0.009 (0.009) seconds
Illegal instruction (core dumped)

On some executions, the dumped core is earlier;

bouke@ubuntu-imac:~/iCloud/NetService$ .build/debug/NetServicePackageTests.xctest
Test Suite 'All tests' started at 07:53:43.005
Test Suite 'debug.xctest' started at 07:53:43.006
Test Suite 'BrowserTests' started at 07:53:43.006
Test Case 'BrowserTests.testBrowse' started at 07:53:43.007
Test Case 'BrowserTests.testBrowse' passed (0.009 seconds)
Test Suite 'BrowserTests' passed at 07:53:43.017
Illegal instruction (core dumped)

And the test:

bouke@ubuntu-imac:/tmp/hello2$ swiftenv which swift
/home/bouke/.swiftenv/versions/3.1.1/usr/bin/swift
bouke@ubuntu-imac:/tmp$ mkdir hello2 && cd hello2 && swift package init && swift test
Creating library package: hello2
Creating Package.swift
Creating .gitignore
Creating Sources/
Creating Sources/hello2.swift
Creating Tests/
Creating Tests/LinuxMain.swift
Creating Tests/hello2Tests/
Creating Tests/hello2Tests/hello2Tests.swift
Compile Swift Module 'hello2' (1 sources)
Compile Swift Module 'hello2Tests' (1 sources)
Linking ./.build/debug/hello2PackageTests.xctest
Test Suite 'All tests' started at 07:51:40.125
Test Suite 'debug.xctest' started at 07:51:40.160
Test Suite 'hello2Tests' started at 07:51:40.160
Test Case 'hello2Tests.testExample' started at 07:51:40.160
Test Case 'hello2Tests.testExample' passed (0.0 seconds)
Test Suite 'hello2Tests' passed at 07:51:40.161
     Executed 1 test, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
Test Suite 'debug.xctest' passed at 07:51:40.161
     Executed 1 test, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
Test Suite 'All tests' passed at 07:51:40.162
     Executed 1 test, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
bouke@ubuntu-imac:/tmp/hello2$ echo $?
0

ankitspd · 2017-05-05T07:39:02Z

I am able to reproduce with 3.1.1 linux toolchain. Looks like this is a crash in BlueSocket package at Socker.swift:3037

s = Glibc.sendto(self.socketfd, buffer.advanced(by: sent), Int(bufSize - sent), sendFlags, &addr, socklen_t(size))

Most likely some compiler bug which is resolved in master because it does work with latest snapshot.

ankitspd · 2017-05-05T07:43:06Z

Oh wait, it fails for @Bouke in latest snapshot, perhaps it not a deterministic didn't crash?

Bouke · 2017-08-26T09:56:37Z

I'm still running into this issue with the latest snapshot (4.0-DEVELOPMENT-SNAPSHOT-2017-08-21-a). I've had a test crash with an invalid operation (incorrect Data slicing). The output on the command line just stopped, no error or stack trace was given.

$ PYTHON=`which python` swift test -c release -Xswiftc -enable-testing
Compile Swift Module 'SipHash' (4 sources)
Compile Swift Module 'Cryptor' (10 sources)
Compile Swift Module 'SRPTests' (4 sources)
Test Suite 'All tests' started at 2017-08-26 10:34:36.334
Test Suite 'SRPPackageTests.xctest' started at 2017-08-26 10:34:36.334
Test Suite 'PySrptoolsTests' started at 2017-08-26 10:34:36.334
Test Case '-[SRPTests.PySrptoolsTests testClient]' started.
$

ankitspd · 2017-08-31T11:42:30Z

@Bouke can you provide the code that reproduces it? A simplified test case would help a lot.

Bouke · 2017-09-03T10:54:36Z

I've attached a simplified test case. On Ubuntu 16.04:

bouke@ubuntu:~/Developer/SR-4754$ /home/bouke/.swiftenv/versions/4.0-DEVELOPMENT-SNAPSHOT-2017-08-21-a/usr/bin/swift test; echo $?
Test Suite 'All tests' started at 2017-09-03 09:52:59.826
Test Suite 'debug.xctest' started at 2017-09-03 09:52:59.828
Test Suite 'SR_4754Tests' started at 2017-09-03 09:52:59.828
Test Case 'SR_4754Tests.testExample' started at 2017-09-03 09:52:59.829
1
bouke@ubuntu:~/Developer/SR-4754$

The failing code is this – invalid bounds on the DataSlice:

import Foundation
public func helloWorld() {
    let _ = Data([0, 1, 2, 3])[2..<4][0..<3]
}

The underlying bug is probably a missing bounds check on the DataSlice, which would've shown up if it threw a fatalError. However it doesn't, the tests just stops half-way with no output nor any indication of a failure. Since my original report the exit code has apparently changed from 0 to 1, so we've already seen some improvement.

ankitspd · 2017-09-04T09:32:01Z

Thanks @Bouke. This problem is easily reproducible with your new sample code. There is a "illegal hardware instruction" message produced by the compiler but swiftpm is not showing that (I'll fix that). An appropriate crash report is placed in `~/Library/Logs/DiagnosticReports`.

ankitspd · 2017-09-04T09:48:51Z

Looks like this message is not really produce by the compiler and is not available in stdout or stderr streams.

ankitspd · 2017-09-04T10:11:10Z

Ah, this happens when the process is exited via a signal.

ankitspd · 2017-09-04T10:14:46Z

I added a print statement when a test exits with a signal.
#1335

ankitspd · 2017-09-04T10:50:32Z

1101f5c

swift-ci transferred this issue from apple/swift-issues Apr 25, 2022

shahmishal transferred this issue from apple/swift May 4, 2022

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SR-4754] [XCTest] Non-zero exit without further output #5028

[SR-4754] [XCTest] Non-zero exit without further output #5028

Bouke commented May 1, 2017

belkadan commented May 2, 2017

belkadan commented May 2, 2017

Bouke commented May 2, 2017

ankitspd commented May 4, 2017

Bouke commented May 4, 2017

ankitspd commented May 4, 2017

Bouke commented May 4, 2017

ankitspd commented May 4, 2017

Bouke commented May 4, 2017

ankitspd commented May 5, 2017

Bouke commented May 5, 2017

ankitspd commented May 5, 2017

ankitspd commented May 5, 2017

Bouke commented Aug 26, 2017

ankitspd commented Aug 31, 2017

Bouke commented Sep 3, 2017

ankitspd commented Sep 4, 2017

ankitspd commented Sep 4, 2017

ankitspd commented Sep 4, 2017

ankitspd commented Sep 4, 2017

ankitspd commented Sep 4, 2017

[SR-4754] [XCTest] Non-zero exit without further output #5028

[SR-4754] [XCTest] Non-zero exit without further output #5028

Comments

Bouke commented May 1, 2017

belkadan commented May 2, 2017

belkadan commented May 2, 2017

Bouke commented May 2, 2017

ankitspd commented May 4, 2017

Bouke commented May 4, 2017

ankitspd commented May 4, 2017

Bouke commented May 4, 2017

ankitspd commented May 4, 2017

Bouke commented May 4, 2017

ankitspd commented May 5, 2017

Bouke commented May 5, 2017

ankitspd commented May 5, 2017

ankitspd commented May 5, 2017

Bouke commented Aug 26, 2017

ankitspd commented Aug 31, 2017

Bouke commented Sep 3, 2017

ankitspd commented Sep 4, 2017

ankitspd commented Sep 4, 2017

ankitspd commented Sep 4, 2017

ankitspd commented Sep 4, 2017

ankitspd commented Sep 4, 2017