Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-10801] swift test --filter should display summary for the entire group of tests run, not per-test #412

Open
neonichu opened this issue May 30, 2019 · 7 comments

Comments

@neonichu
Copy link
Member

Previous ID SR-10801
Radar rdar://problem/45138914
Original Reporter @neonichu
Type Improvement
Additional Detail from JIRA
Votes 0
Component/s Package Manager
Labels Improvement, StarterBug
Assignee ronaldsmartin (JIRA)
Priority Medium

md5: d467ad91fd39d257442a08b5a39a89a5

Issue Description:

When running `swift test` with no --filter, the summary at the bottom is useful to see how many tests failed:When running `swift test` with no --filter, the summary at the bottom is useful to see how many tests failed:

...
Test Case '-[IOTests.TypesTests testPacker]' passed (0.000 seconds).
Test Case '-[IOTests.TypesTests testStruct]' started.
Test Case '-[IOTests.TypesTests testStruct]' passed (0.000 seconds).
Test Suite 'TypesTests' passed at 2018-10-09 13:48:23.339.
     Executed 2 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Test Suite 'IOPackageTests.xctest' failed at 2018-10-09 13:48:23.339.
     Executed 21 tests, with 2 failures (0 unexpected) in 0.116 (0.117) seconds
Test Suite 'All tests' failed at 2018-10-09 13:48:23.339.
     Executed 21 tests, with 2 failures (0 unexpected) in 0.116 (0.117) seconds

With the --filter it seems to run each matched test individually as "selected tests", so the summary is printed per test meaning you can't see at a glance how many tests failed by looking at the last line:

Compile Swift Module 'IOTests' (8 sources)
Linking ./.build/x86_64-apple-macosx10.10/debug/IOPackageTests.xctest/Contents/MacOS/IOPackageTests
Test Suite 'Selected tests' started at 2018-10-09 13:50:42.838
Test Suite 'IOPackageTests.xctest' started at 2018-10-09 13:50:42.838
Test Suite 'MemoryTests' started at 2018-10-09 13:50:42.838
Test Case '-[IOTests.MemoryTests testPageList]' started.
Test Case '-[IOTests.MemoryTests testPageList]' passed (0.077 seconds).
Test Suite 'MemoryTests' passed at 2018-10-09 13:50:42.915.
     Executed 1 test, with 0 failures (0 unexpected) in 0.077 (0.077) seconds
Test Suite 'IOPackageTests.xctest' passed at 2018-10-09 13:50:42.915.
     Executed 1 test, with 0 failures (0 unexpected) in 0.077 (0.077) seconds
Test Suite 'Selected tests' passed at 2018-10-09 13:50:42.915.
     Executed 1 test, with 0 failures (0 unexpected) in 0.077 (0.077) seconds
Test Suite 'Selected tests' started at 2018-10-09 13:50:42.962
Test Suite 'IOPackageTests.xctest' started at 2018-10-09 13:50:42.962
Test Suite 'MemoryTests' started at 2018-10-09 13:50:42.962
Test Case '-[IOTests.MemoryTests testRead16]' started.
Test Case '-[IOTests.MemoryTests testRead16]' passed (0.075 seconds).
Test Suite 'MemoryTests' passed at 2018-10-09 13:50:43.038.
     Executed 1 test, with 0 failures (0 unexpected) in 0.075 (0.076) seconds
Test Suite 'IOPackageTests.xctest' passed at 2018-10-09 13:50:43.038.
     Executed 1 test, with 0 failures (0 unexpected) in 0.075 (0.076) seconds
Test Suite 'Selected tests' passed at 2018-10-09 13:50:43.038.
     Executed 1 test, with 0 failures (0 unexpected) in 0.075 (0.076) seconds
Test Suite 'Selected tests' started at 2018-10-09 13:50:43.085
Test Suite 'IOPackageTests.xctest' started at 2018-10-09 13:50:43.085
Test Suite 'MemoryTests' started at 2018-10-09 13:50:43.085
Test Case '-[IOTests.MemoryTests testRead32]' started.
Test Case '-[IOTests.MemoryTests testRead32]' passed (0.062 seconds).
Test Suite 'MemoryTests' passed at 2018-10-09 13:50:43.147.
     Executed 1 test, with 0 failures (0 unexpected) in 0.062 (0.062) seconds
Test Suite 'IOPackageTests.xctest' passed at 2018-10-09 13:50:43.147.
     Executed 1 test, with 0 failures (0 unexpected) in 0.062 (0.062) seconds
Test Suite 'Selected tests' passed at 2018-10-09 13:50:43.147.
     Executed 1 test, with 0 failures (0 unexpected) in 0.062 (0.062) seconds
Test Suite 'Selected tests' started at 2018-10-09 13:50:43.191
Test Suite 'IOPackageTests.xctest' started at 2018-10-09 13:50:43.191
Test Suite 'MemoryTests' started at 2018-10-09 13:50:43.191
Test Case '-[IOTests.MemoryTests testRead64]' started.
Test Case '-[IOTests.MemoryTests testRead64]' passed (0.060 seconds).
Test Suite 'MemoryTests' passed at 2018-10-09 13:50:43.251.
     Executed 1 test, with 0 failures (0 unexpected) in 0.060 (0.060) seconds
Test Suite 'IOPackageTests.xctest' passed at 2018-10-09 13:50:43.251.
     Executed 1 test, with 0 failures (0 unexpected) in 0.060 (0.060) seconds
Test Suite 'Selected tests' passed at 2018-10-09 13:50:43.252.
     Executed 1 test, with 0 failures (0 unexpected) in 0.060 (0.060) seconds
Test Suite 'Selected tests' started at 2018-10-09 13:50:43.297
Test Suite 'IOPackageTests.xctest' started at 2018-10-09 13:50:43.297
Test Suite 'MemoryTests' started at 2018-10-09 13:50:43.297
Test Case '-[IOTests.MemoryTests testRead8]' started.
/Users/best/Documents/IOTests/MemoryTests.swift:23: error: -[IOTests.MemoryTests testRead8] : XCTAssertTrue failed -
Test Case '-[IOTests.MemoryTests testRead8]' failed (0.059 seconds).
Test Suite 'MemoryTests' failed at 2018-10-09 13:50:43.356.
     Executed 1 test, with 1 failure (0 unexpected) in 0.059 (0.059) seconds
Test Suite 'IOPackageTests.xctest' failed at 2018-10-09 13:50:43.356.
     Executed 1 test, with 1 failure (0 unexpected) in 0.059 (0.059) seconds
Test Suite 'Selected tests' failed at 2018-10-09 13:50:43.356.
     Executed 1 test, with 1 failure (0 unexpected) in 0.059 (0.060) seconds
Test Suite 'Selected tests' started at 2018-10-09 13:50:43.403
Test Suite 'IOPackageTests.xctest' started at 2018-10-09 13:50:43.404
Test Suite 'MemoryTests' started at 2018-10-09 13:50:43.404
Test Case '-[IOTests.MemoryTests testWrite16]' started.
Test Case '-[IOTests.MemoryTests testWrite16]' passed (0.062 seconds).
Test Suite 'MemoryTests' passed at 2018-10-09 13:50:43.465.
     Executed 1 test, with 0 failures (0 unexpected) in 0.062 (0.062) seconds
Test Suite 'IOPackageTests.xctest' passed at 2018-10-09 13:50:43.466.
     Executed 1 test, with 0 failures (0 unexpected) in 0.062 (0.062) seconds
Test Suite 'Selected tests' passed at 2018-10-09 13:50:43.466.
     Executed 1 test, with 0 failures (0 unexpected) in 0.062 (0.062) seconds
Test Suite 'Selected tests' started at 2018-10-09 13:50:43.513
Test Suite 'IOPackageTests.xctest' started at 2018-10-09 13:50:43.514
Test Suite 'MemoryTests' started at 2018-10-09 13:50:43.514
Test Case '-[IOTests.MemoryTests testWrite32]' started.
Test Case '-[IOTests.MemoryTests testWrite32]' passed (0.071 seconds).
Test Suite 'MemoryTests' passed at 2018-10-09 13:50:43.585.
     Executed 1 test, with 0 failures (0 unexpected) in 0.071 (0.071) seconds
Test Suite 'IOPackageTests.xctest' passed at 2018-10-09 13:50:43.585.
     Executed 1 test, with 0 failures (0 unexpected) in 0.071 (0.071) seconds
Test Suite 'Selected tests' passed at 2018-10-09 13:50:43.585.
     Executed 1 test, with 0 failures (0 unexpected) in 0.071 (0.071) seconds
Test Suite 'Selected tests' started at 2018-10-09 13:50:43.633
Test Suite 'IOPackageTests.xctest' started at 2018-10-09 13:50:43.633
Test Suite 'MemoryTests' started at 2018-10-09 13:50:43.633
Test Case '-[IOTests.MemoryTests testWrite8]' started.
Test Case '-[IOTests.MemoryTests testWrite8]' passed (0.062 seconds).
Test Suite 'MemoryTests' passed at 2018-10-09 13:50:43.695.
     Executed 1 test, with 0 failures (0 unexpected) in 0.062 (0.062) seconds
Test Suite 'IOPackageTests.xctest' passed at 2018-10-09 13:50:43.695.
     Executed 1 test, with 0 failures (0 unexpected) in 0.062 (0.062) seconds
Test Suite 'Selected tests' passed at 2018-10-09 13:50:43.695.
     Executed 1 test, with 0 failures (0 unexpected) in 0.062 (0.062) seconds
@belkadan
Copy link

Still working on this, ronaldsmartin (JIRA User)?

(I'm checking in on all the StarterBugs that haven't been touched in over a month; it's totally fine if you just haven't had time but still want to keep it.)

@keith
Copy link
Contributor

keith commented Jul 5, 2020

It looks like this is fixed in Xcode 12 beta 1 / Swift 5.3?

@swift-ci
Copy link

Comment by Yogesh Anand Manghnani (JIRA)

Is this issue resolved? I am planning to take this up and get my hands dirty.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 4, 2022
@Overcot
Copy link

Overcot commented May 14, 2022

I checked this issue on swift-package-manager itself using command

swift test --filter WorkspaceTests.WorkspaceTests/testDownloadArchiveIndexFilesHappyPath --filter WorkspaceTests.WorkspaceTests/testDownloadArchiveIndexServerError --filter SPMBuildCoreTests --filter CommandsTests

and the result was at the end

Test Suite 'XCFrameworkMetadataTests' passed at 2022-05-14 22:25:05.473.
	 Executed 2 tests, with 0 failures (0 unexpected) in 0.001 (0.001) seconds
Test Suite 'SwiftPMPackageTests.xctest' passed at 2022-05-14 22:25:05.473.
	 Executed 109 tests, with 13 tests skipped and 0 failures (0 unexpected) in 171.187 (171.190) seconds
Test Suite 'Selected tests' passed at 2022-05-14 22:25:05.473.
	 Executed 109 tests, with 13 tests skipped and 0 failures (0 unexpected) in 171.187 (171.191) seconds

I guess maybe a point of improvement is to merge all intermediate test target execution results at the end of execution and not in middle

@tomerd
Copy link
Member

tomerd commented May 16, 2022

iiuc this output is from XCTest, @stmontgomery?

@stmontgomery
Copy link
Contributor

iiuc this output is from XCTest, @stmontgomery?

Correct, this comes from XCTest. Maybe this issue should be transferred there

@tomerd

This comment was marked as resolved.

@tomerd tomerd transferred this issue from apple/swift-package-manager May 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants