Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-15037] Add --quiet option to swift run #4395

Closed
BenchR267 opened this issue Aug 6, 2021 · 13 comments · Fixed by #5988
Closed

[SR-15037] Add --quiet option to swift run #4395

BenchR267 opened this issue Aug 6, 2021 · 13 comments · Fixed by #5988

Comments

@BenchR267
Copy link
Member

Previous ID SR-15037
Radar rdar://problem/81620352
Original Reporter @BenchR267
Type New Feature
Additional Detail from JIRA
Votes 1
Component/s Package Manager
Labels New Feature
Assignee None
Priority Medium

md5: c5f31e82419529323d967dcf872f390c

Issue Description:

`swift run` is often used for running Swift scripts. When running those scripts from other scripts, the output is in an expected format and should not contain any output from SwiftPM itself.
Currently, all output from SwiftPM seems to be printed on stderr so it's an option to nuke that in those situations (`2>>/dev/null`). But that doesn't work if the script itself is using stderr too.
I propose to add a new option to `swift run` which suppresses all output about building the script and only prints the output of the executable itself, `--quiet`.
Open for discussion is how build failures are surfaced to the user and I don't have a strong opinion on that. It could suppress that too making the command fail silently and only provide the exit status as indicator for failure or it could print those out instead since the executable will never run anyway.

@BenchR267
Copy link
Member Author

@swift-ci create

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 4, 2022
@hassila
Copy link

hassila commented Sep 20, 2022

I'd also like to see this for e.g. swift package for command plugins, I added a --quiet flag to my command, but I still get the extra output from SwiftPM that isn't helpful:

hassila@max (testpr)> swift package benchmark compare alpha beta --quiet --target Miscellaneous-Benchmark
Building for debugging...
Build complete! (0.13s)
Miscellaneous-Benchmark: currentBaseline > otherBaseline, true
hassila@max (testpr)> 

@clayellis
Copy link
Contributor

I don't have much to add other than my voice to say that I would also benefit from a --quiet flag.

@tomerd
Copy link
Member

tomerd commented Dec 21, 2022

@hassila @BenchR267 would you like to make a PR for this? SwiftPM has LoggingOptions which then sets the logLevel, we can make a new flag for quiet that will set the log level to error. we can also expose log-level flag and use that (ie without sugar)

@clayellis
Copy link
Contributor

clayellis commented Dec 21, 2022

@tomerd I went ahead and followed your suggestions and opened the above PR

@doozMen
Copy link

doozMen commented Oct 10, 2023

I notice although the quiet flag is there when running swift test --quiet it does not change the output to only include the error, which is confusing. Same for swift build --quiet. Is this because it is only implemented on swift run? And available via shared options that have no effect on build and test? If so I consider this a bug as I did not expect that from the docs, at least it should state that it only has effect on run no?

@neonichu
Copy link
Member

It should affect everything imho

@superarts
Copy link

I notice although the quiet flag is there when running swift test --quiet it does not change the output to only include the error, which is confusing. Same for swift build --quiet. Is this because it is only implemented on swift run? And available via shared options that have no effect on build and test? If so I consider this a bug as I did not expect that from the docs, at least it should state that it only has effect on run no?

I'm also curious about this. swift test --help gives me:

USAGE: swift test <options> <subcommand>

OPTIONS:

...

  -v, --verbose           Increase verbosity to include informational output
  --very-verbose, --vv    Increase verbosity to include debug output
  -q, --quiet             Decrease verbosity to only include error output.

However, passed test cases are printed out, too. I'm using Quick/Nimble.

@clayellis
Copy link
Contributor

@superarts @doozMen @neonichu what would you expect the output to be in the swift test case given this output:

Test Suite 'All tests' started at 2023-12-24 19:09:51.933.
Test Suite 'ExamplePackageTests.xctest' started at 2023-12-24 19:09:51.933.
Test Suite 'SomeTests' started at 2023-12-24 19:09:51.934.
Test Case '-[ExampleTests.SomeTests testFail1]' started.
/private/var/folders/zg/zpxfqpls1qddr0mb0zcwgt6c0000gn/T/Miscellaneous_TestPassFail.Eimm6C/Miscellaneous_TestPassFail/Tests/ExampleTests/Tests.swift:14: error: -[ExampleTests.SomeTests testFail1] : XCTAssertEqual failed: ("Hello, World!") is not equal to ("hello, failure")
Test Case '-[ExampleTests.SomeTests testFail1]' failed (0.867 seconds).
Test Case '-[ExampleTests.SomeTests testFail2]' started.
/private/var/folders/zg/zpxfqpls1qddr0mb0zcwgt6c0000gn/T/Miscellaneous_TestPassFail.Eimm6C/Miscellaneous_TestPassFail/Tests/ExampleTests/Tests.swift:18: error: -[ExampleTests.SomeTests testFail2] : XCTAssertEqual failed: ("false") is not equal to ("true")
Test Case '-[ExampleTests.SomeTests testFail2]' failed (0.001 seconds).
Test Case '-[ExampleTests.SomeTests testPass1]' started.
Test Case '-[ExampleTests.SomeTests testPass1]' passed (0.000 seconds).
Test Case '-[ExampleTests.SomeTests testPass2]' started.
Test Case '-[ExampleTests.SomeTests testPass2]' passed (0.000 seconds).
Test Suite 'SomeTests' failed at 2023-12-24 19:09:52.802.
	 Executed 4 tests, with 2 failures (0 unexpected) in 0.868 (0.868) seconds
Test Suite 'ExamplePackageTests.xctest' failed at 2023-12-24 19:09:52.802.
	 Executed 4 tests, with 2 failures (0 unexpected) in 0.868 (0.868) seconds
Test Suite 'All tests' failed at 2023-12-24 19:09:52.802.
	 Executed 4 tests, with 2 failures (0 unexpected) in 0.868 (0.869) seconds

The entire output?

The lines that contain error: - or failure:

/private/var/folders/zg/zpxfqpls1qddr0mb0zcwgt6c0000gn/T/Miscellaneous_TestPassFail.Eimm6C/Miscellaneous_TestPassFail/Tests/ExampleTests/Tests.swift:18: error: -[ExampleTests.SomeTests testFail2] : XCTAssertEqual failed: ("false") is not equal to ("true")
Test Case '-[ExampleTests.SomeTests testFail1]' failed (0.867 seconds).
/private/var/folders/zg/zpxfqpls1qddr0mb0zcwgt6c0000gn/T/Miscellaneous_TestPassFail.Eimm6C/Miscellaneous_TestPassFail/Tests/ExampleTests/Tests.swift:18: error: -[ExampleTests.SomeTests testFail2] : XCTAssertEqual failed: ("false") is not equal to ("true")
Test Case '-[ExampleTests.SomeTests testFail2]' failed (0.001 seconds).
Test Suite 'SomeTests' failed at 2023-12-24 19:09:52.802.
	 Executed 4 tests, with 2 failures (0 unexpected) in 0.868 (0.868) seconds
Test Suite 'ExamplePackageTests.xctest' failed at 2023-12-24 19:09:52.802.
	 Executed 4 tests, with 2 failures (0 unexpected) in 0.868 (0.868) seconds
Test Suite 'All tests' failed at 2023-12-24 19:09:52.802.
	 Executed 4 tests, with 2 failures (0 unexpected) in 0.868 (0.869) seconds

Just the failure lines?

Test Case '-[ExampleTests.SomeTests testFail1]' failed (0.867 seconds).
Test Case '-[ExampleTests.SomeTests testFail2]' failed (0.001 seconds).

Just the summary?

Test Suite 'SomeTests' failed at 2023-12-24 19:09:52.802.
	 Executed 4 tests, with 2 failures (0 unexpected) in 0.868 (0.868) seconds
Test Suite 'ExamplePackageTests.xctest' failed at 2023-12-24 19:09:52.802.
	 Executed 4 tests, with 2 failures (0 unexpected) in 0.868 (0.868) seconds
Test Suite 'All tests' failed at 2023-12-24 19:09:52.802.
	 Executed 4 tests, with 2 failures (0 unexpected) in 0.868 (0.869) seconds

Something else?

I would think that for simplicity's sake, the entire output should be kept.

@superarts
Copy link

Agreed. As a developer, I would say we usually don't care about any successful cases at all (unless metrics data like the total success numbers and coverages etc), but we do expect to see all the details about the failed cases, to help us debugging. And the fact that we are taking about the CLI command here indicates a lot of us solely depend on CLI without Xcode.

In this case, the purpose is a bit different with the traditional "--quite" use case, which usually indicates user just wants to see very brief summary. So some other arguments like "--failed-output-only" would make more sense here.

@clayellis
Copy link
Contributor

clayellis commented Dec 25, 2023

I've started work on resolving the concerns that were raised (I agree, --quiet does not work as advertised on swift test and swift build.) I've opened a draft PR #7216 which already has a fix for swift test. Will work on swift build shortly.

Merry Christmas, I guess 🎄🎅🏻😅

@clayellis
Copy link
Contributor

clayellis commented Dec 26, 2023

Okay I have a working solution for swift build --quiet over there as well. As of right now, there is a slight difference in raw output between a failed build without the quiet flag and one with:

Without:

�[2K
[0/1] Planning build�[2K
[1/1] Planning build�[2K
Building for debugging...
�[2K
[0/6] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/6] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/6] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/6] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/7] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/6] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/5] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/6] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/5] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/4] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/5] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/6] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/7] Write swift-version--58304C5D6DBC2206.txt�[2K
[1/7] Write swift-version--58304C5D6DBC2206.txt�[2K
error: emit-module command failed with exit code 1 (use -v to see invocation)
�[2K
[2/7] Emitting module Foo�[2K
�[1m/Users/clayellis/Developer/swift-package-manager/Fixtures/DependencyResolution/Internal/InternalExecutableAsDependency/Foo/main.swift:1:8: �[0m�[0;1;31merror: �[0m�[1mno such module 'Bar'
�[0mimport Bar
�[0;1;32m       ^
�[0m�[2K
[3/7] Compiling Foo Foo.swift�[2K
�[1m/Users/clayellis/Developer/swift-package-manager/Fixtures/DependencyResolution/Internal/InternalExecutableAsDependency/Foo/main.swift:1:8: �[0m�[0;1;31merror: �[0m�[1mno such module 'Bar'
�[0mimport Bar
�[0;1;32m       ^
�[0m�[2K
[4/7] Compiling Foo main.swift�[2K
�[1m/Users/clayellis/Developer/swift-package-manager/Fixtures/DependencyResolution/Internal/InternalExecutableAsDependency/Foo/main.swift:1:8: �[0m�[0;1;31merror: �[0m�[1mno such module 'Bar'
�[0mimport Bar
�[0;1;32m       ^
�[0m�[2K
[4/7] Compiling Foo main.swift�[2K
error: fatalError
note: module 'Bar' is the main module of an executable, and cannot be imported by tests and other targets
note: module 'Bar' is the main module of an executable, and cannot be imported by tests and other targets
note: module 'Bar' is the main module of an executable, and cannot be imported by tests and other targets

Program ended with exit code: 1

With:

Building for debugging...
[0/6] Write swift-version--58304C5D6DBC2206.txt
error: emit-module command failed with exit code 1 (use -v to see invocation)
[2/7] Emitting module Foo
/Users/clayellis/Developer/swift-package-manager/Fixtures/DependencyResolution/Internal/InternalExecutableAsDependency/Foo/main.swift:1:8: error: no such module 'Bar'
import Bar
       ^
[3/7] Compiling Foo main.swift
/Users/clayellis/Developer/swift-package-manager/Fixtures/DependencyResolution/Internal/InternalExecutableAsDependency/Foo/main.swift:1:8: error: no such module 'Bar'
import Bar
       ^
[4/7] Compiling Foo Foo.swift
/Users/clayellis/Developer/swift-package-manager/Fixtures/DependencyResolution/Internal/InternalExecutableAsDependency/Foo/main.swift:1:8: error: no such module 'Bar'
import Bar
       ^
error: fatalError
Program ended with exit code: 1

Working to close the gap between those two.

@clayellis
Copy link
Contributor

clayellis commented Dec 29, 2023

Okay, based on this comment:

output from background activities goes to stderr, this includes diagnostics and output from build operations, package resolution that take place as part of another action CLI commands that have user facing output, use stdout directly to emit the final result this means that the build output from "swift build" goes to stdout but the build output from "swift test" goes to stderr, while the tests output go to stdout.

I believe the progress animation- and terminal controller-specific output should be omitted when using the --quiet flag and a failure happens (resulting in stdout output.) If this is true, the last code block of output in my previous comment is correct and there isn't any further work to be done (besides possibly adding a few more unit tests for swift build --quiet.)

Please correct me if I'm wrong, @tomerd or anyone else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants