Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SR-9551] Failure to run benchmarks with build-script #52000

Open
palimondo mannequin opened this issue Dec 20, 2018 · 6 comments
Open

[SR-9551] Failure to run benchmarks with build-script #52000

palimondo mannequin opened this issue Dec 20, 2018 · 6 comments
Labels
bug A deviation from expected or documented behavior. Also: expected but undesirable behavior. build-script Area → utils: The build script compiler The Swift compiler in itself

Comments

@palimondo
Copy link
Mannequin

palimondo mannequin commented Dec 20, 2018

Previous ID SR-9551
Radar None
Original Reporter @palimondo
Type Bug
Additional Detail from JIRA
Votes 0
Component/s Compiler
Labels Bug, BuildScript
Assignee None
Priority Medium

md5: 172d28eeb0a096aefb2802a72fbc6c5a

Issue Description:

Running benchmarks with build-script like this:
swift-source $ ./swift/utils/build-script -R -B --no-assertions
fails because the values for --independent-samples parameter are somehow not passed on here.

Here's the tail of the full console output:

[1/1] cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/...ipt /Users/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.p
FAILED: benchmark/CMakeFiles/check-swift-benchmark-macosx-x86_64
cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver run -o O --output-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --independent-samples  && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver run -o Onone --output-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --independent-samples  && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver compare --log-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --compare-script /Users/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.py
usage: Benchmark_Driver run [-h] [-f PATTERN] [-t TESTS] [-o OPT]
                            [-i INDEPENDENT_SAMPLES] [--output-dir OUTPUT_DIR]
                            [--swift-repo SWIFT_REPO]
                            [BENCHMARK [BENCHMARK ...]]
Benchmark_Driver run: error: argument -i/--independent-samples: expected one argument

ninja: build stopped: subcommand failed.

See also https://forums.swift.org/t/changes-to-how-build-script-runs-benchmarks/18730
@belkadan
Copy link
Contributor

@atrick, who do you think should be looking at this?

@atrick
Copy link
Member

atrick commented Dec 20, 2018

@palimondo I've never run the benchmarks using the build-script. Didn't know it was possible. Was `--independent-samples` added to the driver, or did someone change the build-script causing it to break?

If you know the responsible party can you ask them?

I attempted to debug the build-script last night for a different reason and can say for sure that either it's a trivial fix (which you could handle) or it will take me more time than I have right now. (I'm also strongly opposed to opaque layers of abstraction over software configuration).

Meanwhile, I'll file a radar to track it.

@atrick
Copy link
Member

atrick commented Dec 20, 2018

<rdar://problem/46874722> [SR-9551] Failure to run benchmarks with build-script

I also indicated in the radar that I'd be happy removing this option from the build-script.

@palimondo
Copy link
Mannequin Author

palimondo mannequin commented Dec 20, 2018

I'm aware that build-script isn't used internally by many Apple devs now. But it's the officially documented method, right there in the first paragraph.

The Benchmark_Driver parameter `--independent-samples` is not new, it was just renamed from the old and misleading --iterations few months ago.

On casual inspection of build-script, everything seems to be correct. There's also no apparent culprit to ask for a fix. I asked in the forums before filing this, but got dead silence... This mysterious issue was there once before: SR-4357

@atrick
Copy link
Member

atrick commented Dec 20, 2018

@palimondo Things like this usually get fixed by devs who have sufficient motivation to do it in their spare time. It sounds like you at least have the motivation.

@palimondo
Copy link
Mannequin Author

palimondo mannequin commented Dec 21, 2018

:-) I do have are a very particular set of skills...

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug A deviation from expected or documented behavior. Also: expected but undesirable behavior. build-script Area → utils: The build script compiler The Swift compiler in itself
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@belkadan @atrick