Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-1609] Support recording baseline for performance regression tests #386

Open
briancroom opened this issue May 24, 2016 · 0 comments
Open

Comments

@briancroom
Copy link
Collaborator

Previous ID SR-1609
Radar None
Original Reporter @briancroom
Type Bug
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee None
Priority Medium

md5: 89bb688c4da749aeb43653263685237d

Issue Description:

We added basic support for XCTest's performance testing APIs in https://bugs.swift.org/browse/SR-1355, however one crucial component of that is still missing, namely recording baseline results of performance tests and triggering test failures when significant regressions are detected relative to those baselines.

I see a few pieces to this:

  • Persisting baseline data to disk. Xcode uses a Plist format for this. It would probably make sense to use a compatible format here.

  • Comparing test results to a previously recorded baseline and triggering a failure if the performance has regressed. Xcode's supposedly triggers a failure if Average - BaselineAverage is greater than 0.1 seconds and is more than 10% higher than BaselineAverage itself.

  • Providing a mechanism (probably command-line flags?) instructing the runner where to find and create baseline data, and for causing the baseline data to be updated.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant