Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-1355] [XCTest] Support XCTest performance testing API #389

Closed
ddunbar opened this issue Apr 29, 2016 · 5 comments
Closed

[SR-1355] [XCTest] Support XCTest performance testing API #389

ddunbar opened this issue Apr 29, 2016 · 5 comments
Assignees

Comments

@ddunbar
Copy link
Member

ddunbar commented Apr 29, 2016

Previous ID SR-1355
Radar None
Original Reporter @ddunbar
Type New Feature
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 1
Component/s XCTest
Labels New Feature
Assignee @briancroom
Priority Medium

md5: 14030175159ae9057e04dccb414023e0

relates to:

  • SR-1354 [SwiftPM] Improve support for performance tests

Issue Description:

swift-corelibs-xctest should support the performance testing APIs.

Related to:
https://bugs.swift.org/browse/SR-1354

@modocache
Copy link
Mannequin

modocache mannequin commented Apr 29, 2016

I believe the following methods make up the XCTest performance testing API:

class XCTestCase : XCTest {
  /// The names of the performance metrics to measure when invoking `XCTestCase.measure()`.
  /// Returns `XCTPerformanceMetric_WallClockTime` by default. Subclasses can override this to
  /// change the behavior of `XCTestCase.measure()`.
  class func defaultPerformanceMetrics() -> [String]

  /// Call from a test method to measure resources (`XCTestCase.defaultPerformanceMetrics()`)
  /// used by the block in the current process.
  ///     func testPerformanceOfMyFunction() {
  ///         measure {
  ///             // Do that thing you want to measure.
  ///             myFunction()
  ///         }
  ///     }
  /// - Parameter block: A closure whose performance to measure.
  func measure(_ block: () -> Void)

  /// Call from a test method to measure resources (XCTPerformanceMetrics) used by the
  /// block in the current process. Each metric will be measured across calls to the block.
  /// The number of times the block will be called is undefined and may change in the
  /// future. For one example of why, as long as the requested performance metrics do
  /// not interfere with each other the API will measure all metrics across the same
  /// calls to the block. If the performance metrics may interfere the API will measure
  /// them separately.
  /// 
  ///     func testMyFunction2_WallClockTime() {
  ///         measureMetrics(self.dynamicType.defaultPerformanceMetrics(), automaticallyStartMeasuring: false) {
  ///             // Do setup work that needs to be done for every iteration
  ///             // but you don't want to measure before the call to `XCTestCase.startMeasuring()`.
  ///             setupSomething()
  ///             startMeasuring()
  ///             // Do that thing you want to measure.
  ///             myFunction()
  ///             stopMeasuring()
  ///             // Do teardown work that needs to be done for every iteration
  ///             // but you don't want to measure after the call to `XCTestCase.stopMeasuring()`.
  ///             teardownSomething()
  ///         }
  ///     }
  ///
  /// Caveats:
  /// • If true was passed for `automaticallyStartMeasuring` and `XCTestCase.startMeasuring()`
  ///   is called anyway, the test will fail.
  /// • If false was passed for `automaticallyStartMeasuring` then `XCTestCase.startMeasuring()`
  ///   must be called once and only once before the end of the block or the test will fail.
  /// • If `XCTestCase.stopMeasuring()` is called multiple times during the block the test will fail.
  ///
  /// - Parameter metrics: An array of NSStrings (XCTPerformanceMetrics) to measure. Providing an unrecognized string is a test failure.
  /// - Parameter automaticallyStartMeasuring: If false, XCTestCase will not take any measurements until `XCTesrCase.startMeasuring()` is called.
  /// - Parameter block: A closure whose performance to measure.
  func measureMetrics(_ metrics: [String], automaticallyStartMeasuring automaticallyStartMeasuring: Bool, for block: () -> Void)

  /// Call this from within a measure block to set the beginning of the critical section.
  /// Measurement of metrics will start at this point.
  func startMeasuring()

  /// Call this from within a measure block to set the ending of the critical section.
  /// Measurement of metrics will stop at this point.
  func stopMeasuring()
}
/// Records wall clock time in seconds between startMeasuring/stopMeasuring.
let XCTPerformanceMetric_WallClockTime: String

The documentation above is based on the latest Apple headers.

@briancroom
Copy link
Collaborator

PR: #109

@modocache
Copy link
Mannequin

modocache mannequin commented May 22, 2016

This was pretty much finished in #109 – thanks, @briancroom! Please open JIRA issues for anything not finished in the PR and worth tracking. 🙂

@ddunbar
Copy link
Member Author

ddunbar commented May 22, 2016

Awesome, thanks!!

@briancroom
Copy link
Collaborator

I've created https://bugs.swift.org/browse/SR-1609 as a follow-up task to this.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 9, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants