Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-875] Make swift-corelibs-xctest Functional tests regex matching more like FileCheck #404

Closed
modocache mannequin opened this issue Mar 3, 2016 · 3 comments
Closed

Comments

@modocache
Copy link
Mannequin

modocache mannequin commented Mar 3, 2016

Previous ID SR-875
Radar None
Original Reporter @modocache
Type Improvement
Status Closed
Resolution Won't Do
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement, StarterBug
Assignee @modocache
Priority Medium

md5: 36d197324a801db0dd6d5b0d341b6a3e

Issue Description:

swift-corelibs-xctest includes a functional test suite that uses the lit test runner and a Python program that matches output line by line. Here's an example.

Notice that the parentheses in the CHECK lines need to be escaped. This is because each line is interpreted as a regular expression. This makes writing CHECK lines very cumbersome and error-prone.

On the other hand, FileCheck, a matching program used by most of the LLVM and Swift test suites, does not interpret CHECK lines as regexes. Instead, parts of the line may be interpreted as a regex using a special pattern, such as {{.*}}.

The Python program used to match output in corelibs-xctest should use a similar mechanism: regexes should delineated, instead of having the entire line be interpreted as a regex. For example, the CHECK lines in the test linked above should become:

// CHECK: Test Case 'SingleFailingTestCase.test_fails' started.
// CHECK: {{.*}}/Tests/Functional/SingleFailingTestCase/main.swift:24: error: SingleFailingTestCase.test_fails : XCTAssertTrue failed - 
// CHECK: Test Case 'SingleFailingTestCase.test_fails' failed ({{\d+}}.{{\d+}} seconds).
// CHECK: Executed 1 test, with 1 failure (0 unexpected) in {{\d+}}.{{\d+}} ({{\d+}}.{{\d+}}) seconds
// CHECK: Total executed 1 test, with 1 failure (0 unexpected) in {{\d+}}.{{\d+}} ({{\d+}}.{{\d+}}) seconds

(Of course, even better would be to have corelibs-xctest use FileCheck directly, but that program is only available as part of a full LLVM build, and we don't want to introduce such a dependency.)

@swift-ci
Copy link

swift-ci commented Apr 1, 2016

Comment by Kyle Yoon (JIRA)

Hi @modocache, I'm seeing that a lot of the starter tasks for XCTest (SR-1046, SR-1045, SR-1048 and this one) require some knowledge of LLVM and Python. I'm very new to both but am willing to study them. Do you recommend picking up any of these stories for someone with not much experience in this area? If so, which one would be the easiest to go with?

@modocache
Copy link
Mannequin Author

modocache mannequin commented Apr 9, 2016

Hi Kyle, sorry for the late reply. I think this task involves knowledge of Python. If you don't know Python already, it should be fairly easy to learn, and this might be a good starter project!

That being said, based on the discussion in #94 I think we're going to move in another direction: using FileCheck directly, instead of using xctest_checker. So I think this task might be obsolete--I'll close it once mike-ferris-apple (JIRA User) weighs in.

@modocache
Copy link
Mannequin Author

modocache mannequin commented Jul 5, 2016

Closing this in favor of https://bugs.swift.org/browse/SR-1944 – instead of making xctest-checker more like FileCheck, we'll just use FileCheck. 🙂

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 9, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant