This would be like lit's --show-tests, and we could use it to provide an analog of
--filter which would be really nice.

However, XCTest on OS X supports dynamic test case construction, which makes this non trivial and I am not sure if it can be made to work without changing the API. I will investigate though.

It looks like OS X XCTest can support this via a small shim which uses the XCTest APIs directly. Here is code which lists the tests in a bundle:

@import Foundation;
@import XCTest;

int main(int argc, char **argv) {
  @autoreleasepool {
      NSString *bundlePath = @(argv[1]);

      // Normalize the path.
      if (!bundlePath.absolutePath) {
          bundlePath = [NSFileManager.defaultManager.currentDirectoryPath stringByAppendingPathComponent:bundlePath];
      }
      bundlePath = [bundlePath stringByStandardizingPath];
      
      // Load the bundle from the command line.
      NSBundle *bundle = [NSBundle bundleWithPath:bundlePath];
      if (![bundle load]) {
          NSLog(@"unable to load bundle: %s", argv[1]);
      }
    
      // Get the test suite for the given bundle.
      printf("{ \"testSpecifiers\": [\n");
      XCTestSuite *suite = [XCTestSuite testSuiteForBundlePath:bundlePath];
      NSString *suitePrefix = [suite.name stringByDeletingPathExtension];
      NSCharacterSet *splitSet = [NSCharacterSet characterSetWithCharactersInString:@" ]:"];
      for (XCTestSuite *testCaseSuite in suite.tests) {
          for (XCTestCase *test in testCaseSuite.tests) {
              // Extract the test name.
              NSString *name = [test.name componentsSeparatedByCharactersInSet:splitSet][1];

              // Unmangle names for Swift test cases which throw.
              if ([name hasSuffix:@"AndReturnError"]) {
                  name = [name substringWithRange:NSMakeRange(0, name.length - 14)];
              }
                    
              printf("  \"%s/%s\",\n", [[test class] debugDescription].UTF8String, name.UTF8String);
          }
      }
      printf("  null] }\n");
  }
  
  return 0;
}

You can build it with:

clang -fmodules -o BundleTestFinder BundleTestFinder.m  -F $(xcode-select -p)/Platforms/MacOSX.platform/Developer/Library/Frameworks

and run it with something like:

DYLD_FRAMEWORK_PATH=$(xcode-select -p)/Platforms/MacOSX.platform/Developer/Library/Frameworks LitTestAdaptor/BundleTestFinder ../build/Ninja-ReleaseAssert/swiftpm-macosx-x86_64/debug/SwiftPMTests.xctest

We should see how this works with XCTest bundles which generate test cases automatically – that isn't really possible in Swift, but Obj-C can do it

To use that, we would presumably need to build that into a shim tool (or a special hidden subcommand) – it loads the test bundles into the calling process, so we wouldn't want to actually run that code directly in the same process as the actual SwiftPM logic.

I gave this tool a try on a couple of test bundles which do dynamic test case generation. It generally worked as expected, and I do think that this is the correct method to use for this purpose. Specifically, I ran the tool on test bundles that used these BDD-style test frameworks:

• Quick: Worked perfectly, except that the framework sometimes writes to stdout while the classes are being enumerated, so the tool should probably redirect stdout until it is ready to begin enumerating the tests (see below).

• Cedar: It didn't work here, because Cedar dynamically adds test cases to the test suite that XCTest passes to the -testSuiteWillStart: observation method. I don't think this is a case that the test lister should try to support.

Here is a modified version of @ddunbar's code that performs the stdout redirection:

@import Foundation;
@import XCTest;

id withRedirectedStdout(id (^block)(void)) {
    fflush(stdout);
    int oldFd = dup(1);
    int newFd = open("/dev/null", O_WRONLY);
    dup2(newFd, 1);
    close(newFd);

    id result = block();

    fflush(stdout);
    dup2(oldFd, 1);
    close(oldFd);

    return result;
}

int main(int argc, char **argv) {
    @autoreleasepool {
        NSString *bundlePath = @(argv[1]);

        // Normalize the path.
        if (!bundlePath.absolutePath) {
            bundlePath = [NSFileManager.defaultManager.currentDirectoryPath stringByAppendingPathComponent:bundlePath];
        }
        bundlePath = [bundlePath stringByStandardizingPath];

        XCTestSuite *suite = withRedirectedStdout(^{
            // Load the bundle from the command line.
            NSBundle *bundle = [NSBundle bundleWithPath:bundlePath];
            if (![bundle load]) {
                NSLog(@"unable to load bundle: %s", argv[1]);
            }

            // Get the test suite for the given bundle.
            return [XCTestSuite testSuiteForBundlePath:bundlePath];
        });

        printf("{ \"testSpecifiers\": [\n");
        NSString *suitePrefix = [suite.name stringByDeletingPathExtension];
        NSCharacterSet *splitSet = [NSCharacterSet characterSetWithCharactersInString:@" ]:"];
        for (XCTestSuite *testCaseSuite in suite.tests) {
            for (XCTestCase *test in testCaseSuite.tests) {
                // Extract the test name.
                NSString *name = [test.name componentsSeparatedByCharactersInSet:splitSet][1];

                // Unmangle names for Swift test cases which throw.
                if ([name hasSuffix:@"AndReturnError"]) {
                    name = [name substringWithRange:NSMakeRange(0, name.length - 14)];
                }

                printf("  \"%s/%s\",\n", [[test class] debugDescription].UTF8String, name.UTF8String);
            }
        }
        printf("  null] }\n");
    }
    
    return 0;
}

@ddunbar Is there a particular reason why you chose to use +[XCTestSuite testSuiteForBundlePath:] instead of +[XCTestSuite defaultTestSuite]? I believe the latter is closer to what Xcode uses when it runs tests, and xctool uses it when it does test listing as well.

@briancroom: Thanks for the investigation, I was also curious how well test listing would work with third-party frameworks. I assume the Quick prints to stdout when tests are marked "pending", correct? That's definitely something we can fix. We should create a GitHub issue to track it.

@modocache Yes, that's right. As it turns out, there is already a relevant issue and languoshong PR on Quick about pending tests:
Quick/Quick#490
Quick/Quick#491

Fixed by #397