@swift-ci create

Could you provide more information about what goes wrong on Gitlab CI? Specifically, what do you mean by 'llvm-cov finds no coverage data'? Is it that `llvm-cov report` prints out expected results in the "Filename" column, but that all the files have 0 coverage? Does this contradict what's inside the .profdata file? You can check by running `llvm-profdata show --all-functions --counts $the_profdata_file`.

There's nothing Haswell-specific about llvm's code coverage implementation, AFAIK. I wonder if something went wrong while collecting the profile data.

@vedantk Sure, sorry for the lack of details. I'm not 100% it's haswell specific, but it's the only (that I've found) difference.

Where I say "I get code coverage report" I mean, this is the output I receive:

Filename                                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ChannelHandlers/RedisByteDecoder.swift              5                 0   100.00%           3                 0   100.00%          10                 0   100.00%
ChannelHandlers/RedisCommandHandler.swift          15                 5    66.67%           8                 3    62.50%          45                11    75.56%
ChannelHandlers/RedisMessageEncoder.swift           3                 1    66.67%           3                 1    66.67%          13                 6    53.85%
Commands/BasicCommands.swift                       28                 4    85.71%          16                 2    87.50%          99                 7    92.93%
Commands/HashCommands.swift                        38                 4    89.47%          29                 1    96.55%         156                 1    99.36%
Commands/ListCommands.swift                        56                 8    85.71%          48                 5    89.58%         217                11    94.93%
Commands/SetCommands.swift                         46                12    73.91%          30                 4    86.67%         147                 4    97.28%
Commands/SortedSetCommands.swift                  172                19    88.95%         105                 6    94.29%         555                18    96.76%
Commands/StringCommands.swift                      23                 2    91.30%          21                 1    95.24%         100                 1    99.00%
Extensions/StandardLibrary.swift                   10                 2    80.00%           6                 1    83.33%          21                 1    95.24%
Extensions/SwiftNIO.swift                           9                 1    88.89%           7                 0   100.00%          38                 1    97.37%
RESP/RESPTranslator.swift                          69                 7    89.86%          10                 2    80.00%         172                10    94.19%
RESP/RESPValue.swift                               39                11    71.79%          14                 3    78.57%          69                17    75.36%
RESP/RESPValueConvertible.swift                    52                19    63.46%          15                 3    80.00%          99                22    77.78%
RedisClient.swift                                   2                 0   100.00%           2                 0   100.00%           7                 0   100.00%
RedisConnection.swift                              72                23    68.06%          47                10    78.72%         228                31    86.40%
RedisErrors.swift                                  12                 4    66.67%           6                 1    83.33%          23                 3    86.96%
RedisKey.swift                                     15                 9    40.00%          12                 6    50.00%          38                20    47.37%
RedisMetrics.swift                                  9                 2    77.78%           9                 2    77.78%          23                 2    91.30%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                             675               133    80.30%         391                51    86.96%        2060               166    91.94% 

versus, on GitLab CI this is the output I receive:

 Filename                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 TOTAL                               0                 0         -           0                 0         -           0                 0         - 

On all systems I also run ls -l and receive the following (same on all environments, working or not):

Profdata:  -rw-r--r--. 1 root root 575608 Feb 8 19:51 .build/x86_64-unknown-linux/debug/codecov/default.profdata
Test binary:  -rwxr-xr-x. 1 root root 16309424 Feb 8 19:51 .build/x86_64-unknown-linux/debug/redi-stackPackageTests.xctest 

I wrote a StackOverflow question about this, that admittedly has more clear details: https://stackoverflow.com/questions/60130969/llvm-cov-fails-to-generate-report-when-run-on-cloud-gitlab-ci

Interesting, the filenames are supposed to be encoded within the `.xctest` binary. The fact that they don't appear at all in one configuration is interesting. What's the output of running:

`llvm-cov report -dump-collected-paths $BINARY_PATH -instr-profile="$PROF_DATA_PATH"`

Do all the filenames match `IGNORE_FILENAME_REGEX`?

If not, perhaps the compiler fails to encode them (although, that's never been an issue before). Could you attach two versions of `.build/x86_64-unknown-linux/debug/redi-stackPackageTests.xctest`, one from a working configuration, and one from the non-working configuration?

@vedantk I've attached a zip with the xctest binary from a local docker instance (hosted on macOS Catalina), along with its codecov folder.

The files are found in the zip as well, generated from the GitLab CI job that ran the same docker instance.

While gathering these artifacts, I also ran the command you specified as -dump rather than -dump-collected-paths as the latter doesn't seem to be a supported argument.

llvm-cov report -dump $BINARY_PATH -instr-profile="$PROF_DATA_PATH" -ignore-filename-regex="$IGNORE_FILENAME_REGEX" 

The output from the local docker instance that is correctly outputting the code coverage is attached to the ticket.

The output of the same command on GitLab CI:

 Filename                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover575 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------576 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------577 TOTAL                               0                 0         -           0                 0         -           0                 0         - 

I also tried with LLVM-8 & LLVM-9 with the same results on GitLab CI

It seems like something is wrong with your llvm-cov tool. Locally I'm able to inspect the .xctest binary:

```
$ ~/src/builds/llvm-project-master-RA/bin/llvm-cov report /Users/vsk/Downloads/SR-12142_XCTest_Binaries/SR-12142_LocalDocker_Swift-5.1.4_Ubuntu/redi-stackPackageTests.xctest -instr-profile /Users/vsk/Downloads/SR-12142_XCTest_Binaries/SR-12142_LocalDocker_Swift-5.1.4_Ubuntu/codecov/default.profdata -ignore-filename-regex='(.build|TestUtils|Tests)'
Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover

ChannelHandlers/RedisByteDecoder.swift 5 0 100.00% 3 0 100.00% 10 0 100.00%
ChannelHandlers/RedisCommandHandler.swift 15 5 66.67% 8 3 62.50% 45 11 75.56%
ChannelHandlers/RedisMessageEncoder.swift 3 1 66.67% 3 1 66.67% 13 6 53.85%
Commands/BasicCommands.swift 28 4 85.71% 16 2 87.50% 99 7 92.93%
Commands/HashCommands.swift 38 4 89.47% 29 1 96.55% 156 1 99.36%
Commands/ListCommands.swift 56 8 85.71% 48 5 89.58% 217 11 94.93%
Commands/SetCommands.swift 46 12 73.91% 30 4 86.67% 147 4 97.28%
Commands/SortedSetCommands.swift 172 19 88.95% 105 6 94.29% 555 18 96.76%
Commands/StringCommands.swift 23 2 91.30% 21 1 95.24% 100 1 99.00%
Extensions/StandardLibrary.swift 10 2 80.00% 6 1 83.33% 21 1 95.24%
Extensions/SwiftNIO.swift 9 1 88.89% 7 0 100.00% 38 1 97.37%
RESP/RESPTranslator.swift 69 7 89.86% 10 2 80.00% 172 10 94.19%
RESP/RESPValue.swift 39 11 71.79% 14 3 78.57% 69 17 75.36%
RESP/RESPValueConvertible.swift 52 19 63.46% 15 3 80.00% 99 22 77.78%
RedisClient.swift 2 0 100.00% 2 0 100.00% 7 0 100.00%
RedisConnection.swift 72 19 73.61% 47 7 85.11% 228 25 89.04%
RedisErrors.swift 12 4 66.67% 6 1 83.33% 23 3 86.96%
RedisKey.swift 15 9 40.00% 12 6 50.00% 38 20 47.37%
RedisMetrics.swift 9 2 77.78% 9 2 77.78% 23 2 91.30%

TOTAL 675 129 80.89% 391 48 87.72% 2060 160 92.23%
```

I'm not able to reproduce the issue with llvm-cov from llvm/master, or from swift 5.2.

Does removing the -ignore-filename-regex flag change the output? What is `llvm-cov show --version`?

Removing -ignore-filename-regex does not change any output (aside from adding the parts that were previously filtered).

Did you check the binary in the SR-12142_GitLabCIDocker_Swift-5.1.4_Ubuntu folder in the zip? That's the one that is failing, not the SR-12142_LocalDocker_Swift-5.1.4_Ubuntu

GitLab CI fails to show a report for Swift 5.1, Swift 5.2, and master every time I've tested

Sorry for not including both commands. On my machine, `llvm-cov report /Users/vsk/Downloads/SR-12142_XCTest_Binaries/SR-12142_GitLabCIDocker_Swift-5.1.4_Ubuntu/redi-stackPackageTests.xctest -instr-profile /Users/vsk/Downloads/SR-12142_XCTest_Binaries/SR-12142_GitLabCIDocker_Swift-5.1.4_Ubuntu/codecov/default.profdata` also produces a complete report.

So where does that leave us for trying to find a root cause? This is the official Docker image, with the llvm-cov that ships with the Swift tool chain.

The same Docker image is producing working test binaries on any platform, but llvm-cov from the same image is not producing a report when hosted on GitLab CI

Apologies, I should have been clearer. I'll need to set up a swift:5.1 docker image on a GitLab CI instance and step through what the llvm-cov binary is doing to determine the root cause (rdar://problem/59302413 tracks this). As things stand, I'm not sure that updating the llvm-cov binary would fix the issue, as the same version of the binary apparently behaves correctly on other machines.

At the moment I'm at a loss for an explanation of the output you see. FWIW, we do run integration tests for llvm-cov on a variety of different OS/hardware combinations to qualify each swift toolchain, and we've never detected an issue like this. The numbers 575, 576, and 577 show up in your output without any accompanying context/coverage data, and this seems very strange to me.

@vedantk Ah, thanks for the clarification.

This might require deeper help with the GitLab team, then. I wrote this bug against their services: https://gitlab.com/gitlab-org/gitlab/issues/202793

The reason for this, is that this only occurs when using their cloud Docker runner, which will be difficult for us to try and debug in a black-box way. If I trigger a build using GitLab CI on my own personal Docker runner - this bug does not reproduce. So it seems isolated directly to their cloud Docker CI hosts.

For anyone who runs into this we had this problem as well. I don't believe it is a haswell CPU specific issue. The problem is that gitlab CI checks out your code into a directory that starts "/build/..." so the regex `(.build|TestUtils|Tests)` doesn't actually ignore the "/build/.build" directory. The solution is to instead fix the regex:

(\.build|TestUtils|Tests)

Cheers!

@Austinpayne And that works for you? Oh my gosh

Yup! Very good find by another member of our team.

I have confirmed this was an issue related to the Regex pattern used for filtering file names.