Based on the description, you are running intro the issue discussed in https://bugs.swift.org/browse/SR-12783. The short answer is, if you are planning to ship a dSYM to another machine, you need to make sure to compiler with -no-serialize-debugging-options. This will avoid hardcoding any search paths into the serialized Swift modules that are stored in the .dSYM bundle.

Looks like I hit reply too early. You already tried that.

I just double-checked the compiler sources and -no-serialize-debugging-options is the right option:

It gets lowered to SerializeOptionsForDebugging here:
https://github.com/apple/swift/blob/4e0fad869c440489053753d059c57a4ea9dd66e6/lib/Frontend/ArgsToFrontendOptionsConverter.cpp#L175

and that option controls whether the Clang search paths are serialized in the .swiftmodule or not here:

https://github.com/apple/swift/blob/4e0fad869c440489053753d059c57a4ea9dd66e6/lib/Serialization/Serialization.cpp#L911

Now evidently this isn't working as expected, so let's figure out why!

Comment by Peter Steinberger (JIRA)

Thanks for looking into this, Adrian! I did verify that my xcconfig settings work, if I add a typo to the flag, I get compile errors. Can I send you anything else to help?

> Should it work like that? Is the dSYM bound to the machine it was built?

With -no-serialize-debugging-options it is supposed to work, however, since no Clang search paths are serialized, you may need to tell LLDB where to find the Clang header dependencies on the other machine manually. This can be done by setting the target.swift-extra-clang-flags option. Starting with Swift 5.2, LLDB is generally capable of also importing the Clang dependencies straight from the debug info, so even if you don't do that, debugging should still work. Because this is currently only a fallback mechanism, you will still see clang compiler errors about the missing headers, but debugging should still work.

The goal for LLDB is to make this code path the default rather than a fallback, which will allow us to make -no-serialize-debugging-options the default. The Swift 5.3 branch is still missing a few features (most notably apinotes support) that are necessary to complete this transition.

> I cannot find any binary size differences with -no-serialize-debugging-options; what does this change?

I'll investigate. You are supposed to see a difference in the binary .swiftmodule file, and by extension in the .dSYM (which contains a copy of all Swift modules).

> How does lldb find dSYM files? Same name+folder as the framework? Why does this fail in /tmp?

LLDB looks

1. next to the executable/dylib
2. through any custom .dSYM location mechanism ( https://lldb.llvm.org/use/symbols.html )
3. through Spotlight, using the UUID of the executable/dylib. Note that Spotlight doesn't index /tmp.

Thank you so much for filing this report! I was able to identify a bug in dsymutil that would misdirect LLDB to the .swiftmodule files on disk (the ones on the machine the .dSYM was built with), and disregard the ones included in the .dSYM itself. That could explain part of what we're seeing.

> Can I send you anything else to help?

Yes. I was unable to reproduce your report that -no-serialize-debugging-options has no effect. I did the following experiment:

$ touch /tmp/t.swift && swiftc -g /tmp/t.swift -o /tmp/a.out
$ touch /tmp/t.swift && swiftc -Xfrontend -no-serialize-debugging-options -g /tmp/t.swift -o /tmp/b.out
$ strings /tmp/a.out.dSYM/Contents/Resources/DWARF/a.out > /tmp/1
$ strings /tmp/b.out.dSYM/Contents/Resources/DWARF/b.out > /tmp/2
$ opendiff /tmp/1 /tmp/2

I can see that the Clang option block (for example -working-directory) is missing in b.out. So I wonder if your setup is somehow different from my experiment, or if perhaps you were diffing the object files instead of the .swiftmodules or .dSYMs when you reached the conclusion that the option doesn't do anything.

There is also a second thing I would curious about: In response to https://bugs.swift.org/browse/SR-12783, I changed LLDB to surface the underlying Clang errors instead of just saying Couldn't IRGen expression, no additional error. I understand that this may be a lot to ask, but if you reproduce the issue with a recent downloadable toolchain — can you see the underlying error, and what is it?

Comment by Franz Busch (JIRA)

@adrian-prantl I was just trying to debug this in our application as well since we are running into this problem constantly. We use Carthage and share the artefacts with ROME. That means one dev is building all the artefacts and then uploading them to an S3 bucket where everybody pulls them from. As far as I understood from the above we should be using -no-serialize-debugging-options to make this work.

What I tried now was to just build all deps locally again and then try to debug but I am still running into this IRGen issue. So I pulled the latest toolchain from June 4th and tried to debug again. Now I get a more expressive error but don't know how to properly interpret this:

warning: Swift error in fallback scratch context: error: module 'Swift' was created for incompatible target x86_64-apple-ios13.0-simulator: /Users/franzbusch/Library/Developer/Xcode/DerivedData/ModuleCache.noindex/Swift-2XSJ1TSBVFM44.swiftmodule

note: This error message is displayed only once. If the error displayed above is due to conflicting search paths to Clang modules in different images of the debugged executable, this can slow down debugging of Swift code significantly, since a fresh Swift context has to be created every time a conflict is encountered.

Can't construct shared Swift state for this process after repeated attempts.
Giving up.  Fatal errors:
error: module 'Swift' was created for incompatible target x86_64-apple-ios13.0-simulator: /Users/franzbusch/Library/Developer/Xcode/DerivedData/ModuleCache.noindex/Swift-2XSJ1TSBVFM44.swiftmodule

One other interesting thing is that this is happening a lot if our static libs and their associated test bundle but in our main target we are able to po properly. However, we have to set a breakpoint in didFinishLaunching that does po application.
Is there anything more we can provide you around this IRGen issue to help you debug it, we have been stuck with this for months now.

Peter, I'm sorry. I just realized that you had the answer to my second question already in your fantastic blog post:

warning: Swift error in fallback scratch context: error: module 'Swift' was created for incompatible target x86_64-apple-ios13.0-simulator: /Users/steipete/Library/Developer/Xcode/DerivedData/ModuleCache.noindex/Swift-3K8REJ00QGV2U.swiftmodule

That is very valuable information. This means that this is not failing because of Clang search path options, which explains, why -no-serialize-debugging-options does nothing to fix the problem for you!

The error message itself points at something else going wrong:

1. The Swift stdlib module is not supposed to be built from a .swiftinterface (in which case it would end up in the module cache) but it's supposed to get loaded from Xcode's prebuilt module cache.

2. I also think the error diagnostic is likely wrong, since I'm assuming that x86_64-apple-ios13.0-simulator is actually the correct triple for your target, right?

I'll see if I can reproduce this.

The dsymutil issue I mentioned is fixed here: https://reviews.llvm.org/D81205

One more thing I noticed in https://gist.github.com/steipete/515646a9840c91a61b73d1ab9f255bb3:

SwiftASTContext("PSPDFKit")::DeserializeAllCompilerFlags() – Found 1 AST file data entries.
Unable to load Swift AST for module "PSPDFKit" from library "/Users/steipete/Library/Developer/Xcode/DerivedData/SwiftExample-gkpyxmqpselkitgjpzrvuzwcnruj/Build/Products/Debug-iphonesimulator/SwiftExample.app/Frameworks/PSPDFKit.framework/PSPDFKit".
The module file format is too old to be used by this version of the debugger.

At the moment, you can not yet load a dSYM that contains a .swiftmodule produced by an older Swift compiler into an LLDB that was linked against a newer Swift compiler. LLDB may be able to find some of the types used in the public interface of PSPDFKit.framework in other dSYMs, and in the PSDPDFKit .swiftinterface but it will not be able to peek at the internals of the library, since most of the Swift types will be missing. That's just not supported (but being worked on).

There is one more thing that could play into this: Does the problem also reproduce when you bump the deployment target to iOS 13?

Comment by Peter Steinberger (JIRA)

Adrian: Thanks so much for debugging this and for fixing the issue in dsymutil!

Apologies for the late reply. I've been running into a bunch of issues (https://twitter.com/steipete/status/1270756342295334914) trying to build our project with the Swift Trunk toolchain, but Slava Pestov [mentioned|https://twitter.com/slava_pestov/status/1271150466404155399] that a fix for Catalyst will land in a few days (most other issues I was able to work around or modify build settings)

Thanks also for explaining how lldb performs the dSYM lookup - this explains the behavior I've seen and I updated my blog post with it. In there I did a bin diff of the binaries and not the dSYM, which explains why I didn't find any differences in size with/without the flag - I looked at the wrong spot.

I'll wait for trunk to be fixed and then build a new version to test. Curiously we dropped iOS 11 with our upcoming 9.4 release and also enabled library evolution, and with the draft I can't reproduce the lldb issue anymore. Given how much trouble this made to reliably reproduce, I need to try this in a fresh VM again - will do both after our release next week.

That's great, by then I may actually have found the time to fix the issue I'm suspecting regarding the deployment target. I'll ping the Jira ticket once that fix has landed.

The deployment target issues are fixed (see SR-12932). Does this still reproduce with the bugfixes from SR-12932?

(Unfortunately Xcode 12 beta 2 shipped before the last fix went in. You may see a "cannot load Swift stdlib" error there).

Comment by keith siilats (JIRA)

I ran into this when trying to attach the Firebase binaries with carthage in Xcode 11.5 and 11.6. I think its only the performance one that does this. I tried the lldb log file but don't see anything obvious there. I also read your blog post but I dont see and Firebase related dsym files in the Carthage Build IOS folder, although there are some facebook ones that have older dates. Could anyone try to replicate with Cartfile containing:

binary "https://dl.google.com/dl/firebase/ios/carthage/FirebaseAnalyticsBinary.json"

binary "https://dl.google.com/dl/firebase/ios/carthage/FirebaseCrashlyticsBinary.json"

#binary "https://dl.google.com/dl/firebase/ios/carthage/FirebasePerformanceBinary.json"

binary "https://dl.google.com/dl/firebase/ios/carthage/FirebaseProtobufBinary.json"

Comment by Peter Steinberger (JIRA)

We're still working on getting everything built with Xcode 12b3; there's another lldb bug where it doesn’t work when the iOS deployment target is set to anything below iOS 13. FB8128033, also reported on https://developer.apple.com/forums/thread/651375.

There's also a crasher in Apple's security helper on the DTK that currently breaks our build script (FB8127506), but that's something we can work around.

I hope that by b4 FB8128033 is resolved and I can report back if this fix worked.

Unfortunately there is very little information in FB8128033 that would help diagnose what the root cause is. Based on the symptoms, it could be the bug that was fixed in #1548 (not yet in beta 3) but it's difficult to tell with neither a log nor a reproducer. [It is that bug if the types log under SwiftASTContextForExpressions::LogConfiguration shows an iOS target triple, where is should be using an ios-simulator target triple]. I understand that sharing a reproducer can be difficult. Perhaps you can try with a downloadable 5.3 toolchain https://swift.org/download/#snapshots from Jul 27+? Otherwise I need more information to work with.

All the issues I'm aware of in this area should be resolved in beta 4.

Comment by Peter Steinberger (JIRA)

Many thanks for looking into this issue!

I can no longer reproduce the issues here as of Xcode 12b4 - debugging works as expected, no matter if the dSYM is available or not.

I'm happy to hear that we finally got to the bottom of this. Thanks a lot for you patience and the excellent bug reports!