Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-14257] Can't add a package with a dynamic framework inside of a binary target. #4448

Open
swift-ci opened this issue Feb 19, 2021 · 5 comments
Labels

Comments

@swift-ci
Copy link
Contributor

Previous ID SR-14257
Radar rdar://problem/74616575
Original Reporter molind (JIRA User)
Type Bug

Attachment: Download

Environment

macOS 11.2.1 on Apple M1

Xcode 12.4 and Xcode 12.5 beta 2

Additional Detail from JIRA
Votes 2
Component/s Package Manager
Labels Bug
Assignee None
Priority Medium

md5: 16f97a03341b6799b8a5f3e7b7ce8e97

Issue Description:

Hello,

I'm testing swift package for my framework. And during that tests I'm getting some strange errors in Xcode 12.4 and Xcode 12.5 beta 2.

artifact of binary target 'GLMap' failed extraction: The operation couldn’t be completed. (TSCBasic.StringError error 1.)

I've checked Console.

default 10:26:57.489507+0100 kernel AMFI: '/private/var/folders/lf/vzq1qc7530b07bgzfv97spl80000gn/T/TemporaryDirectory.x1jJNI/glmapswift-manifest' has no CMS blob?default 10:26:57.489507+0100 kernel AMFI: '/private/var/folders/lf/vzq1qc7530b07bgzfv97spl80000gn/T/TemporaryDirectory.x1jJNI/glmapswift-manifest' has no CMS blob?default 10:26:57.489518+0100 kernel AMFI: '/private/var/folders/lf/vzq1qc7530b07bgzfv97spl80000gn/T/TemporaryDirectory.x1jJNI/glmapswift-manifest': Unrecoverable CT signature issue, bailing out.default 10:26:57.489520+0100 kernel AMFI: code signature validation failed.

I've checked binary frameworks used in swift package. There is no signature inside:

evgen@Evgens-MacBook-Air ios-arm64_armv7 % codesign --verify --verbose GLMap.framework/GLMap GLMap.framework/GLMap: code object is not signed at all In architecture: arm64

Swift package repository: https://github.com/GLMap/GLMapSwift branch spm.

I had an issue with cloning this repo with git-lfs enabled and git installed from brew. Don't know why. Removed git from brew and it worked using git from console tools.

@typesanitizer
Copy link

@swift-ci create

@1ec5
Copy link

1ec5 commented Apr 8, 2021

I often encounter this error and get around it by going to File ‣ Swift Packages ‣ Clear Package Caches and waiting while Xcode redownloads everything. If I’m not mistaken, this SPM error is triggered by this unzip invocation.

I wonder if the messages about code signature validation are a red herring, since the error is transient (though frustratingly common). I don’t know why this particular unzip invocation would sometimes require code signing and sometimes not.

@swift-ci
Copy link
Contributor Author

Comment by Cormac Relf (JIRA)

I believe this is because frameworks and by extension xcframeworks contain symlinks. Use `zip -y -r MyFramework.xcframework.zip MyFramework.xcframework`, emphasis on the -y flag to preserve symlinks that frameworks use for bundling multiple versions and referring to one of them as "Current".

If you create two zips, ` zip -y -r good.zip folder` and `zip -r bad.zip folder`, then bad.zip contains fully resolved files in every spot where the folder had a symlink in it, whereas good.zip has symlinks. I don't know what messes things up with the bad.zip way, but when you unzip an xcframework zipped into bad.zip, it spits out a ton of errors like this:

```
checkdir error: MyFramework.xcframework/macos-arm64_x86_64/MyFramework.framework/Modules exists but is not directory
unable to process MyFramework.xcframework/macos-arm64_x86_64/MyFramework.framework/Modules/MyFramework.swiftmodule/.
```

BOTH `unzip bad.zip` and `unzip good.zip` finish with this (other platforms than macOS do not use these symlinks apparently):

```
finishing deferred symbolic links:
MyFramework.xcframework/macos-arm64_x86_64/MyFramework.framework/Resources -> Versions/Current/Resources
MyFramework.xcframework/macos-arm64_x86_64/MyFramework.framework/Headers -> Versions/Current/Headers
MyFramework.xcframework/macos-arm64_x86_64/MyFramework.framework/Modules -> Versions/Current/Modules
MyFramework.xcframework/macos-arm64_x86_64/MyFramework.framework/Versions/Current -> A

```

This looks like a problem with macOS' /usr/bin/zip and its handling of zip-internal symlinks, since without -y they're meant to be gone and it appears bad.zip contains fully resolved files in symlinked folders, and ALSO symlinks for the folders themselves.

The flakiness probably comes from nondeterminism in zip's recursive add code. Sometimes it adds things to the archive in such a way that it doesn't break when it comes back out.

Anyway, upshot: `zip -y -r good.zip folder` not only works, but comes out smaller too.

@typesanitizer
Copy link

Is this still an issue with the latest Xcode beta?

@swift-ci
Copy link
Contributor Author

swift-ci commented Sep 3, 2021

Comment by Cormac Relf (JIRA)

@ theindigamer (JIRA User) Yes, 13 beta 5 still affected. I believe all versions of Xcode are affected, because they all use the same /usr/bin/unzip (unless Xcode bundles its own!). I think this is a widespread issue that's just going to need a bunch of binary framework publishers to fix their zip commands in their build scripts. Nothing much you can do about it beyond fixing /usr/bin/zip, so probably just close the issue.

I found a few issues scattered around GitHub where people had similar problems to share my findings, but they're all by a certain corporation that has a policy of locking issues after closing, so I guess they'll just miss out.

I have a repro: [repo](https://github.com/cormacrelf/CiteprocRsKit-Binary) with two versions:

  • `v0.2.0`, built with `zip -r`

  • `v0.3.0`, built with `zip -y -r`

You can verify for yourself that all Xcodes choke on the first one but the second is fine.


I discovered a second issue where you can also get unzip to choke by naming a repository like that one (consisting only of a Package.swift with a binaryTarget pointing to a zip URL) like so:

  • repo name: `Abc.xcframework` (pretty logical name for it)

  • package name: `Abc`

  • zip file name: `Abc.xcframework.zip`

  • folder inside the zip file name: `Abc.xcframework`

Presumably because of the folder structure used to extract this stuff. Same error as above, because it's all caused by `unzip`.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants