Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-15140] Binary target SSH URLs #4390

Closed
swift-ci opened this issue Aug 31, 2021 · 6 comments
Closed

[SR-15140] Binary target SSH URLs #4390

swift-ci opened this issue Aug 31, 2021 · 6 comments

Comments

@swift-ci
Copy link
Contributor

Previous ID SR-15140
Radar rdar://problem/82683725
Original Reporter Lazarenko (JIRA User)
Type New Feature
Status Closed
Resolution Invalid
Additional Detail from JIRA
Votes 0
Component/s Package Manager
Labels New Feature
Assignee None
Priority Medium

md5: c70478a85334abfdc9dcf8629b6b2540

Issue Description:

Currently .binaryTarget supports only HTTPS-based URLs.
This doesn't work with private repositories.
Specifying an SSH url like so (with or without "ssh://" prefix):

.binaryTarget(
    name: "skunkworks",
    url: "ssh://git@github.com:acme/myprivaterepo/releases/download/skunkworks.xcframework.zip",
),

Produces an error:

invalid URL scheme for binary target 'skunkworks'; valid schemes are: 'https'

This validation is hardcoded here:
https://github.com/apple/swift-package-manager/blob/main/Sources/PackageLoading/ManifestLoader.swift#L393

Since packages themselves can be fetched from SSH, it would be nice that the related binaries could be downloaded via the same channel.

@typesanitizer
Copy link

@swift-ci create

@neonichu
Copy link
Member

neonichu commented Sep 3, 2021

Not sure we want to support this as you can already use the path-based version of binary targets and add them to the actual package's repo. What would be the incentive to have two repos?

@swift-ci
Copy link
Contributor Author

swift-ci commented Sep 6, 2021

Comment by Daniel Lazarenko (JIRA)

@neonichu

Our use case is having large multi-platform binaries (e.g. OpenSSL, WebRTC etc.), especially debug builds with debug symbols. We have multiple versions of those builds: v1, v2, v3... v100 - that quickly accumulates to gigabytes. If we put them into a git repo, SPM has to download it in its entirety (all 100 versions are inside the git clone directory) even though we just need a single version.

If we want to limit the cloned git repo size we need to store the large binaries outside of git. One standard choice is to use GitHub releases/packages storage. We can make a "release", attach a binary file there, and point SPM to "https://github.com/acme/myprivaterepo/releases/download/skunkworks.xcframework.zip".

This works, but it has issues for us:

  • https has no authentication with SPM by default, so the repository has to become public;

  • if used without authentication GitHub sometimes limits the number of downloads, and produce errors (due to API limits);

Having SSH support solves both issues, and also reuses the same authentication mechanism as the main source code repo, which makes it easy to use: set up SSH key auth once and get the project with all the binary deps via Xcode (not having to pay for what we don't need, because it only downloads binaries that are needed).

@neonichu
Copy link
Member

neonichu commented Sep 7, 2021

Ah, so GH releases are available using SSH auth? I didn't realize that, I thought this would be another repository.

@swift-ci
Copy link
Contributor Author

swift-ci commented Sep 8, 2021

Comment by Daniel Lazarenko (JIRA)

I'm sorry. I've checked, and GH doesn't actually support SSH for accessing releases.
It seems to only be served via HTTPS and supports 2 ways to authenticate: token-based, and basic auth.
(see also: https://stackoverflow.com/questions/20396329/how-to-download-github-release-from-private-repo-using-command-line )

What we'd really want is to be able to use SPM binaries with private repos and GitHub "releases" or GitLab "job artifacts" or such.

Since it is not related to SSH, I'm closing the issue.

If you know a workaround for us (to do auth over HTTPS with SPM), please point to some solution.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 4, 2022
@battlmonstr
Copy link

HTTP auth for fetching private binaries could be done via ~/.netrc:
https://stackoverflow.com/questions/69834361/downloading-binary-only-swift-packages-that-require-authentication

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants