Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SR-15725] Downloading many (50) binary targets runs into HTTP timeouts #4359

Closed
swift-ci opened this issue Jan 13, 2022 · 2 comments
Closed

[SR-15725] Downloading many (50) binary targets runs into HTTP timeouts #4359

swift-ci opened this issue Jan 13, 2022 · 2 comments
Labels
bug

Comments

@swift-ci
Copy link
Contributor

Previous ID SR-15725
Radar None
Original Reporter joplunien (JIRA User)
Type Bug
Status Resolved
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s Package Manager
Labels Bug
Assignee joplunien (JIRA)
Priority Medium

md5: b0f34e2358f5a3afa8d76e5609aa7ac8

Issue Description:

We have a fairly large Package.swift with 50 binary dependencies:

grep binaryTarget Package.swift | wc -l
      50

In total the size of these 50 zip files is 400 MB. On a slower internet connection (16 MBit) the swift package resolve command errors out with some HTTP timeouts. It seems the main problem is that all 50 HTTP requests are executed at the same time:

https://github.com/apple/swift-package-manager/blob/main/Sources/Workspace/Workspace.swift#L2273-L2291

Applying following diff already resolves our problem:

diff --git a/Sources/Workspace/Workspace.swift b/Sources/Workspace/Workspace.swift
index 6c501bf5..08883ec8 100644
--- a/Sources/Workspace/Workspace.swift
+++ b/Sources/Workspace/Workspace.swift
@@ -2288,6 +2288,7 @@ extension Workspace {
             var request = HTTPClient.Request.download(url: artifact.url, headers: headers, fileSystem: self.fileSystem, destination: archivePath)
             request.options.authorizationProvider = self.authorizationProvider?.httpAuthorizationHeader(for:)
             request.options.validResponseCodes = [200]
+            request.options.retryStrategy = .exponentialBackoff(maxAttempts: 3, baseDelay: .milliseconds(50))
             self.httpClient.execute(
                 request,
                 progress: { bytesDownloaded, totalBytesToDownload in

However this is not the best solution. Requests still run into some timeout and are just retried. So network bandwith is being wasted on the client and on the server. What do you think about introducing some HTTP request queue so that only n requests are executed at the same time?
@swift-ci
Copy link
Contributor Author

Comment by Johannes Plunien (JIRA)

See also: #4017

@swift-ci
Copy link
Contributor Author

Comment by Johannes Plunien (JIRA)

Fixed: d41c85b

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 4, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@swift-ci