[SR-648] swift package manager should have the option to produce statically linked binaries #5432
Comments
|
Comment by Kostiantyn Koval (JIRA) can you please provide your build output, and your "Hello world" program. Are you running it on Linux or Mac OS ? Example: //Exe target
print("Hi Main")
//Lib target
func a() {
print("Hi Main")
}
//Exe target
swift build -vv
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swiftc --driver-mode=swift -I /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/lib/swift/pm -L /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/lib/swift/pm -lPackageDescription -target x86_64-apple-macosx10.10 /Users/konstantin/Work/spmExample/Simple/Package.swift
/usr/bin/xcrun --sdk macosx --show-sdk-path
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift-build-tool -v -f /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/llbuild.yaml
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swiftc -module-name Simple -incremental -emit-dependencies -emit-module -emit-module-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftmodule -output-file-map /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/output-file-map.json -c /Users/konstantin/Work/spmExample/Simple/main.swift -I /Users/konstantin/Work/spmExample/Simple/.build/debug -j8 -Onone -g -v -target x86_64-apple-macosx10.10 -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /usr/local/include -D SWIFT_PACKAGE
Apple Swift version 3.0-dev (LLVM f95d47afa7, Clang f66c5bb67b, Swift b745691a38)
Target: x86_64-apple-macosx10.10
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift -frontend -c -primary-file /Users/konstantin/Work/spmExample/Simple/main.swift -target x86_64-apple-macosx10.10 -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /Users/konstantin/Work/spmExample/Simple/.build/debug -I /usr/local/include -g -D SWIFT_PACKAGE -emit-module-doc-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/main~partial.swiftdoc -Onone -module-name Simple -emit-module-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/main~partial.swiftmodule -emit-dependencies-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/main.d -emit-reference-dependencies-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/main.swiftdeps -o /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/main.swift.o
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift -frontend -emit-module /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/main~partial.swiftmodule -parse-as-library -target x86_64-apple-macosx10.10 -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /Users/konstantin/Work/spmExample/Simple/.build/debug -I /usr/local/include -g -D SWIFT_PACKAGE -emit-module-doc-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftdoc -module-name Simple -o /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftmodule
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swiftc -o /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/main.swift.o -target x86_64-apple-macosx10.10 -Xlinker -all_load -g -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -L/usr/local/lib
//lib target
swift build -vv
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swiftc --driver-mode=swift -I /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/lib/swift/pm -L /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/lib/swift/pm -lPackageDescription -target x86_64-apple-macosx10.10 /Users/konstantin/Work/spmExample/Simple/Package.swift
/usr/bin/xcrun --sdk macosx --show-sdk-path
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift-build-tool -v -f /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/llbuild.yaml
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swiftc -module-name Simple -incremental -emit-dependencies -emit-module -emit-module-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftmodule -output-file-map /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/output-file-map.json -parse-as-library -c /Users/konstantin/Work/spmExample/Simple/A.swift -I /Users/konstantin/Work/spmExample/Simple/.build/debug -j8 -Onone -g -v -target x86_64-apple-macosx10.10 -enable-testing -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /usr/local/include -D SWIFT_PACKAGE
Apple Swift version 3.0-dev (LLVM f95d47afa7, Clang f66c5bb67b, Swift b745691a38)
Target: x86_64-apple-macosx10.10
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift -frontend -c -primary-file /Users/konstantin/Work/spmExample/Simple/A.swift -target x86_64-apple-macosx10.10 -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /Users/konstantin/Work/spmExample/Simple/.build/debug -I /usr/local/include -enable-testing -g -D SWIFT_PACKAGE -emit-module-doc-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/A~partial.swiftdoc -Onone -parse-as-library -module-name Simple -emit-module-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/A~partial.swiftmodule -emit-dependencies-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/A.d -emit-reference-dependencies-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/A.swiftdeps -o /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/A.swift.o
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift -frontend -emit-module /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/A~partial.swiftmodule -parse-as-library -target x86_64-apple-macosx10.10 -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /Users/konstantin/Work/spmExample/Simple/.build/debug -I /usr/local/include -enable-testing -g -D SWIFT_PACKAGE -emit-module-doc-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftdoc -module-name Simple -o /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftmodule
/bin/sh -c "rm -f '/Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.a'; ar cr '/Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.a' '/Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/A.swift.o'" |
|
Comment by Kostiantyn Koval (JIRA) Also I've verified that generated `Simple.a` static library don't have any dynamic loaded dependencies otool -L Simple.a
Archive : Simple.a
Simple.a(A.swift.o): |
|
Comment by Jay Buffington (JIRA) This is in Ubuntu 15.10 |
|
Comment by Kostiantyn Koval (JIRA) Here is the output I get for the MacOS. It includes 3 dynamic libraries. otool -L .build/debug/Simple
.build/debug/Simple:
/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
@rpath/libswiftCore.dylib (compatibility version 0.0.0, current version 0.0.0)Also the package manager doesn't control this behaviour. I think it's more related to the LLBD (the build system). |
|
Discussion of this on the lists suggests this isn't yet possible, so we need to open a bug on Swift itself so that it gains the option to link statically on Linux. |
|
This should be unblocked, it would be great to get this in the Swift 3.0 timeframe. |
|
I'm checking to see if this is working now (this issue hasn't been updated for a while now). If it works, great, but if not, it seems to me that we are out of time for Swift 3 for this. |
|
When I try `swift build -Xswiftc -static-stdlib` on Ubuntu 14.0.4, I get this: |
|
It isn't that clear to me where this comes from. SwiftPM doesn't seem to be passing it down. |
|
On macOS it works, so this might be a toolchain incompatibility with Ubuntu 14.0.4. I will try on Ubuntu 15.10 and file a JIRA if it's incompatible. |
|
I can reproduce this in a small test case, and I have filed SR-2280. |
|
I am removing the swift-3.0 label from this, since it is blocked by SR-2280. |
|
Should a static executable option be added to swiftc or the swift-build-tool? I think an `-emit-static-executable' or '-emit-executable -static' option added to swiftc might be the best approach rather than the package manager so it can be used directly I have a solution for this for Linux/ELF which uses a small stub to fix up the linker issues which I can turn into a PR static_stub.c #if defined(__LINUX__) && defined(__ELF__)
#error "This only works on Linux/ELF" // Needs testing on other ELF platforms
#else
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
#define __USE_GNU
#include <link.h>
// This forces resolving of these weak symbols but keeps them hidden
// externally
const void *unused1 __attribute__ ((unused, visibility("internal"))) = pthread_self;
const void *unused2 __attribute__ ((unused, visibility("internal"))) = pthread_key_create;
const void *unused3 __attribute__ ((unused, visibility("internal"))) = pthread_once;
// libdl emulation to load the protocol and metadata sections, these get fixed
// using the --defsym linker commands below
extern void *__swift2_protocol_conformances_start;
extern void *__swift2_type_metadata_start;
// Fake a handle if opening NULL (this binary) else abort
void *
dlopen(const char *filename, int flag)
{
if (filename != NULL) {
fprintf(stderr, "dlopen called with filename=%s\n", filename);
abort();
}
return (void *)1;
}
// Hardcoded to allow lookup of 2 known symbols, enough for now
void *
dlsym(void *handle, const char *symbol)
{
if (handle != (void *)1) {
fprintf(stderr, "dlsym(): bad handle: %p\n", handle);
abort();
}
if (!strcmp(symbol, ".swift2_protocol_conformances_start")) {
return &__swift2_protocol_conformances_start;
} else if (!strcmp(symbol, ".swift2_type_metadata_start")) {
return &__swift2_type_metadata_start;
} else {
return NULL;
}
}
int
dladdr(const void *addr, Dl_info *info)
{
// FIXME: Dummy implementation for now, returns `not found'
fprintf(stderr, "dladdr() required for %p\n", addr);
// info declared as nonnull in dlfcn.h
info->dli_fname = NULL;
info->dli_fbase = NULL;
info->dli_sname = NULL;
info->dli_saddr = NULL;
return 0;
}
int
dlclose(void *handle)
{
if (handle != (void *)1) {
fprintf(stderr, "dlclose(): bad handle: %p", handle);
}
return 0;
}
// Hardcoded to return an empty structure, the caller of this only
// cares about the filename anyway and can deal with it being NULL
int
dl_iterate_phdr(int (*callback) (struct dl_phdr_info *info,
size_t size, void *data), void *data)
{
// Dummy empty structure for dl_iterate_phdr
struct dl_phdr_info phdr_info = { .dlpi_addr = 0 };
return callback(&phdr_info, sizeof(struct dl_phdr_info), data);
}
// These commands will get extracted by `swift-autolink-extract'
// and passed to clang++
#define AUTOLINK_ENTRY(x) asm(".asciz \"" x "\"")
asm(".section .swift1_autolink_entries");
AUTOLINK_ENTRY("-static");
AUTOLINK_ENTRY("-Xlinker --defsym=__swift2_protocol_conformances_start=.swift2_protocol_conformances_start");
AUTOLINK_ENTRY("-Xlinker --defsym=__swift2_type_metadata_start=.swift2_type_metadata_start");
AUTOLINK_ENTRY("-licui18n");
AUTOLINK_ENTRY("-licuuc");
AUTOLINK_ENTRY("-licudata");
AUTOLINK_ENTRY("-lpthread");
#endif // linux && ELFI have a slightly different solution that patches swift/stdlib which reduces the size of the libdl emulation code and makes it simpler to understand which might be a better approach if its decided to add the option to swiftc rather than swiftpm |
|
Updated fix that doesnt use any libdl hacks but instead builds its own version of libicu: |
|
Is this done now? |
|
Let me clarify, is the |
|
It should be done for swiftc and libswiftCore however the other corelibs will need fixing to produce static libraries (that dont use libdl) |
|
Ive created a partial solution in apple/swift#9958 It gets the build command down to: swift build -Xswiftc -static-stdlib -Xlinker -lcurl -Xlinker -lxml2 -Xlinker -lbsd -Xlinker --allow-multiple-definition Some further work is required to eliminate the ' -Xlinker --allow-multiple-definition' option and a bit more will be needed to automatically generate the list of libs to link. |
|
Just a small update: Currently have 3 PRs to implement the solution in the previous comment: apple/swift#9958 Once that is done I will open a PR adding a `test_static` target to Foundation to create TestFoundationStatic binary as a test. Then need to remove the duplicate |
|
Thanks @spevans! |
|
@swift-ci create |
|
SwiftPM will link stdlib statically by default on macOS after #1246 |
|
Comment by Frank Langel (JIRA) Awesome endeavor. With this feature, Swift will adopt the killer feature that made GOlang so successful, the deployment of executables as ONE file and will strengthen its ability as a server side language. There hasn't been much progress in a while. Any ETA? |
|
Comment by Nader Rahimizad (JIRA) This would be a fantastic feature. Any ETA? |
|
Comment by Alexis Gallagher (JIRA) Also, are there any resources or links, besides Simon Evans’s comments in this thread in 2017, which describe what work would be needed to complete this? Just to provide a bit of context, every few months I need to write a command line tool for colleagues who are not developers. I get excited and think “maybe I should use Swift!” Then I think of the deployment downsides, and I end up writing it in python 2.7, restricting myself to python standard libraries, and it makes me sad. It would change everything if Swift let you just ship a single executable, which would work and keep working on a foreign system. I believe rust and go can do this, fwiw. |
|
We actually do have fully static executables working on macOS with SwiftPM, just not on Linux. IIUC, the difficulty on Linux comes to dealing with the other dependencies Swift and in particular Foundation bring in (libcurl, libatomic, libicu). I don't really have the context on which of these are hard to deal with, and which are not. I don't know why we can't just try and get static versions of each of the libraries built in, for example. It would be awesome if someone who understands the issue could at least comment on the current status. @spevans are you able to describe where we are currently? |
|
Comment by Alexis Gallagher (JIRA) Static executables are already working on macOS? That's great! So it seems like, if I am only shipping my command line executables to colleagues on Macs, then I can already use Swift as long as I rely on pure Swift code, Foundation (and presumably other macOS Frameworks), and Swift packages with a similar dependency profile. (I suppose it would need special handling of an external Swift package itself relied on a C library, but was not already configured to build it for static linking.) I'll wait for news on what needs doing for Linux. If there's some grunt work I can help with, I'd like to, but I'm no expert on linking. |
|
The issue with libicu on Linux is that it is built by Ubuntu with some parts compiled as a .so file so can only be used in a dynamic executable. The best solution would be to always build libicu as part of the Swift toolchain build. This would also allow building the latest version of ICU (or even Apple's fork) which would solve numerous other ICU issues on Linux. Foundation on Linux has the problem that libcurl relies on about 25 other libraries and some of these are only supplied as .so files. The solution here would be to build a version of libcurl as part of the swift-corelibs-foundation build that disables much of the extra unused functionality. This will then allow creating a `lib curl.a` file. |
|
alexis@alexisgallagher.com (JIRA User) yup! --static-swift-stdlib to SwiftPM. @spevans makes sense, thanks for the information. While those seem cumbersome and painstaking, they don't seem insurmountable I think we just need to find someone willing to do the work. I would be happy to help review/mentor this project, for any future readers of this bug. |
|
@ddunbar For the ICU part, it just requires agreement from Apple/core team to go ahead with building it as part of the toolchain, creating the repo and then deciding which version to use. I can update the ICU work I did for building the library to solve that part. |
|
Comment by Alexis Gallagher (JIRA)
Just to follow up on your note @ddunbar, I wanted to ask, is there anything that I myself could do that would help move this long? Based on Simon's and Ankit's comment above, it sounds as if the next steps are at least as much procedural as technical: 1. Apple/core team agrees to include a clone or fork of ICU, and build it as part of building Swift. 2. Similarly, Apple/core team agrees to include a clone or fork of a stripped-down version of libcurl, which contains only what is needed to support Foundation's networking requirements on Linux. So as regards item 2, there's probably some technical work to figure out how to build the stripped down libcurl on Ubuntu that would meet Foundation's needs. With a bit of help, I could probably figure that out, and would be glad to do so. But as regards the procedural work, who is the right person to explain and plan for these changes in how Swift is built? I doubt it's me, since it seems like a core team issue. Or is this a feature that realistically is at a stage where it first needs to be "pitched" for, because many folks won't think that the benefit (easily deployable executables for language users) is worth the cost (a more complicated language build system)? |
we... don't as of Swift 5 It's painful especially on Linux though. |
|
Comment by Gilles Magalhaes (JIRA) Would it make sense to disable the command line `--static-swift-stdlib` option in the `swift build` command for the time being? Statically linking binaries doesn't work right now on Linux. On macOS, it also doesn't make sense to have that option, since statically linking binaries is not possible anymore (if I understand correctly). I think it's confusing for Linux users that calling `swift build` with the `--static-swift-stdlib` option doesn't actually produce the desired result and does not produce a warning like on macOS. If disabling that option is not favourable, I would propose that we at least add a warning. As a side note, what would it take to get statically linked binaries working on Linux? I would be interested in working on this issue if someone could mentor me. |
|
Comment by Mickaël Rémond (JIRA) I agree it would be best to disable the option if it does not work. |
|
Hi, just wanted to mention that I've just been able to produce a binary with statically linked stdlibs with the latest 5.3 nightly docker image.
In other words: Using |
|
I've submitted #2921 which enables --static-swift-stdlib with SwiftPM |
|
I tried
After investigation, the link error was avoided with the following changes:
|
|
I think |
Since #3048 has been merged, |
Additional Detail from JIRA
md5: a97136565bbf5fd9b15515ab552f61b8
is blocked by:
is duplicated by:
relates to:
Issue Description:
When I do swift build for a simple hello world swift program it ldd tells me that it is linked against libstdc++.so.6, libswiftCore.so and other .so files. I would like to ask swift build to produce a single file that is not dynamically linked against any shared objects.
The text was updated successfully, but these errors were encountered: