[SR-730] Flag to statically link Swift standard library Created: 13 Feb 2016  Updated: 8 Oct 2016  Resolved: 9 Jun 2016

Status: Closed
Project: Swift
Component/s: Compiler

Type: Bug Priority: Medium
Reporter: Drew Crawford Assignee: Drew Crawford
Resolution: Done Votes: 6
Labels: Driver
Environment:

swift-DEVELOPMENT-SNAPSHOT-2016-02-08-a


Attachments: HTML File test    
Issue Links:
Relates
relates to SR-648 swift package manager should have the... Open
relates to SR-2280 swiftc -static-stdlib option fails on... Resolved
relates to SR-727 (emscripten support) Compile Swift co... Resolved

 Description   

If I build an executable with swiftc:

$ swiftc test.swift

Then I find out the Swift standard library was dynamically linked:

$ otool -L test
@rpath/libswiftCore.dylib (compatibility version 0.0.0, current version 0.0.0)

However, the binary distributions also have a static version that I didn't use:

/Library/Developer/Toolchains/swift-latest.xctoolchain//usr/lib/swift_static/macosx/libswiftCore.a

I want to pass e.g. -static-stdlib to swiftc to compile a program that is statically linked with the Swift standard library.



 Comments   
Comment by Drew Crawford [ 14 Feb 2016 ]

I attempted to go diving to find the correct flags, but have not worked it out just yet.

swiftc -o foo foo.swift /usr/local/lib/swift_static/linux/libswiftCore.a /usr/local/lib/swift_static/linux/libswiftGlibc.a -licuuc -licui18n -lbsd

"works", but still generates a dynamically-linked executable.

Diving deeper, on Linux x64, somebody is passing -lswiftCore to clang++:

/usr/bin/clang++ /tmp/foo-4c5e3f.o /usr/local/lib/swift_static/linux/libswiftCore.a /usr/local/lib/swift_static/linux/libswiftGlibc.a -licuuc -licui18n -lbsd -L /usr/local/lib/swift/linux --target=x86_64-unknown-linux-gnu -Xlinker -rpath -Xlinker /usr/local/lib/swift/linux -lswiftCore @/tmp/libswiftCore-0309d2.autolink -Xlinker -T /usr/local/lib/swift/linux/x86_64/swift.ld -o foo

But I can't figure out who did that and where I can disable it.

Comment by Drew Crawford [ 14 Feb 2016 ]

On OSX, the flags I'm looking for are

/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/lib/swift_static/macosx/libswiftFoundation.a /Library/Developer/Toolchains/swift-latest.xctoolchain/usr/lib/swift_static/macosx/libswiftObjectiveC.a /Library/Developer/Toolchains/swift-latest.xctoolchain/usr/lib/swift_static/macosx/libswiftCore.a -lc++ -framework Foundation

Which works ✅

It seems that OSX uses ld for linking, which (for reasons unknown to me) do not try to dynamically link libswiftCore.

However, Linux uses clang++ as its linker, and attempts to link libswiftCore.so no matter what I do. So I still can't figure out how to do this on Linux.

Comment by Drew Crawford [ 14 Feb 2016 ]

cc: Jordan Rose you're the culprit here

In ToolChains.cpp appears the code

// Always add the stdlib
1175	  Arguments.push_back("-lswiftCore");

We should not "always" add -lswiftCore because then the stdlib cannot be statically linked.

Comment by Jordan Rose [ 15 Feb 2016 ]

Um. We should certainly not require people to explicitly specify -lswiftCore normally, though. What do you recommend?

Comment by Drew Crawford [ 15 Feb 2016 ]

Of course. What I would recommend would be one or both of:

1. -no-stdlib, which would not use -lswiftCore and allow me to cobble together the behavior I want
2. -static-stdlib, which would just go ahead and link the standard library/runtime statically

For background, the standard library breaks frequently (e.g.,each time the last few snapshots) right now on Linux. I currently have to coordinate binary releases around language snapshots (user needs to run same snapshot as binary was built with) and I don't have good visibility into when those come out to plan my releases around it.

One or both of these would let me (with a little effort) create static binaries, so that I'm not chasing the wind every few days.

Comment by Drew Crawford [ 27 Feb 2016 ]

Just a friendly bump on this. I don't see that a solution has been blessed by core.

cc: Max Howell also expressed an interest in this bug.

Comment by Drew Crawford [ 4 Mar 2016 ]

Friendly bump.

At the moment I have to recompile all binaries with each new snapshot; that's quite an effort.

Comment by Geordie Jay [ 4 Mar 2016 ]

Just so I understand- you were able to build a binary on OS X compiled from swift sources that is entirely statically linked? This would be big news for emscripten support.

Is it really just commenting out Arguments.push_back("- lswiftCore") and running build-script with --extra-swift-flags --build-static-stdlib=1? Would you might putting together a short rundown on what changes you had to make to support this?

Comment by Drew Crawford [ 4 Mar 2016 ]

Sure, let me tell The Story So Far™.

You can actually build a statically-linked binary on OSX right now. This has always worked, because Xcode does it by default for OSX executables. What you must do is simply

swiftc -L path/to/swift_static -lc++ -framework Foundation -Xlinker -force_load_swift_libs

I discovered this by simply looking through Xcode build logs.

On Linux, however, it doesn't work, for starters, because Linux linker doesn't have -force_load_swift_libs. I believe the correct incantation is

swiftc -licuuc -licu18n -lbsd /path/to/swift_static/a.a /path/to/swift_static/b.a /path/to/swift_static/c.a ...

However because swiftc forcibly injects -lswiftCore on Linux this results in a library that is both statically and dynamically linked. Which defeats the purpose.

I was hoping to hear back from Jordan Rose or perhaps even Max Howell regarding how to proceed on the -lswiftCore issue (we need an option to disable it I guess?), but I don't yet know what is acceptable on that question.

Comment by Max Howell [ 4 Mar 2016 ]

We should have this feature and then open a bug for the swift Linux driver to fix it.

I want to propose this goes into some other file than Package.swift.

My rationale is that the Package.swift describes modules and products, but not the installation or system linkage of those products.

The enduser should have complete control over the final binary composition of their products, and not be at the mercy of their dependencies.

Comment by Drew Crawford [ 4 Mar 2016 ]

This bug is the swift driver bug.

I'm not especially concerned with how this gets exposed from SwiftPM other than it should, but to me that is kind of a side issue. Right now we can't even statically link a Swift binary with a makefile.

Comment by Max Howell [ 4 Mar 2016 ]

My bad, got mixed up since there is another bug open on SwiftPM for this.

Comment by Drew Crawford [ 13 Mar 2016 ]

So I took a pass at this, you can see the WIP at https://github.com/drewcrawford/swift/tree/static-stdlib

That tree introduces a new flag, -static-stdlib, that in theory statically links the stdlib. This is Linux-only, I haven't yet gotten to OSX.

The immediate problem is that something odd is happening with the binaries that get built. I try to build this simple hello world program:

print("Hello world")

When linked dynamically against libswiftCore.so, it works fine. However, when linked statically against libswiftCore.a, the hello world program prints this mysterious thing:

String(_core: Swift._StringCore(_baseAddress: Swift.OpaquePointer(_rawValue: (Opaque Value)), _countAndFlags: Swift.UInt(_value: (Opaque Value)), _owner: Swift.Optional<Swift.AnyObject>.none))

It looks like the internal representation of a string, but I'm not sure why this happens. Can someone ID this print and figure out why I am printing it?

Here's a verbose build, and these invocations can reproduce the problem without any stuff in my tree (which is merely driver stuff):

'/swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swiftc' -v  -static-stdlib test2.swift
Swift version 3.0-dev (LLVM 699a786c15, Clang 77080f2c03, Swift 21c9372d82)
Target: x86_64-unknown-linux-gnu
/swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift -frontend -c -primary-file test2.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -color-diagnostics -module-name test2 -o /tmp/test2-9cbb9a.o
/swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift-autolink-extract /tmp/test2-9cbb9a.o -o /tmp/test2-49baa1.autolink
/usr/bin/clang++ -ldl -lpthread -lbsd -licui18n -licuuc /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/linux/x86_64/swift_begin.o /tmp/test2-9cbb9a.o -L /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift_static/linux --target=x86_64-unknown-linux-gnu -Xlinker -rpath -Xlinker /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift_static/linux -lswiftCore @/tmp/test2-49baa1.autolink /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/linux/x86_64/swift_end.o -o test2
Comment by Geordie Jay [ 14 Mar 2016 ]

Great stuff for working on this!

Off the top of my head, this is the same thing that was happening before either
a) the swift.ld linker script, or
b) the runtime protocol conformance mapping
... was incorporated into the Android port of Swift. I forget which, my recollection is leaning me towards the runtime. I think it was something to do with the type introspection.

Comment by Geordie Jay [ 14 Mar 2016 ]

I'm not at my computer so I can't edit my comment above.. Try declaring an enum or conforming a custom Struct or Class to a protocol, if the program crashes on startup it's the runtime.

The swift.ld linker script was changed (by an apple employee from memory) to an inline assembly version as far as I understand, I lost track of that somewhere along the way. I understand the script basically performed a recursive loop to find the "end-of-chain" memory space for things like printing values (instead of just printing info about the Struct wrapper, as you're seeing here). So this contradicts what I said above, I now actually think it's more likely to be swift.ld, assuming it really is one of the two issues. In any case it's worth checking whether the linker script (in whatever current form it has) is being invoked in your statically linked branch

Comment by Drew Crawford [ 14 Mar 2016 ]

Protocol conformance and enums work fine. Maybe it's the linker script?

Unfortunately, I am a complete "project internals" noob. My attempt to understand what you are talking about was to grep the tree for swift.ld. It returned one item in a changelog, but no code or script.

Could you talk a little bit about where I could find this thing and how I could determine if it causes the problem?

Comment by Wojtek Czekalski [ 15 Mar 2016 ]

Drew Crawford swift.ld has been discarded.

I've been trying to get rid of dylibs from the statically linked binary on OS X.

By running the command Drew posted with -v I was able to reduce it so that it doesn't link dylibs.

swift -frontend -c -primary-file main.swift -target x86_64-apple-macosx10.9 -color-diagnostics -module-name main -o /var/folders/.../main-4193f8.o

ld  /var/folders/.../main-4193f8.o -force_load /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/arc/libarclite_macosx.a -force_load_swift_libs -L /path/to/toolchain/usr/lib/swift_static/macosx -arch x86_64 -macosx_version_min 10.9.0 -o main -static

results in:
ld: library not found for -lobjc for architecture x86_64

The part I don't understand is where -lobjc flag is actually set.

Any suggestions are appreciated!

Comment by Geordie Jay [ 16 Mar 2016 ]

Wojtek Czekalski thanks for the link. So the two points that came to my mind were actually one in the same, the runtime reading the protocol conformance table. From what I understand William Dillon has fixed this avoid using the linker script altogether, and instead writes the conformances into the final binaries (nice work!).

Maybe William Dillon could comment here quickly, because I'm pretty sure we saw this same issue in early versions of the armv7 port too? If not, a 'no idea' would suffice Thanks

Comment by William Dillon [ 16 Mar 2016 ]

Hi.

Yep, I've seen this a time or two. When you link your static lib, you have to make sure that all the objects are preceeded by swift_begin.o and followed by swift_end.o. These add the metadata markers into the resulting binary so that Runtime can find the type metadata.

When the conformances are missing, the overrides of 'var description: String' can't be located, and the generic description is used. It's a pretty reliable troubleshooting tool.

Comment by Geordie Jay [ 16 Mar 2016 ]

William Dillon thanks! Edit: Any idea why protocol conformances and enums would work, but

print("A string")

wouldn't? My memory of the early stages of SwiftAndroid is that we either had all or nothing

Drew Crawford So it seems that somewhere along the line swift_begin / end is being skipped. I am not familiar with the changes you've made here but hopefully that gives you an idea of how to proceed.

Cheers

Comment by William Dillon [ 16 Mar 2016 ]

that part, I don't know. IIRC, if you do .description in print() you can force it to work.

To check whether the conformance markers are in there, use objdump:

wdillon@tegra-ubuntu:~$ objdump -t example |grep \.swift._
0000a008 g       .swift2_protocol_conformances	00000000              .protected .swift2_protocol_conformances_start
0000a010 g       .swift2_type_metadata	00000000              .protected .swift2_type_metadata_start
0000a008 g       .swift3_assocty	00000000              .protected .swift3_assocty_section
0000a008 g       .swift3_fieldmd	00000000              .protected .swift3_fieldmd_section
0000a008 g       .swift3_reflstr	00000000              .protected .swift3_reflstr_section
0000a008 g       .swift3_typeref	00000000              .protected .swift3_typeref_section
0000a010 g       .swift2_protocol_conformances	00000000              .protected .swift2_protocol_conformances_end
0000a018 g       .swift2_type_metadata	00000000              .protected .swift2_type_metadata_end

(meta: Why does "preformatted" never work??)

You want to see a start and end for both conformances and metadata.

Comment by Drew Crawford [ 17 Mar 2016 ]

Thanks for the leads everyone! I am definitely open to the possibility that I have screwed up swift_begin.o and swift_end.o but they do appear in plausible places in my invocation (at least to someone like me who has no idea what he's doing!)

/usr/bin/clang++ -ldl -lpthread -lbsd -licui18n -licuuc /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/linux/x86_64/swift_begin.o /tmp/test2-9cbb9a.o -L /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift_static/linux --target=x86_64-unknown-linux-gnu -Xlinker -rpath -Xlinker /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift_static/linux -lswiftCore @/tmp/test2-49baa1.autolink /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/linux/x86_64/swift_end.o -o test2

Can someone who understands the swift_begin behavior eyeball that and tell me if it looks wrong and what might be more correct?

I wanted to address one other clue from William Dillon:

IIRC, if you do .description in print() you can force it to work.

This is actually not true in my case: print("hello world!".description) returns the same internal string nonsense as print("hello world!"). I don't know if that makes a difference in the diagnosis.

Comment by William Dillon [ 17 Mar 2016 ]

Hmm, yah, they do seem to be in the right places. Also, your behavior with print is different than what I was seeing. To put the metadata issue to rest, would you mind pasting the output of your objdump?

Comment by Drew Crawford [ 17 Mar 2016 ]

Sure!

objdump -t test2 | grep \.swift._
00000000006215f0 l    d  .swift1_autolink_entries	0000000000000000              .swift1_autolink_entries
0000000000886a78 l    d  .swift2_protocol_conformances	0000000000000000              .swift2_protocol_conformances
00000000008899e0 l    d  .swift2_type_metadata	0000000000000000              .swift2_type_metadata
0000000000886a80 l     O .swift2_protocol_conformances	0000000000002f60              l_protocol_conformances
00000000008899e8 l     O .swift2_type_metadata	0000000000000288              l_type_metadata_table
0000000000889c70 l     O .swift2_type_metadata	0000000000000010              l_type_metadata_table
0000000000889c80 g       .swift2_type_metadata	0000000000000000              _edata
00000000008899e0 g       .swift2_type_metadata	0000000000000000              .protected .swift2_type_metadata_start
0000000000886a78 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_assocty_section
00000000008899e0 g       .swift2_protocol_conformances	0000000000000000              .protected .swift2_protocol_conformances_end
0000000000886a78 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_fieldmd_section
0000000000886a78 g     O .swift2_protocol_conformances	0000000000000000              .hidden __TMC_END__
0000000000886a78 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_reflstr_section
0000000000886a78 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_typeref_section
0000000000889c80 g       .swift2_type_metadata	0000000000000000              .protected .swift2_type_metadata_end
0000000000886a78 g       .swift2_protocol_conformances	0000000000000000              .protected .swift2_protocol_conformances_start
Comment by Drew Crawford [ 17 Mar 2016 ]

For comparison, here is the objdump for a dynamically-linked (working) program:

objdump -t test2 | grep \.swift._
0000000000400f30 l    d  .swift1_autolink_entries	0000000000000000              .swift1_autolink_entries
0000000000601378 l    d  .swift2_protocol_conformances	0000000000000000              .swift2_protocol_conformances
0000000000601380 l    d  .swift2_type_metadata	0000000000000000              .swift2_type_metadata
0000000000601388 g       .swift2_type_metadata	0000000000000000              _edata
0000000000601380 g       .swift2_type_metadata	0000000000000000              .protected .swift2_type_metadata_start
0000000000601378 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_assocty_section
0000000000601380 g       .swift2_protocol_conformances	0000000000000000              .protected .swift2_protocol_conformances_end
0000000000601378 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_fieldmd_section
0000000000601378 g     O .swift2_protocol_conformances	0000000000000000              .hidden __TMC_END__
0000000000601378 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_reflstr_section
0000000000601378 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_typeref_section
0000000000601388 g       .swift2_type_metadata	0000000000000000              .protected .swift2_type_metadata_end
0000000000601378 g       .swift2_protocol_conformances	0000000000000000              .protected .swift2_protocol_conformances_start
Comment by William Dillon [ 17 Mar 2016 ]

Very strange. There is a clue, but I'm not sure how to interpret it... I notice that the order of the sections is wrong in both examples. On my system, they come up in order (start before end), but the address of them indicates that they're ordered correctly. This could be a red herring, but I think it's strange none the less.

Comment by Drew Crawford [ 17 Mar 2016 ]

Wojtek Czekalski Can you post the exact test program you're using?

Comment by Drew Crawford [ 17 Mar 2016 ]

William Dillon Objdump from a program built with swift-DEVELOPMENT-SNAPSHOT-2016-03-01-a (e.g. definitely not something I broke):

objdump -t test | grep \.swift._
0000000000400f08 l    d  .swift1_autolink_entries	0000000000000000              .swift1_autolink_entries
0000000000601350 l    d  .swift2_protocol_conformances	0000000000000000              .swift2_protocol_conformances
0000000000601358 l    d  .swift2_type_metadata	0000000000000000              .swift2_type_metadata
0000000000601360 g       .swift2_type_metadata	0000000000000000              _edata
0000000000601358 g       .swift2_type_metadata	0000000000000000              .protected .swift2_type_metadata_start
0000000000601350 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_assocty_section
0000000000601358 g       .swift2_protocol_conformances	0000000000000000              .protected .swift2_protocol_conformances_end
0000000000601350 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_fieldmd_section
0000000000601350 g     O .swift2_protocol_conformances	0000000000000000              .hidden __TMC_END__
0000000000601350 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_reflstr_section
0000000000601350 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_typeref_section
0000000000601360 g       .swift2_type_metadata	0000000000000000              .protected .swift2_type_metadata_end
0000000000601350 g       .swift2_protocol_conformances	0000000000000000              .protected .swift2_protocol_conformances_start

Let's see if we can put this minor discrepancy to rest. I wonder if maybe this discrepancy is due to some difference in objdump? Here is the actual binary in question: test. Can you objdump and compare the output and see if your output is inverted as well? Can you also confirm the swift version where you're seeing the non-inverted behavior?

This is probably a dead end, but it's the only lead I have.

Comment by Drew Crawford [ 17 Mar 2016 ]

Update: I found some mystery libraries libsection_magic_begin.a and libsection_magic_end.a which "seem" similar to swift_begin.o and swift_end.o, perhaps for the static case. However, no futzing around with those files seem to have any effect on the print behavior

Comment by William Dillon [ 17 Mar 2016 ]

libsection_magic_begin.a and libsection_magic_end.a should only exist during compilation of the standard library. They are identical (or should be) to the swift_begin.o and swift_end.o files. We had to make those weird object files to deal with CMake's agressive reordering of .o's during linking the stdlib.

Comment by Drew Crawford [ 17 Mar 2016 ]

libsection_magic_begin.a etc. are actually distributed with Swift; if you download the snapshot tarball from swift.org they are in there.

Regardless, I will ignore them if they aren't relevant, as slimming the install is not a goal of this bug

Unfortunately I am out of leads here.

Comment by William Dillon [ 17 Mar 2016 ]

Interesting, it's not distributed in the Linux buildbot tarballs.

I guess one thing to note is it sounds like you're using MacOS, and I don't have (perhaps ironically) as much experience with open source swift in MacOS.

Comment by Drew Crawford [ 17 Mar 2016 ]

It definitely is in the Linux buildbot tarballs. 100%. Go download https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-03-01-a/swift-DEVELOPMENT-SNAPSHOT-2016-03-01-a-ubuntu15.10.tar.gz and check. It is in /usr/lib/swift_static/linux.

I actually run both Linux/OSX pretty heavily, my immediate interest is trying to statically link on Linux, as I have already sketched out a working solution for OSX (although there may be more than meets the eye, Wojtek Czekalski suggests the OSX solution doesn't work for him.)

Comment by Wojtek Czekalski [ 17 Mar 2016 ]

Drew Crawford The OS X solution does work, however it also links to dylibs.

The program I am trying to compile is simple print("hello world"). When I compile it using the command, it does link statically to libraries but it also links dynamically to the following (otool -L output):

	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1256.14.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1256.1.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

Well, I'm no expert in the build process (I got engaged in this because would love to dive deeper into it) but it looks like things are indeed linked dynamically. Please correct me if I'm wrong.

Comment by William Dillon [ 17 Mar 2016 ]

Weird. I didn't know about swift_static. Well, good luck!

Comment by Drew Crawford [ 17 Mar 2016 ]

Cezary Wojcik You are correct but I think that's expected behavior, or at least outside the scope of the present bug. If you aren't linking libswiftCore.dylib you have dynamically linked the Swift standard library.

Your otool output shows you are dynamically linking CoreFoundation, Foundation, libobjc and libSystem, all of which ship with OSX and should be present on any OSX install, and your binary should be portable to any El Cap system (certainly report if this isn't true). We should probably support the static linking of corelibs Foundation, but I don't think corelibs is even supported dynamically on OSX yet. And IMO static Foundation should be its own bug.

Comment by Drew Crawford [ 17 Mar 2016 ]

Sadly I am totally out of ideas on why print is printing String(_core: Swift._StringCore(_baseAddress: Swift.OpaquePointer(_rawValue: (Opaque Value)), _countAndFlags: Swift.UInt(_value: (Opaque Value)), _owner: Swift.Optional<Swift.AnyObject>.none)) instead of a string. I am now desperate enough to ping the folks who wrote print to see if they have any idea why this might occur: Dmitri Gribenko Dave Abrahams

Comment by Dmitri Gribenko [ 17 Mar 2016 ]

Likely print() is not working because conditional casts to protocols don't work because protocol conformance tables can't be found.

Comment by Wojtek Czekalski [ 17 Mar 2016 ]

Drew Crawford You are correct. Thanks for clarifying. Looks like I misunderstood the issue (which is btw clearly stated by the title of it). Yeah, it's definitely out of scope.

Comment by Drew Crawford [ 19 Mar 2016 ]

I'm still at a dead-end here. For folks following along at home:

  • swift-dev post, no replies
  • I am pretty confident, based on playing around with old snapshots, that linux static linking has never worked. Asking around suggests I'm the first one to try it, and the old snapshots fail similarly to the current failure (although there are minor differences in behavior with older snapshots).
  • I've been not-very-subtly begging folks who "might know" off-list, and I've gathered some great suggestions but all of them are now dead ends.
  • I dunno how to go any further.

Comment by Drew Crawford [ 20 Mar 2016 ]

William Dillon Here's an oddball idea. I am pretty sure the section of size 2f60 above is the one we want. This is because it's the largest (plausibly the stdlib) and there's a table of that size in libswiftCore.a.

I notice, however, that this entry appears definitely outside the start/end markers (this is besides the observation of my markers being reversed). Could an entry appearing outside those markers be related?

Comment by Drew Crawford [ 20 Mar 2016 ]

Actually, I think I'm just misreading it. When I pipe the output to sort, start/end are in order and the large table is between the markers again. I think objdump is just unsorted, which creates the confusion.

00000000006215f0 l    d  .swift1_autolink_entries	0000000000000000              .swift1_autolink_entries
0000000000886a78 g       .swift2_protocol_conformances	0000000000000000              .protected .swift2_protocol_conformances_start
0000000000886a78 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_assocty_section
0000000000886a78 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_fieldmd_section
0000000000886a78 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_reflstr_section
0000000000886a78 g       .swift2_protocol_conformances	0000000000000000              .protected .swift3_typeref_section
0000000000886a78 g     O .swift2_protocol_conformances	0000000000000000              .hidden __TMC_END__
0000000000886a78 l    d  .swift2_protocol_conformances	0000000000000000              .swift2_protocol_conformances
0000000000886a80 l     O .swift2_protocol_conformances	0000000000002f60              l_protocol_conformances
00000000008899e0 g       .swift2_protocol_conformances	0000000000000000              .protected .swift2_protocol_conformances_end
00000000008899e0 g       .swift2_type_metadata	0000000000000000              .protected .swift2_type_metadata_start
00000000008899e0 l    d  .swift2_type_metadata	0000000000000000              .swift2_type_metadata
00000000008899e8 l     O .swift2_type_metadata	0000000000000288              l_type_metadata_table
0000000000889c70 l     O .swift2_type_metadata	0000000000000010              l_type_metadata_table
0000000000889c80 g       .swift2_type_metadata	0000000000000000              .protected .swift2_type_metadata_end
0000000000889c80 g       .swift2_type_metadata	0000000000000000              _edata
Comment by William Dillon [ 20 Mar 2016 ]

That confused me several times. I had to ask someone about it. The output is in file order, not address order.

Comment by Drew Crawford [ 20 Mar 2016 ]

Okay, I found the problem! Open source FTW. But I need help figuring out the right solution. Pinging Greg Parker Han Sangjin, who wrote the code in question.

In ProtocolConformances.cpp we look up the protocol conformance table with dlsym. However, we aren't finding the section because dlsym looks in the dynamic symbol table, not the global symbol table, and in a statically linked executable the section is inside the latter.

Here are some suggestions for solving this, which one do we pick?

  • Link with -rdynamic to promote all symbols to the dynamic symbol table. This feels heavyhanded to me.
  • We could use --dynamic-list link option to export this symbol particularly? That would require us to emit a random file as part of the build process I think?
  • Maybe there is a way to grab this symbol out of the global table without dlsym? I'm not sure how, I'm not much of an ELF expert.

Appreciate any direction.

Comment by Drew Crawford [ 20 Mar 2016 ]

Discussions offline suggest door #2. It's time I self-assigned this issue, I think

Comment by Drew Crawford [ 23 Mar 2016 ]

I've sent up the first PR (of several that will be required) on this, 1806. Review appreciated.

Will likely start on the next PR tomorrow.

Comment by Drew Crawford [ 23 Mar 2016 ]

Second PR is up 1817. This should resolve this bug for the Darwin target family. Review appreciated.

Comment by Drew Crawford [ 27 Mar 2016 ]

PR wars episode 3: return of the ELF.

We now have Linux support, completing (AFAIK) the trilogy of PRs on this topic, and resolving the underlying bug.

It's now a question of getting the changes reviewed.

Comment by Daniel Dunbar [ 30 Apr 2016 ]

Pinged PRs, would be nice to see these land so SwiftPM static lib support is unblocked.

Comment by Daniel Dunbar [ 9 Jun 2016 ]

All the PRs above have landed, Drew Crawford is this done?

Comment by Drew Crawford [ 9 Jun 2016 ]

Done! Closing

Comment by Daniel Dunbar [ 9 Jun 2016 ]

Awesome, thanks!!

Generated at Wed Feb 21 08:59:45 CST 2018 using JIRA 7.3.4#73015-sha1:a262b3457b3605f12635df4b0a0c3dc71d631a1e.