@compnerd Can we get rid of those ___start/___stop symbols now?

Oh, I was confused here. These are the new symbols that replace the old begin/end sections. They're defined by the DECLARE_SWIFT_SECTION macro in stdlib/public/runtime/SwiftRT-ELF.cpp. @compnerd suggested to me that this might be a bug in lld.

Alex, do you get the same warnings when using gold?

I don't see the same warnings emitted with either gold:

$ ./ld.gold --version
GNU gold (GNU Binutils 2.27) 1.12

or with LLD 5:

$ ./ld.lld --version
LLD 5.0.0 (compatible with GNU linkers)

The warnings are only seen with LLD 6:

$ ./ld.lld --version
LLD 6.0.0 (compatible with GNU linkers)

That is consistent with @compnerd's hypothesis that this is a bug in lld.

It looks like it is a new feature introduced into lld explicitly:

https://llvm.googlesource.com/lld/+/2f69df6d20fdf8e49fd3685e69b51df7406ac918%5E%21/#F1

Use warn() instead of error() to report a bad symbol in a DSO.

Specifically, libwidevinecdm.so in Chrome has such bad symbol.
It seems the BFD linker handles them as local symbols, so instead
of inserting them to the symbol table, we should skip them too.

Differential Revision: https://reviews.llvm.org/D41257

The question is, is the warning correct?

So I reached out to the author of the change above, and received a response:

Look at Figure 1-13 on this PDF page 19 of http://www.skyfree.org/linux/references/ELF_Format.pdf to see what it is. It says that sh_info member in a section header for the symbol type (of type SHT_SYMTAB and SHT_DYNSYM) must have an index one greater than the symbol table index of the last local symbol (binding STB_LOCAL).

So, in a symbol table, all symbols after sh_info index must be non-local. That warning message is emitted when lld found a violation of the spec.

If you want to fix it, you probably need to sort a symbol table so that all local symbols are at beginning of a table, and set its last index + 1 to sh_info member of the symbol table section header.

In that case, it seems like an LLVM problem.

One other thing (which may be worth noting here) - when linking an app with Gold on Swift 4.0.3, the older variants of the start/stop symbols were present in the dynamic symbol table but not when linked with lld 5.0. To get LLD working with Swift 4.0.3, I had to add --export-dynamic. Otherwise the following symbols weren't copied over:

.swift2_type_metadata
.swift2_type_metadata
.swift3_reflstr
.swift3_fieldmd
.swift2_protocol_conformances
.swift2_protocol_conformances
.swift3_assocty
.swift3_typeref

I suspect the same issue is present for this issue as well, where the symbols are appearing local but sorted in place with the global symbols to mark start/end sections. If the strict reading of the spec is correct, it's not possible to have local symbols intermingled with the global ones.

Raised cross-reference bug at LLVM:

https://bugs.llvm.org/show_bug.cgi?id=36771

So for the record, it seems to be that the creation of the libFoundation.so (using these start/stop symbols) causes problems.

Readelf (from binutils 2.28+) has a warning message, added last year, to warn when local symbols show up in the .dysm:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blobdiff;f=binutils/readelf.c;h=586d13c7dea2068d9adeeb4bdfccd42ade6e682a;hp=e6674c2f5fcd85d2cf55a64a99b85bce441d2cd3;hb=52c3c391f4a4a38ca921e5e7b5dce92e62079f34;hpb=3785f51aa2454dba199db8aafa80019795d536ec

Running readelf --symbols on libFoundation.so seems to suggest it is the only problematic library; others in the Swift runtime don't appear to be affected. It's not clear if there are other aspects that the compiler will generate that will cause those symbols to be emitted.

{{
root@c364d7f3cfde:/usr/lib/swift/linux# grep NAME /etc/os-release
NAME="Ubuntu"
PRETTY_NAME="Ubuntu 17.10"
VERSION_CODENAME=artful
UBUNTU_CODENAME=artful
root@c364d7f3cfde:/usr/lib/swift/linux# readelf --version
GNU readelf (GNU Binutils for Ubuntu) 2.29.1
Copyright (C) 2017 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.
root@c364d7f3cfde:/usr/lib/swift/linux# readelf --symbols libFoundation.so | head -20
readelf: Warning:
Symbol table '.dynsym' contains 18355 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000b6f750 0 NOTYPE LOCAL DEFAULT 28 __stop_swift4_protocols
local symbol 1 found at index >= .dynsym's sh_info value of 1
readelf: Warning: local symbol 2 found at index >= .dynsym's sh_info value of 1
readelf: Warning: local symbol 3 found at index >= .dynsym's sh_info value of 1
readelf: Warning: local symbol 4 found at index >= .dynsym's sh_info value of 1
2: 0000000000b71358 0 NOTYPE LOCAL DEFAULT 31 __start_swift4_typeref
3: 000000000084c100 0 NOTYPE LOCAL DEFAULT 14 __start_swift4_reflstr
4: 0000000000b6f690 0 NOTYPE LOCAL DEFAULT 28 __start_swift4_protocols
readelf: Warning: 5: 0000000000b84774 0 NOTYPE LOCAL DEFAULT 32 __stop_swift4_fieldmd
local symbol 5 found at index >= .dynsym's sh_info value of 1
readelf: Warning: local symbol 6 found at index >= .dynsym's sh_info value of 1
6: 0000000000b7cf44 0 NOTYPE LOCAL DEFAULT 32 __start_swift4_fieldmd
readelf: Warning: local symbol 7 found at index >= .dynsym's sh_info value of 1
readelf: Warning: local symbol 8 found at index >= .dynsym's sh_info value of 1
readelf: Warning: local symbol 9 found at index >= .dynsym's sh_info value of 1
7: 0000000000b7cf43 0 NOTYPE LOCAL DEFAULT 31 __stop_swift4_typeref
8: 0000000000b71358 0 NOTYPE LOCAL DEFAULT 30 __stop_swift4_type_metada
9: 0000000000b87554 0 NOTYPE LOCAL DEFAULT 33 __stop_swift4_assocty
readelf: Warning: local symbol 10 found at index >= .dynsym's sh_info value of 1
readelf: Warning: local symbol 11 found at index >= .dynsym's sh_info value of 1
readelf: Warning: local symbol 12 found at index >= .dynsym's sh_info value of 1
readelf: Warning: local symbol 13 found at index >= .dynsym's sh_info value of 1
readelf: Warning: local symbol 14 found at index >= .dynsym's sh_info value of 1
10: 0000000000b70a44 0 NOTYPE LOCAL DEFAULT 30 __start_swift4_type_metad
11: 0000000000b70a44 0 NOTYPE LOCAL DEFAULT 29 __stop_swift4_protocol_co
12: 0000000000852f18 0 NOTYPE LOCAL DEFAULT 14 __stop_swift4_reflstr
13: 0000000000b84774 0 NOTYPE LOCAL DEFAULT 33 __start_swift4_assocty
14: 0000000000b6f750 0 NOTYPE LOCAL DEFAULT 29 __start_swift4_protocol_c
15: 0000000000000000 0 FUNC GLOBAL DEFAULT UND udat_toPatternRelativeTim
16: 0000000000000000 0 FUNC GLOBAL DEFAULT UND remainder@GLIBC_2.2.5 (2)
}}

The issue is that these symbols are being emitted as LOCAL in the .dynsym section, which is presumably not the intent.

Also, I should point out that these stray symbols aren't part of Swift 4.0.3 libFoundation.so, so the way that Foundation is built between then and now has caused this issue.

We didn't use those symbols at all in Swift 4.0.3.

I can't see how they would end up being present in libFoundation.so's .dynsym section. That seems like a bug on the Swift side. @compnerd, any ideas?

Sorry @bob-wilson I should have been clearer ... when I tested against 4.0.3, I didn't see any symbols reported as warnings from either readElf or LLD. I wanted to highlight that this is a new issue for the 4.1-branch, as opposed to the specific symbols in the original bug report.

@compnerd can you comment on this?

The start/stop symbols should be created by the linker. We have declarations for them in the object. However, we explicitly adjust the visibility to mark them hidden as we do not want the symbols exported. The section symbol doesn't need to be exported at all (e.g. {{.swift4_reflstr}}). The addition of the attributes should actually prevent the start/stop symbols associated with the section from being exported.

@compnerd can you investigate why they are in fact being exported in the build of libFoundation.so, and find a way to fix it?

I can try to take a look at that. I suspect that it may be the result of using a different linker (BFD) rather than gold or lld.

Yes, it looks like the build on Ubuntu will pick up gold as the linker for the rest of the build, but CoreFoundation's build.py script doesn't get the option passed through.

I ran a test building Foundation on Ubuntu 14.04. When 'ld' points to 'ld.bfd' we see the errors reported above. When 'ld' points to 'ld.gold' then the library is built correctly.

So if we're using ld.gold for the Swift build (and passing -fuse-ld=gold) then we should ensure this is propagated through to the Foundation build, which will fix this issue.

So the solution appears to be to add --linker=gold to the invocation of the Foundation configure script. Once this is done, the linker used to create libFoundation is the same as is used to create the rest of the Swift library stack, and the problems are fixed.

This has been fixed with apple/swift-corelibs-foundation@6c54c84 on master, but needs to be backported - leaving this open until that has been done.

This has now been back ported to release branches:

Swift 4.1: apple/swift-corelibs-foundation#1535

Swift 4.2: apple/swift-corelibs-foundation#1533