[SR-10539] swift-api-digester: can't get it to work on Linux #52939

weissi · 2019-04-23T17:17:39Z


Previous ID	SR-10539
Radar	None
Original Reporter	@weissi
Type	Bug
Status	Resolved
Resolution	Done

Additional Detail from JIRA


Votes	0
Component/s	Source Tooling
Labels	Bug
Assignee	@nkcsgexi
Priority	Medium

md5: 053bdb70bea20d227cceaf10b751fc9b

Issue Description:

When I try to run swift-api-digester on Linux, I get

root@e9c20c8ba46d:/Users/johannes/devel/swift-nio# ./swift-api-digester -sdk / -dump-sdk -module NIO -o nio-1.13.2.json -I .build/debug 
Failed to load Swift stdlib

One of the issues seems to be that it assumes Swift.swiftmodule in one of those places:

openat(AT_FDCWD, ".build/debug/Swift.swiftmodule", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/Users/johannes/devel/lib/swift/linux/x86_64/Swift.swiftmodule", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

which doesn't make much sense. Just to see where we get, I've made a symlink

ln -s /usr/lib /Users/johannes/devel/lib

and sure enough, we're getting quite a bit further:

root@e9c20c8ba46d:/Users/johannes/devel/swift-nio# ./swift-api-digester -sdk / -dump-sdk -module NIO -o nio-1.13.2.json -I .build/debug 
<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "DispatchOverlayShims.h"
        ^
/Users/johannes/devel/lib/swift/shims/DispatchOverlayShims.h:219:20: error: implicit conversion of C pointer type 'dispatch_data_t _Nonnull' (aka 'struct dispatch_data_s *') to Objective-C pointer type '__swift_shims_dispatch_data_t _Nonnull' (aka 'id') requires a bridged cast
    return applier(data, off, loc, size);
                   ^
/Users/johannes/devel/lib/swift/shims/DispatchOverlayShims.h:219:20: note: use __bridge to convert directly (no change in ownership)
    return applier(data, off, loc, size);
                   ^
/Users/johannes/devel/lib/swift/shims/DispatchOverlayShims.h:219:20: note: use __bridge_transfer to transfer ownership of a +1 'dispatch_data_t _Nonnull' (aka 'struct dispatch_data_s *') into ARC
    return applier(data, off, loc, size);
                   ^
<unknown>:0: error: could not build Objective-C module '_SwiftDispatchOverlayShims'

but now swift-api-digester seems to think that we have ObjC support enabled on Linux...

Version is 5.0.1.

The text was updated successfully, but these errors were encountered:

belkadan · 2019-04-24T16:43:41Z

cc @akyrtzi

akyrtzi · 2019-04-24T17:25:11Z

It seems to me like you copied the `swift-api-digester` binary to some `/Users/johannes/devel/swift-nio` directory and try to run it from there, is this correct ?

If this is correct and I did not misunderstood then this is not going to work, the same way it's not going to work if you copy the swift binary out of the toolchain and try to run it. You should run `swift-api-digester` from inside the toolchain it came with. I cannot guarantee it will definitely work though, we haven't tried to run it on linux.

weissi · 2019-04-24T17:26:46Z

@akyrtzi aaah, sorry! Yes, you are absolutely right 🙂. Let me copy it next to swiftc

weissi · 2019-04-24T17:28:30Z

@akyrtzi that gets us to the same point as after the symlink:

root@595fa0a7127f:/Users/johannes/devel/swift-nio# /usr/bin/swift-api-digester -sdk / -dump-sdk -module NIO -o nio-1.13.2.json -I .build/debug
<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "DispatchOverlayShims.h"
        ^
/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: error: implicit conversion of C pointer type 'dispatch_data_t _Nonnull' (aka 'struct dispatch_data_s *') to Objective-C pointer type '__swift_shims_dispatch_data_t _Nonnull' (aka 'id') requires a bridged cast
    return applier(data, off, loc, size);
                   ^
/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: note: use __bridge to convert directly (no change in ownership)
    return applier(data, off, loc, size);
                   ^
/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: note: use __bridge_transfer to transfer ownership of a +1 'dispatch_data_t _Nonnull' (aka 'struct dispatch_data_s *') into ARC
    return applier(data, off, loc, size);
                   ^
<unknown>:0: error: could not build Objective-C module '_SwiftDispatchOverlayShims'

akyrtzi · 2019-04-24T17:43:15Z

Where did you get it from, you should not be copying it at all, try running it from the OSS toolchain it came with (after making sure the same OSS toolchain was used to build swift-nio).

weissi · 2019-04-24T17:47:35Z

@akyrtzi the OSS toolchains don't ship it:

curl -s https://swift.org/builds/swift-5.0.1-release/ubuntu1804/swift-5.0.1-RELEASE/swift-5.0.1-RELEASE-ubuntu18.04.tar.gz | tar tz | grep swift-api-digester
-1-

So @tomerd built it from source and I'm copying it into the OSS toolchain of the same version. So they should be compatible I'd assume.

belkadan · 2019-04-24T17:58:06Z

That "Objective-C pointer type" bit is suspect. That kind of sounds like it's building for an Apple target. What happens if you pass an explicit -target?

weissi · 2019-04-24T18:01:54Z

@belkadan I tried it with x86_64-unknown-linux but no change. Just to be sure I also deleted the Darwin build in the .build directory and specified the LInux one:

$ swift-api-digester -sdk / -target x86_64-unknown-linux -dump-sdk -module NIO -o nio-1.13.2.json -I .build/x86_64-unknown-linux/debug/
<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "DispatchOverlayShims.h"
        ^
/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: error: implicit conversion of C pointer type 'dispatch_data_t _Nonnull' (aka 'struct dispatch_data_s *') to Objective-C pointer type '__swift_shims_dispatch_data_t _Nonnull' (aka 'id') requires a bridged cast
    return applier(data, off, loc, size);
                   ^
/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: note: use __bridge to convert directly (no change in ownership)
    return applier(data, off, loc, size);
                   ^
/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: note: use __bridge_transfer to transfer ownership of a +1 'dispatch_data_t _Nonnull' (aka 'struct dispatch_data_s *') into ARC
    return applier(data, off, loc, size);
                   ^
<unknown>:0: error: could not build Objective-C module '_SwiftDispatchOverlayShims'

akyrtzi · 2019-04-24T18:03:34Z

Could you try with an OSS toolchain that already contains it ? And build swift-nio using the same toolchain before using it.
Otherwise it's not clear this is an actual issue or a mismatch with the binary and the toolchain you want to run it from.

tomerd · 2019-04-24T20:39:30Z

it was compiled off the swift-5.0-branch on ubuntu 18.04. i don’t think it ships with the linux distributable, which is why we compiled it ourselves

akyrtzi · 2019-04-24T20:52:06Z

The binary is in latest snapshots, master and swift-5.1

weissi · 2019-04-25T11:54:55Z

Thanks @akyrtzi. I tried with the latest 5.1 snapshot, same issue:

$ /usr/local/swift/swift-5.1-DEVELOPMENT-SNAPSHOT-2019-04-17-a-ubuntu16.04/usr/bin/swift-api-digester -sdk / -target x86_64-unknown-linux -dump-sdk -module NIO -o nio-1.13.2.json -I .build/x86_64-unknown-linux/debug/
<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "DispatchOverlayShims.h"
        ^
/usr/local/swift/swift-5.1-DEVELOPMENT-SNAPSHOT-2019-04-17-a-ubuntu16.04/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: error: implicit conversion of C pointer type 'dispatch_data_t _Nonnull' (aka 'struct dispatch_data_s *') to Objective-C pointer type '__swift_shims_dispatch_data_t _Nonnull' (aka 'id') requires a bridged cast
    return applier(data, off, loc, size);
                   ^
/usr/local/swift/swift-5.1-DEVELOPMENT-SNAPSHOT-2019-04-17-a-ubuntu16.04/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: note: use __bridge to convert directly (no change in ownership)
    return applier(data, off, loc, size);
                   ^
/usr/local/swift/swift-5.1-DEVELOPMENT-SNAPSHOT-2019-04-17-a-ubuntu16.04/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: note: use __bridge_transfer to transfer ownership of a +1 'dispatch_data_t _Nonnull' (aka 'struct dispatch_data_s *') into ARC
    return applier(data, off, loc, size);
                   ^
<unknown>:0: error: could not build Objective-C module '_SwiftDispatchOverlayShims'

weissi · 2019-04-25T12:24:56Z

even with the Swift toolchain installed in /usr/bin, same error:

$ jw-docker norionomura/swift:5120190416a
root@c9181fd5d4d8:/Users/johannes/devel/swift-nio# rm -rf .build/
root@c9181fd5d4d8:/Users/johannes/devel/swift-nio# swift build
[125/125] Linking NIOChatClient
root@c9181fd5d4d8:/Users/johannes/devel/swift-nio# swift-api-digester -sdk / -target x86_64-unknown-linux -dump-sdk -module NIO -o nio-1.13.2.json -I .build/x86_64-unknown-linux/debug/

<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "DispatchOverlayShims.h"
        ^
/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: error: implicit conversion of C pointer type 'dispatch_data_t _Nonnull' (aka 'struct dispatch_data_s *') to Objective-C pointer type '__swift_shims_dispatch_data_t _Nonnull' (aka 'id') requires a bridged cast
    return applier(data, off, loc, size);
                   ^
/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: note: use __bridge to convert directly (no change in ownership)
    return applier(data, off, loc, size);
                   ^
/usr/lib/swift/shims/DispatchOverlayShims.h:219:20: note: use __bridge_transfer to transfer ownership of a +1 'dispatch_data_t _Nonnull' (aka 'struct dispatch_data_s *') into ARC
    return applier(data, off, loc, size);
                   ^
<unknown>:0: error: could not build Objective-C module '_SwiftDispatchOverlayShims'
root@c9181fd5d4d8:/Users/johannes/devel/swift-nio# 
root@c9181fd5d4d8:/Users/johannes/devel/swift-nio# which swift-api-digester
/usr/bin/swift-api-digester

weissi · 2019-04-25T13:35:39Z

@akyrtzi here's also a one-line repro that you can run straight from your Mac (or Linux obvs):

docker run -it --rm norionomura/swift:5120190416a bash -c 'git clone https://github.com/apple/swift-nio && cd swift-nio && swift build && swift-api-digester -sdk / -target x86_64-unknown-linux -dump-sdk -module NIO -o nio.json -I .build/x86_64-unknown-linux/debug/ && cat nio.json'

akyrtzi · 2019-04-25T21:49:22Z

I can reproduce. I suspect this tool needs a '-disable-objc-interop' option. Or for something fancier disable it automatically like the driver does.

weissi · 2019-04-30T08:43:32Z

@akyrtzi Cool, thanks, that makes sense. On Linux it could also just always disable objc interop, right?

weissi · 2019-05-14T21:28:34Z

looks like this should fix it: #24776

nkcsgexi · 2019-05-15T19:34:55Z

These two PRs should disable '-disable-objc-interop' on linux:

master branch PR: #24776
5.1 branch PR: #24789

weissi · 2019-05-17T17:25:20Z

I can confirm it now works on Linux, thanks so much @nkcsgexi!

nkcsgexi · 2019-05-17T18:54:33Z

No worries, @weissi. Please let us know any issues or feature requests you may have for this tool 🙂

weissi · 2019-05-17T19:00:10Z

thank you @nkcsgexi, we will definitely do that. We're working on using it in SwiftNIO's CI system!

Btw, did you see https://bugs.swift.org/browse/SR-10489 ?

nkcsgexi · 2019-05-17T19:27:50Z

ah, I just self-assigned the issue and will take a look.

swift-ci transferred this issue from apple/swift-issues Apr 25, 2022

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SR-10539] swift-api-digester: can't get it to work on Linux #52939

[SR-10539] swift-api-digester: can't get it to work on Linux #52939

weissi commented Apr 23, 2019

belkadan commented Apr 24, 2019

akyrtzi commented Apr 24, 2019

weissi commented Apr 24, 2019

weissi commented Apr 24, 2019

akyrtzi commented Apr 24, 2019

weissi commented Apr 24, 2019

belkadan commented Apr 24, 2019

weissi commented Apr 24, 2019

akyrtzi commented Apr 24, 2019

tomerd commented Apr 24, 2019

akyrtzi commented Apr 24, 2019

weissi commented Apr 25, 2019

weissi commented Apr 25, 2019

weissi commented Apr 25, 2019

akyrtzi commented Apr 25, 2019

weissi commented Apr 30, 2019

weissi commented May 14, 2019

nkcsgexi commented May 15, 2019

weissi commented May 17, 2019

nkcsgexi commented May 17, 2019

weissi commented May 17, 2019

nkcsgexi commented May 17, 2019

[SR-10539] swift-api-digester: can't get it to work on Linux #52939

[SR-10539] swift-api-digester: can't get it to work on Linux #52939

Comments

weissi commented Apr 23, 2019

belkadan commented Apr 24, 2019

akyrtzi commented Apr 24, 2019

weissi commented Apr 24, 2019

weissi commented Apr 24, 2019

akyrtzi commented Apr 24, 2019

weissi commented Apr 24, 2019

belkadan commented Apr 24, 2019

weissi commented Apr 24, 2019

akyrtzi commented Apr 24, 2019

tomerd commented Apr 24, 2019

akyrtzi commented Apr 24, 2019

weissi commented Apr 25, 2019

weissi commented Apr 25, 2019

weissi commented Apr 25, 2019

akyrtzi commented Apr 25, 2019

weissi commented Apr 30, 2019

weissi commented May 14, 2019

nkcsgexi commented May 15, 2019

weissi commented May 17, 2019

nkcsgexi commented May 17, 2019

weissi commented May 17, 2019

nkcsgexi commented May 17, 2019