Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-4443] Android: Cannot run swift executable on Android 7.0 emulator (armv7) as it is not a PIE #47020

Closed
swift-ci opened this issue Mar 31, 2017 · 46 comments
Labels
Android Platform: Android bug A deviation from expected or documented behavior. Also: expected but undesirable behavior. compiler The Swift compiler in itself

Comments

@swift-ci
Copy link
Collaborator

Previous ID SR-4443
Radar None
Original Reporter qasim (JIRA User)
Type Bug
Status Resolved
Resolution Done
Additional Detail from JIRA
Votes 3
Component/s Compiler
Labels Bug, Android
Assignee jblatecky (JIRA)
Priority Medium

md5: c637650368058adb0695bd0eb668326a

relates to:

  • SR-5405 Security: Swift on Linux not compiled as position independent executable

Issue Description:

I'm having trouble running a compiled swift file on an Android 7.0 emulator. I've set up swift based on https://github.com/apple/swift/blob/master/docs/Android.md, and the error I get on device is:

"/data/local/tmp/hello": error: only position independent executables (PIE) are supported. Aborted

Naively, I thought maybe I can pass -Xlinker -pie to the swiftc command to build hello.swift, but that results in the following when I execute the produced file:

CANNOT LINK EXECUTABLE "./hello": cannot locate symbol "__atomic_load" referenced by "/data/local/tmp/libswiftCore.so"... Aborted

I'm using android-ndk-r14b, and built the swift compiler for API level 21. All other steps seem to have run fine except for running the executable at the end.

@belkadan
Copy link
Contributor

We should probably just make everything PIE on Android. I don't think we have any non-PIE features.

@swift-ci
Copy link
Collaborator Author

Comment by Qasim Iqbal (JIRA)

@belkadan Agreed. Is this something I could do with flags on my current built swiftc, or does it have to do with how Swift itself is built? I should learn more about this 😃

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

@milseman Can you help in this? How can we generate all .so libraries with -pie ?

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

Adding @modocache

@milseman
Copy link
Mannequin

milseman mannequin commented May 26, 2017

My guess is that you'll want to append to SWIFT_RUNTIME_CORE_CXX_FLAGS and maybe CMAKE_CXX_FLAGS on Android. If you look at e.g. swift/CMakeLists.txt and swift/stdlib/public/CMakeLists.txt, there are conditional additions based on platform. Unfortunately I'm not too well versed beyond that.

@belkadan I think this could be a good starter task and Amr seems motivated. Could you help give direction?

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

Adding @gottesmm

@swift-ci
Copy link
Collaborator Author

Comment by Julien Blatecky (JIRA)

Hi! I'm encountering the same problem on Android and I found a PR that seems to be similar to what it's intended to be done here. It was about disabling PIC on Windows : https://github.com/apple/swift/pull/6407/files.

I'm testing a modification on this point to be able to run a Swift library on Android 7, if it works, I could make a PR on Github.

@milseman
Copy link
Mannequin

milseman mannequin commented May 30, 2017

If that works, please do!

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

I tried with -fno-pic and I got error:
/usr/bin/ld.gold: error: lib/libswiftDemangling.a(Remangler.cpp.o): requires dynamic R_X86_64_PC32 reloc against '_Znwm' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: lib/libswiftDemangling.a(Remangler.cpp.o): requires dynamic R_X86_64_PC32 reloc against '_Znwm' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: lib/libswiftDemangling.a(Remangler.cpp.o): requires dynamic R_X86_64_PC32 reloc against 'ZNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_mutateEmmPKcm' which may overflow at runtime; recompile with -fPIC
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

@swift-ci
Copy link
Collaborator Author

Comment by Julien Blatecky (JIRA)

I tried with -fpie and -fpic for Android and the "non-PIE" problem seems to have gone. But the missing "__atomic_load" seems to persist so I searched the NDK doc and I found that we need "-latomic" to build for Android NDK. In the AddSwift.cmake file the flag is set for Linux target, not for Android. I'm currently testing it with this flag and if it works I'll open a Pull Request.

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

I tried with:
list(APPEND SWIFT_RUNTIME_CORE_CXX_FLAGS "-fpie")
in public/CMakeLists.txt

And I got this error:
swift/stdlib/public/stubs/LibcShims.cpp:32:1: error: static_assert failed "__swift_ssize_t must be defined as equivalent to ssize_t"
static_assert(std::is_same<ssize_t, swift::__swift_ssize_t>::value,

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

I am still getting this error even after commenting out -fpie line:

swift/stdlib/public/stubs/LibcShims.cpp:32:1: error: static_assert failed "__swift_ssize_t must be defined as equivalent to ssize_t"
static_assert(std::is_same<ssize_t, swift::__swift_ssize_t>::value,
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Should I create a separate bug for this?

@milseman
Copy link
Mannequin

milseman mannequin commented May 31, 2017

Yes please

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

Done https://bugs.swift.org/browse/SR-5059

@swift-ci
Copy link
Collaborator Author

swift-ci commented Jun 8, 2017

Comment by Amr Aboelela (JIRA)

This PR #9869

Fixed the libatomic issue, but now I am getting:

CANNOT LINK EXECUTABLE: could not load library "libswiftCore.so" needed by "/data/local/tmp/hello"; caused by could not load library "libicudata.so" needed by "libswiftCore.so"; caused by library "libicudata.so" not found

Because in that PR, he also added "-licudata" "-licui18n" "-licuuc" so not sure if we really need those libraries.

@swift-ci
Copy link
Collaborator Author

swift-ci commented Jun 9, 2017

Comment by Julien Blatecky (JIRA)

I tried a lot of things about this problem and it seems to be multiple ones.

Firstly, the non-PIE problem prevents to launch any swift executable on Android. I made a little modification in AddSwift.cmake but I can't say if it changes anything because I cannot launch anything coming from a Swift code on an Android device because of the second problem.

Secondly, the missing __atomic_load symbol comes from the fact that Swift for Android was not link against -latomic as specified in the Android documentation (Atomic support part): https://developer.android.com/ndk/guides/cpp-support.html#cs

But, if we only add -latomic when building Swift, the linker does not seem to link it. I suppose that it found the atomic references in the shared ICU library.

Here comes another Android problem: we cannot safely load our own shared ICU library because, for some reason, Android keeps loading its own ICU library that does not contain the atomic symbols, causing the __atomic_load crash. In the SwiftJava repository they used a workaround: renaming libicu*.so files to libscu*.so files. There is an existing bug in the tracker with some details: https://bugs.swift.org/browse/SR-1359.

That's why the PR #9869 added -licudata -licui18n -licuuc to the Swift build for Android. However, I think it could be an incomplete fix because, as stated in the Gist from https://bugs.swift.org/browse/SR-1359, the order matters when statically linking to ICU.

To sum it up: I'm testing it with the right order and I come back to you with the results.

@swift-ci
Copy link
Collaborator Author

swift-ci commented Jun 9, 2017

Comment by Amr Aboelela (JIRA)

Anybody knows why the Hello World app used to be working ok before? So what changed to make it fail? Is it because using a different Android API version? Or did something happened in the swift packages?

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

I tried it with this order: "-ldl" "-llog" "-latomic" "-licui18n" "-licudata" "-licuuc"

$ adb shell LD_LIBRARY_PATH=/data/local/tmp /data/local/tmp/hello

And I got: dlopen() failed on `/data/local/tmp/hello': dlopen failed: Aborted

@swift-ci
Copy link
Collaborator Author

Comment by Julien Blatecky (JIRA)

Yesterday I had the same problem on my app, it was failing differently whether I was using a device or another. On the first one it was trying to load a library named as the package name com.xxxxxx.app, on the second one it was trying to load a private library that is not allowed to be loaded anymore by Android libcutils.so.

I found that this error comes from stdlib/public/runtime/ImageInspectionELF.cpp, starting from this PR.

I don't know why, but Swift Runtime on Android is trying to load some libraries that are not even libraries, and others that are not needed and private, and adding this fatalError causes the app failure. When I remove this fatalError I can run my app with Swift as native library.

@milseman
Copy link
Mannequin

milseman mannequin commented Jun 13, 2017

This is way beyond my familiarity with the build system for Android. @modocache does any of this look familiar to you?

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

@milseman After many research I found that the possible problem in Android is has to do with your introduction to pthread usage so I tried to revert the pthread code but it was too much, then i tried to add "-lpthread" in the AddSwift.cmake then I got arm-linux-androideabi-4.9/prebuilt/linux-x86_64/arm-linux-androideabi/bin/ld.gold: error: cannot find -lpthread

@swift-ci
Copy link
Collaborator Author

Comment by Adrian Catana (JIRA)

Did anyone manage to solve the PIE issue? At the moment, I also tried -Xlinker -pie. The thing is that I can only run the emulator on Mac, so I had to manually copy all the .so and the hello executable and push them from Mac on the emulator using adb. Without the -Xlinker -pie I get PIE error, while when I use it I get CANNOT LINK EXECUTABLE "/data/local/tmp/hello": library "libicudata.so" not found.

@milseman
Copy link
Mannequin

milseman mannequin commented Jul 20, 2017

amraboelela (JIRA User) For Android, I think that their stdlib might have pthread in it without needing an explicit link command. You can probably drop the "-lpthread".

adicatana (JIRA User) That sounds like you're getting further and just need to find the library path for libicudata.so. Or, if libicudata.so is bundled up in another library already in the library path, adjust the link command to that.

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

adicatana (JIRA User) You will have to push the libicudata.so too, but still it didn't work with me, I am suspecting that Android don't allow running executables natively anymore.

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 3, 2017

Comment by Mathew Polzin (JIRA)

Based on https://stackoverflow.com/questions/36710934/android-unsatisfiedlinkerror-with-libicudata-after-update-ndk-to-r11, `libicudata.so` happened to be included in the NDK at one point but it has never been officially supported. So, if it is needed, it will need to be built and included along with our swift projects (as attempted by Amr). I'm just starting to look into this myself, but hopefully one of us can get this sorted out!

@milseman
Copy link
Mannequin

milseman mannequin commented Aug 3, 2017

Interesting. There's also an argument to be made that even Linux might want to consider this route, as the LTS releases don't update their ICU.

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 3, 2017

Comment by Amr Aboelela (JIRA)

Is there anyone able to run Hello World or any swift app in Android recently? because I still can't 🙁

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 3, 2017

Comment by Julien Blatecky (JIRA)

Yes, just before my holidays I succeeded to build and run a swift lib integrated inside an android app, with libdispatch and Foundation built in it (and a custom version of Alamofire). I currently don't have access to my computer until the end of my holidays but on the 14th August I plan to report here all the steps to succeed.

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 3, 2017

Comment by Amr Aboelela (JIRA)

jblatecky (JIRA User) Ok thanks for your efforts, we will wait for your build script.

What is strange for me is that I was able to run the Hello World app in Android early this year maybe on May or April. Is the problem comes when using Android 7.0 emulator, was I using a previous version like Android 6.0 or something?

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 3, 2017

Comment by Mathew Polzin (JIRA)

This appears to be a problem that was solved in ewmailing@8af36d6 but that is a fork off of the `SwiftAndroid` github so the solution is not trivial to apply to the current `apple/Swift` repository. I am giving it a shot nonetheless in case I get somewhere with that before we hear back from jblatecky (JIRA User).

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 4, 2017

Comment by Adrian Catana (JIRA)

Has anyone tried making a dynamic library out of a .swift source and calling it from Java using JNI? Like here https://medium.com/@ephemer/using-jni-in-swift-to-put-an-app-into-the-android-play-store-732e542a99dd? Or at least calling from Java using JNI a C function which calls a swift function?

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 4, 2017

Comment by Amr Aboelela (JIRA)

adicatana (JIRA User) I tried, and the Android Studio was not able to recognize my .so file built from swift code, giving file not found error!
jblatecky (JIRA User) said he was able to do it, so we have to wait for him later to show us how.

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Amr Aboelela (JIRA)

Good news, I am just able to run "Hello Android" app using the following emulator:

Name: Nexus 4 API 24, Android 7.0

The build script for the hello.swift is:

swiftc \

-tools-directory ${ANDROID_NDK_DIR}/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/arm-linux-androideabi/bin \\

-target armv7-none-linux-androideabi \\

-sdk ${ANDROID_NDK_DIR}/platforms/android-21/arch-arm \\

-L ${ANDROID_NDK_DIR}/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a \\

-L ${ANDROID_NDK_DIR}/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/lib/gcc/arm-linux-androideabi/4.9.x \\

-Xlinker -pie \\

hello.swift

${ANDROID_NDK_DIR} value is ~/android/android-ndk-r14b

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Amr Aboelela (JIRA)

Also you need to push icu libs:

adb push swift-android-toolchain/usr/lib/swift/android/libicudata.so /data/local/tmp

adb push swift-android-toolchain/usr/lib/swift/android/libicui18n.so /data/local/tmp

adb push swift-android-toolchain/usr/lib/swift/android/libicuuc.so /data/local/tmp

So someone need to update the swift/docs/Android.md file

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Amr Aboelela (JIRA)

While using emulator Nexus API 21, Android 5.0, the one i was trying for couple months, I get this error:

adb shell LD_LIBRARY_PATH=/data/local/tmp /data/local/tmp/hello

dlopen() failed on `/data/local/tmp/hello': dlopen failed: Aborted

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Amr Aboelela (JIRA)

In Nexus 5X API 22 (Android 5.1.1, API 22)

I got:

WARNING: linker: /data/local/tmp/hello: unused DT entry: type 0x6ffffffe arg 0xcb8

WARNING: linker: /data/local/tmp/hello: unused DT entry: type 0x6fffffff arg 0x1

WARNING: linker: libswiftCore.so: unused DT entry: type 0x6ffffffe arg 0x9d25c

WARNING: linker: libswiftCore.so: unused DT entry: type 0x6fffffff arg 0x3

WARNING: linker: libswiftSwiftOnoneSupport.so: unused DT entry: type 0x6ffffffe arg 0x1db18

WARNING: linker: libswiftSwiftOnoneSupport.so: unused DT entry: type 0x6fffffff arg 0x1

WARNING: linker: libicui18n.so: unused DT entry: type 0x6ffffffe arg 0x85e84

WARNING: linker: libicui18n.so: unused DT entry: type 0x6fffffff arg 0x2

WARNING: linker: libicuuc.so: unused DT entry: type 0x6ffffffe arg 0x31c10

WARNING: linker: libicuuc.so: unused DT entry: type 0x6fffffff arg 0x3

WARNING: linker: libc++_shared.so: unused DT entry: type 0x6ffffffe arg 0x2adb0

WARNING: linker: libc++_shared.so: unused DT entry: type 0x6fffffff arg 0x3

dlopen() failed on `/data/local/tmp/hello': dlopen failed: library "/data/local/tmp/hello" wasn't loaded and RTLD_NOLOAD prevented itAborted

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Amr Aboelela (JIRA)

I can update the Android.md file and send a pull request if that's ok.

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Adrian Catana (JIRA)

I confirm that it works for me as well for Nexus 4. Great job!

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Amr Aboelela (JIRA)

Nexus 5X API 24 Android 7.0 also works.

Nexus 4 API 23 Android 6 crashed after running it.

So it seems like API 24 emulators are the only ones working.

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Amr Aboelela (JIRA)

It works even without the -Xlinker -pie in the swiftc for compiling the hello.swift file.

I am using the latest swift repos, maybe they made everything pie by default?

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Amr Aboelela (JIRA)

I see that in swift repository Driver/ToolChains.cpp

if (getTriple().getOS() == llvm::Triple::Linux) {

Arguments.push_back("-pie");

}

Added by: Grzegorz Milos in Jul 31, 2017: Add -pie flag to clang++ linker invocation.

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Amr Aboelela (JIRA)

@milseman Do you want me to update the Android.md file?

@milseman
Copy link
Mannequin

milseman mannequin commented Aug 9, 2017

Please do!

@swift-ci
Copy link
Collaborator Author

swift-ci commented Aug 9, 2017

Comment by Amr Aboelela (JIRA)

#11404

#11406

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

Created new issue: https://bugs.swift.org/browse/SR-5672

I think we can close this one after my PRs get merged.

@swift-ci
Copy link
Collaborator Author

Comment by Amr Aboelela (JIRA)

Marking it resolved as both pull requests got merged

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Android Platform: Android bug A deviation from expected or documented behavior. Also: expected but undesirable behavior. compiler The Swift compiler in itself
Projects
None yet
Development

No branches or pull requests

2 participants