@compnerd, I know PowerPC64LE isn't one of our supported platforms yet (though Sarvesh is making progress on it!), but does this ICU-related error look familiar?

Comment by Sarvesh Tamba (JIRA)

Exported LD_LIBRARY_PATH to reflect the path containing libicui18nswift.so.61. It worked after this change. However I am facing the below problem:-

Out of the 109 remaining lldb related test failures I am seeing the following kind of errors for 107 tests. On careful observation I found that the --arch parameter passed to the dotest.py is empty. i.e. "Command invoked: /root/swift-source/lldb/test/dotest.py -q --arch= -s /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-traces". Is this expected or this could be causing issues?

Detailed error is as below(seen for 107/109 failures):-

********************

TEST 'lldb-Suite :: lang/swift/archetype_resolution_subclass/TestArchetypeResolutionSubclass.py' FAILED

********************

lldb version 7.0.0 (https://github.com/apple/swift-lldb.git revision a6ad475f9ee5558baf6baf49e2be7b77ff46df5f)
Swift-5.0 (revision d29b545)
clang revision 38ef19dd1f3ea3d09a8b7e77b29aa95fd6ad7d87
llvm revision 28bfcfbd091e6112f164c00f9c555cd2cadfcc64
LLDB library dir: /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/bin
LLDB import library dir: /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/bin
Skipping following debug info categories: ['dsym', 'gmodules']

Session logs for test failures/errors/unexpected successes will go into directory '/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-traces'
Command invoked: /root/swift-source/lldb/test/dotest.py -q --arch= -s /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-traces --build-dir /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex -S nm -u CXXFLAGS -u CFLAGS --executable /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/./bin/lldb --dsymutil /root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/./bin/dsymutil --filecheck /root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/./bin/FileCheck -C /root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang --swift-compiler /root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swiftc --swift-library /root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy --build-dir /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex --test-subdir lang/swift --skip-category=watchpoint --skip-category=dwo -t -E -Xcc -F/root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/CoreFoundation-prefix/System/Library/Frameworks -I/root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/swift -I/root/swift-source/swift-corelibs-libdispatch -L/root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le -L/root/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le -L/root/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le/src -Xlinker -rpath -Xlinker /root/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le/src -Xlinker -rpath -Xlinker /root/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le -Xlinker -rpath -Xlinker /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass -p TestArchetypeResolutionSubclass.py
Change dir to: /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass
UNSUPPORTED: LLDB (/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7-ppc64le) :: test_dsym (lldbsuite.test.lldbtest.TestArchetypeResolutionSubclass) (test case does not fall in any category of interest for this run)
runCmd: settings set symbols.clang-modules-cache-path "/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/module-cache-lldb"
output: None

runCmd: settings set symbols.enable-external-lookup false
output: None

os command: make VPATH=/root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass -C /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf -I /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass -f /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass/Makefile MAKE_DSYM=NO ARCH=ppc64le CC=/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7 SWIFTC=/root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swiftc
with pid: 5241
stdout: make: Entering directory '/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf'

Compiling /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass/main.swift

/root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swift -frontend -c -primary-file /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass/main.swift

-g -Onone -serialize-debugging-options -module-cache-path /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/module-cache-clang -sdk / -enable-anonymous-context-mangled-names -module-name a -emit-module-path
/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf/main.partial.swiftmodule
-o /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf/main.o

Merging swift modules for a

/root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swift -frontend -g -Onone -serialize-debugging-options -module-cache-path /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/module-cache-clang -sdk / -enable-anonymous-context-mangled-names -merge-modules
-emit-module main.partial.swiftmodule
-parse-as-library -sil-merge-partial-modules
-disable-diagnostic-passes -disable-sil-perf-optzns
-module-name a
-o /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf/a.swiftmodule

Wrapping swift module

/root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swift -modulewrap a.swiftmodule -o /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf/a.swiftmodule.o

Linking a.out

/root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swiftc a.swiftmodule.o main.o -Onone -Xfrontend -serialize-debugging-options -module-cache-path /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/module-cache-clang -sdk "/" -o "a.out"
make: Leaving directory '/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf'

stderr: /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass/main.swift:11:1: warning: result of call to 'f' is unused
f(x)
^~~~

retcode: 0

FAIL: LLDB (/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7-ppc64le) :: test_dwarf (lldbsuite.test.lldbtest.TestArchetypeResolutionSubclass)
<bound method SBProcess.Kill of <lldb.SBProcess; proxy of <Swig Object of type 'lldb::SBProcess *' at 0x3fff767cee70> >>: success

UNSUPPORTED: LLDB (/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7-ppc64le) :: test_dwo (lldbsuite.test.lldbtest.TestArchetypeResolutionSubclass) (test case does not fall in any category of interest for this run)
UNSUPPORTED: LLDB (/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7-ppc64le) :: test_gmodules (lldbsuite.test.lldbtest.TestArchetypeResolutionSubclass) (test case does not fall in any category of interest for this run)
Restore dir to: /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/test-results

FAIL: test_dwarf (lldbsuite.test.lldbtest.TestArchetypeResolutionSubclass)

Traceback (most recent call last):
File "/root/swift-source/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1744, in test_method
return attrvalue(self)
File "/root/swift-source/lldb/packages/Python/lldbsuite/test/decorators.py", line 144, in wrapper
func(*args, **kwargs)
File "/root/swift-source/lldb/packages/Python/lldbsuite/test/lldbinline.py", line 134, in _test
self.do_test()
File "/root/swift-source/lldb/packages/Python/lldbsuite/test/lldbinline.py", line 161, in do_test
"inline test did not hit a single breakpoint")
AssertionError: False is not True : inline test did not hit a single breakpoint
Config=ppc64le-/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7

Ran 4 tests in 3.177s

RESULT: FAILED (0 passes, 1 failures, 0 errors, 3 skipped, 0 expected failures, 0 unexpected successes)

Comment by Sarvesh Tamba (JIRA)

@belkadan @compnerd, sorry to bother once again, does the above error look familiar?

Comment by Sarvesh Tamba (JIRA)

The following values were traced while debugging in File "/root/swift-source/lldb/packages/Python/lldbsuite/test/lldbinline.py", line 171, in 'do_test', which is involved in the lldb test failures and reports "AssertionError: False is not True : inline test did not hit a single breakpoint" for 51 test failures:-

exe:-
/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/associated_self_type/lldbsuite.test.lldbtest.test_dwarf/a.out

source_files:-
['main.swift']

target:-
a.out

process:-
SBProcess: pid = 20460, state = exited, threads = 1, executable = a.out

Since the process is in exited state on start up, the intended breakpoints do not get hit in the while loop. Hence the failures are seen.
Wondering why the process exits on start up.

Comment by Sarvesh Tamba (JIRA)

All failing lldb test cases have the following type of error message:-
<bound method SBProcess.Kill of <lldb.SBProcess; proxy of <Swig Object of type 'lldb::SBProcess *' at 0x3fff9c15cf90> >>: success

Comment by Sarvesh Tamba (JIRA)

After skipping the failing lldb tests, I am able to complete the libdispatch test suite. However, facing the following issues in TestFoundation test suite:-

[257/257][100%][91.650s] Running utility command for TestFoundation

— Running tests for foundation

• • • • • • test —

+ /usr/bin/cmake --build /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le – -j8 test

[0/1][ 0%][0.000s] Running tests...

Test project /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le

Start 1: TestFoundation

1/1 Test #1: TestFoundation ...................***Failed 0.00 sec

/root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/TestFoundation/TestFoundation: error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 0.00 sec

The following tests FAILED:

1 - TestFoundation (Failed)

Errors while running CTest

FAILED: CMakeFiles/test.util

cd /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le && /usr/bin/ctest --force-new-ctest-process

ninja: build stopped: subcommand failed.

Building the standard library for: swift-stdlib-linux-powerpc64le

Running Swift tests for: check-swift-all-linux-powerpc64le check-swift-all-optimize-linux-powerpc64le

./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

Comment by Sarvesh Tamba (JIRA)

1.) Tried setting ICU_ROOT & DICU_ROOT in "swift/utils/build-script-impl". However faced the same errors as before - "error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory". Correct me if this is not correct approach to set the two variables.
2.) Used the 'llvm-readelf' utility to find details about the libraries linked with TestFoundation binary:-

{{ sar@e2641fd15168:~$ /home/sar/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/llvm-readelf -dynamic-table /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/TestFoundation/TestFoundation
DynamicSection [ (40 entries)
Tag Type Name/Value 0x0000000000000003 PLTGOT 0x7d9eb80x0000000000000002 PLTRELSZ 41448 (bytes) 0x0000000000000017 JMPREL 0x86b800x0000000000000014 PLTREL RELA 0x0000000000000007 RELA 0x321180x0000000000000008 RELASZ 346728 (bytes) 0x0000000000000009 RELAENT 24 (bytes) 0x000000006ffffff9 RELACOUNT 59570x0000000000000015 DEBUG 0x00x0000000070000000 PPC64_GLINK 0x6457d00x0000000000000006 SYMTAB 0x2900x000000000000000b SYMENT 24 (bytes) 0x0000000000000005 STRTAB 0x103b80x000000000000000a STRSZ 132946 (bytes) 0x000000006ffffef5 GNU_HASH 0x30b100x0000000000000001 NEEDED Shared library: [libswiftCore.so] 0x0000000000000001 NEEDED Shared library: [libpthread.so.0] 0x0000000000000001 NEEDED Shared library: [libutil.so.1] 0x0000000000000001 NEEDED Shared library: [libdl.so.2] 0x0000000000000001 NEEDED Shared library: [libm.so.6] 0x0000000000000001 NEEDED Shared library: [libdispatch.so] 0x0000000000000001 NEEDED Shared library: [libswiftDispatch.so] 0x0000000000000001 NEEDED Shared library: [libBlocksRuntime.so] 0x0000000000000001 NEEDED Shared library: [libXCTest.so] 0x0000000000000001 NEEDED Shared library: [libswiftGlibc.so] 0x0000000000000001 NEEDED Shared library: [libFoundation.so] 0x0000000000000001 NEEDED Shared library: [libstdc++.so.6] 0x0000000000000001 NEEDED Shared library: [libgcc_s.so.1] 0x0000000000000001 NEEDED Shared library: [libc.so.6] 0x000000000000000c INIT 0x90d800x000000000000000d FINI 0x6472f00x000000000000001a FINI_ARRAY 0x7922200x000000000000001c FINI_ARRAYSZ 8 (bytes) 0x0000000000000019 INIT_ARRAY 0x7922280x000000000000001b INIT_ARRAYSZ 16 (bytes)
0x000000000000001d RUNPATH /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux:/home/sar/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le/src 0x000000006ffffff0 VERSYM 0x30b480x000000006ffffffe VERNEED 0x320b80x000000006fffffff VERNEEDNUM 30x0000000000000000 NULL 0x0 ]}}

Figured out that the directories mentioned in RUNPATH did not have the required library - libicui18nswift.so.61

sar@e2641fd15168:~$ ll /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux total 16004 drwxr-xr-x 3 sar sar 4096 Feb 13 07:11 ./ drwxr-xr-x 5 sar sar 4096 Feb 13 07:00 ../rwxr-xr-x 1 sar sar 8805520 Feb 13 07:11 libswiftCore.so*rwxr-xr-x 1 sar sar 84192 Feb 13 07:11 libswiftGlibc.so*rw-rr 1 sar sar 4674 Feb 13 06:57 libswiftImageInspectionShared.a-rw-rr- 1 sar sar 1100452 Feb 13 06:57 libswiftReflection.a-rwxr-xr-x 1 sar sar 688424 Feb 13 06:57 libswiftRemoteMirror.so*-rwxr-xr-x 1 sar sar 72576 Feb 13 07:11 libswiftRuntimeUnittest.so*-rwxr-xr-x 1 sar sar 71280 Feb 13 07:11 libswiftSIMDOperators.so*-rwxr-xr-x 1 sar sar 3534296 Feb 13 07:11 libswiftStdlibCollectionUnittest.so*-rwxr-xr-x 1 sar sar 237328 Feb 13 07:11 libswiftStdlibUnicodeUnittest.so*-rwxr-xr-x 1 sar sar 961320 Feb 13 07:11 libswiftStdlibUnittest.so*-rwxr-xr-x 1 sar sar 529968 Feb 13 07:11 libswiftSwiftOnoneSupport.so*-rwxr-xr-x 1 sar sar 97056 Feb 13 07:11 libswiftSwiftPrivate.so*-rwxr-xr-x 1 sar sar 81360 Feb 13 07:11 libswiftSwiftPrivateLibcExtras.so*-rwxr-xr-x 1 sar sar 82736 Feb 13 07:11 libswiftSwiftPrivateThreadExtras.so*drwxr-xr-x 4 sar sar 4096 Feb 13 09:46 powerpc64le/

However, the /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le has all the required libraries. I tried setting RUNPATH through various files which referenced RUNPATH, but it did not help much.

Finally I manually copied the libraries in the directory mentioned in the RUNPATH and it worked. The TestFoundation test suite was executed, though it failed for 4 test cases since I was running as a root user. Running as normal non root user worked and all the test cases for TestFoundation passed successfully:-

[256/257][ 99%][98.957s] Generating TestFoundation.dir/TestFoundation
[257/257][100%][99.479s] Running utility command for TestFoundation

• • • Running tests for foundation ---

    • test ---

• /usr/bin/cmake --build /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le – -j7 test
  [0/1][ 0%][0.000s] Running tests...
  Test project /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le
  Start 1: TestFoundation
  1/1 Test #​1: TestFoundation ................... Passed 53.48 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 53.49 sec

• • test finished --

    • • Finished tests for foundation ---*

Comment by Sarvesh Tamba (JIRA)

Used the 'llvm-readelf' utility to find details about the libraries linked with lldb binary and realised that the directories mentioned in RUNPATH might not have the required libraries.

Similar to reasons mentioned in my earlier issue "https://forums.swift.org/t/testfoundation-test-suite-failing-while-building-swift-toolchain/20279/6", I manually copied the libraries and it worked. The lldb test suite was executed, though it fails now for only 7 test cases:-
Failing Tests (7):
LLDB :: SwiftREPL/ErrorReturn.test
LLDB :: tools/lldb-mi/symbol/symbol-list-lines.test
lldb-Suite :: lang/swift/dwarfimporter/C/TestSwiftDWARFImporterC.py
lldb-Suite :: lang/swift/reference_storage_types/TestSwiftReferenceStorageTypes.py
lldb-Suite :: lang/swift/repl_in_c/TestSwiftReplInC.py
lldb-Suite :: lang/swift/unknown_self/TestSwiftUnknownSelf.py
lldb-Suite :: lang/swift/variables/swift/TestSwiftTypes.py

Comment by Sarvesh Tamba (JIRA)

Attached to this issue are the details of the final 7 failing lldb tests.