[SR-280] swiftc - Can't compile project with +1500 Swift files Created: 17 Dec 2015  Updated: 24 May 2018  Resolved: 19 Jan 2016

Status: Closed
Project: Swift
Component/s: Compiler

Type: Bug Priority: Medium
Reporter: Tomasz Gebarowski Assignee: Jordan Rose
Resolution: Done Votes: 0
Labels: Driver
Environment:

OS X 10.11.2, Xcode 7.2 (7C68)
Apple Swift version 2.1.1 (swiftlang-700.1.101.15 clang-700.1.81)

Affects also latest compiled version from github.


Attachments: File generate_pure_swift_class.py    
Issue Links:
Relates
relates to SR-7399 Cannot build in WMO with many/long so... Open
Radar URL: rdar://problem/23878192

 Description   

Background

I'm working on a mixed Swift/Objective-C project with a large code base, which has more than 1500 Swift files and several hundred Objective-C files. The project is written in Swift 1.2 and is currently build successfully on Xcode 6.4. I started to migrate it to Swift 2.1 and Xcode 7.2, but it seems that swiftc has a bug which does not allow for successfuly migration/compilation. Note, that the project has to support iOS 7, so I cannot use Swift dynamic frameworks and all Swift files has to be compiled in the same target.

When trying to compile such a project, compilaion fails just after successful termination of Compile Swift sources step (with swiftc), no MergeSwitfModules step is invoked and -Swift.h umbrella header is not generated. Because there is no umbrella header the project compilation fails.

It seems that swiftc is not generating umbrella header when it has more than 1500 Swift files to process. Reducing number (to about 1100) of Swift files solves the problem, same as concatenating Swift classes into one big file.

When enabling -whole-module-optimization option in swiftc (which helped me with some problems under Xcode 6.4 and MergeSwiftModules crashes), the compiler process is returning immediatelly and is not processing any Swift files.

How to reproduce?

The bug is easy to reproduce especially with -whole-module-optimization, by following these steps:

  • Run attached script (./generate_pure_swift_class.py) to generate 2500 pure swift classes (simple classes having two dummy methods)
  • Try to compile all swift files:

swiftc -num-threads 1 -g -whole-module-optimization -parseable-output *.swift

  • Compiler exits immediatelly without performing any steps and without any warning or error. Note that when setting -num-threads to 0 or not setting at all compilation seems to work.

Potential problem

I spent some time on investigating the issue with -whole-module-optimization and immediate compiler exit. After checking swiftc source code I found out that the culprit is posix_spawn invoked from Task::execute() (lib/Basic/Unix/TaskQueue.inc:142), it returns error code 7, which suggests [EFAULT] aka Path, argv, or envp point to an illegal address.

I checked both argv and envp and they contain correct entries, doesn't look like corrupted. I think that reason for posix_spawn() failure can be too long list of arguments stored in argvp (about 5900 entries, which is much bigger than original list of input arguments passed to swiftc - about 2000 args). Strangely this worked correctly in Swift 1.2.

Maybe posix_spawn is passed more arguments to a spawned process (since what was done for Swift 1.2), because some object dependency graph is passed together with input files to avoid unnecessary recompilation or something?

The situation could be solved by blocking posix_spawn (#undef HAVE_POSIX_SPAWN) and using fork, but this is more like a workaround and needs recompilation of swift.

Note: Both problems (not generating Swift umbrella header and immediate exit of swiftc in -whole-module-optimization mode) are also reproducible with Xcode 7.2, when creating new project and adding about 1.2 - 1.5k Swift files to it. Tested both on pure Swift files and Swift files inheriting from NSObject.



 Comments   
Comment by Jordan Rose [ 17 Dec 2015 ]

I guess the right answer here would be to pass a list of files down to the frontend, instead of trying to pass them all individually, or to have a mode where we pass them via stdin. We may also want to try supporting something like GCC's @file command-line syntax, but that's (a) more complicated, and (b) potentially dangerous.

Comment by Tomasz Gebarowski [ 18 Dec 2015 ]

Hi Jordan, thanks for answering! I would probably opt for using pipe/stdin to pass parameters to a spawned process. I've noticed that you opened an internal radar for that issues, any idea if that is a serious issue for you and is going to be fixed anytime soon? It's a bit blocker for our project. Setting -num-threads 0 seems to help, but only from command line. Xcode keeps overriding this value when invoking swiftc so setting

defaults write com.apple.dt.Xcode IDEBuildOperationMaxNumberOfConcurrentCompileTasks 0

doesn't help as the value is simply ignored by Xcode. Any idea how to force Xcode to use -num-threads 0 when invoking swiftc?

Comment by Jordan Rose [ 18 Dec 2015 ]

You might be able to add -num-threads 0 to your OTHER_SWIFT_FLAGS build setting for now. It's probably a coincidence that that helps, though.

I can't speak to schedule at this point, just that you're not the only one with such a big target (unless the other reports are also from your team). If someone else gets to it first, that's also great.

Comment by Tomasz Gebarowski [ 18 Dec 2015 ]

I'm the only person from my team who reported this problem, so I guess that more projects are affected.
I tried appending -num-threads 0 to OTHER SWIFT FLAGS, but unfortunately Xcode adds its -num-threads just after values
from OTHER SWIFT FLAGS and it gets overwritten. I solved the problem temporarily by writing a simple wrapper for swiftc in Python (which replaced the swiftc symlink in Xcode archive) that invokes swiftc under the hood and appends -num-threads 0 as last argument. This seems to work, at least the code compiles.

Unfortunately it fails at the linking stage. It seems that -whole-module-optimization is not working at all in Xcode 7.1 and Xcode 7.2.
There is already a radar for that:
rdar://23472326

Even the simplest Xcode project fails at linking stage when -whole-module-optimization is turned on. I will keep investigating this second problem, maybe there is some workaround for that as well.

Anyway thanks for help!

Comment by Tomasz Gebarowski [ 4 Jan 2016 ]

Small update, I managed to bypass this bug by modifying the way swiftc is invoked from Xcode. With this approach I managed to successfully compile my Xcode project with +1500 files in it (with -whole-module-optimization turned on)

The wrapper script I used is on my github account, more info can be found there as well:
https://github.com/tgebarowski/swiftc-wrapper

Comment by Jordan Rose [ 4 Jan 2016 ]

It looks like you're just silently throwing away arguments. I can't recommend that.

Comment by Jordan Rose [ 4 Jan 2016 ]

Ah, I see what it's doing. Okay, I can't call it supported, and it may break other things in Xcode, but it shouldn't be dangerous.

Comment by Tomasz Gebarowski [ 5 Jan 2016 ]

I am not dropping any arguments, but this is also not a fix, it's just a way to bypass incorrect compiler behavior. Maybe someone facing same problem will find it useful. With this workaround I managed to successfully compile,archive and run my project on both device and simulator. At least it's not blocking me and I can migrate my app to Swift 2 and wait either for (A) getting if fixed in next Xcode release or (B) dropping iOS 7 support in my app and dividing it to smaller components.

Comment by Jordan Rose [ 12 Jan 2016 ]

I've added a -filelist parameter for input files in ad945426. I'd also like to remove the O(N) arguments from

  • swiftmodule inputs to the merge-module build phase
  • /output/ files for multithreading single-frontend builds (WMO)
  • object file inputs to the linker, at least on OS X
Comment by Jordan Rose [ 13 Jan 2016 ]

Finshed the rest of the work in commits up to bbc14af.

Comment by Jordan Rose [ 13 Jan 2016 ]

...but asking Chris to review and waiting for a buildbot cycle before cherry-picking the later commits to the 2.2 branch.

Comment by Jordan Rose [ 15 Jan 2016 ]

Test fix-up in 9cb775a1.

Comment by Jordan Rose [ 19 Jan 2016 ]

Reviewed by ChrisW and cherry-picked to the 2.2 branch!

Comment by Jordan Rose [ 19 Jan 2016 ]

It should also be in the next development snapshot from the master branch, if you want to test sooner than Xcode 7.3.

Comment by Tomasz Gebarowski [ 20 Jan 2016 ]

Thanks Jordan for such a quick reaction. When I have time I will test the snapshot and let you know if the problem is fixed in the snapshot build.

Comment by Brian Ivan Gesiak [ 20 Aug 2016 ]

Any updates here, Tomasz Gebarowski? If you've been able to compile your 1500+ file Swift/Objective-C project using Xcode 7.3, then I think we can close this issue.

Comment by Tomasz Gebarowski [ 20 Aug 2016 ]

Hi Brian.

The project compilation started to work with Xcode 7.3 with -Owholemodule (where the fix made by Jordan was first introduced). Compilation with -Onone was still not working, but I think that this was another issue. So yes, I think that you can close this.

Regards,
Tomasz

Comment by Brian Ivan Gesiak [ 20 Aug 2016 ]

Cool, thanks! Post a link to the other issue you mentioned, if you can find it.

Comment by Vladislav Alexeev [ 15 Mar 2018 ]

I can confirm Xcode 9.2 still invokes swift and it passes all swift files through command line instead of using -filelist option. I understand there is support for -filelist in Swift, but how do people make Xcode work properly with large projects?

Comment by Jordan Rose [ 15 Mar 2018 ]

This bug was about communication between the swiftc driver and the subprocesses it spawns, not about any tools that need to invoke swiftc themselves. SR-4517 talks about adding general response file support to the driver, but even just acceping a filelist option would be a reasonable task on its own. (This is tracked by rdar://problem/33635183 within Apple.)

Comment by Vladislav Alexeev [ 16 Mar 2018 ]

Thanks Jordan for references and explanation!

Comment by Paolo Musolino [ 24 May 2018 ]

Is this problem fixed? On Xcode 9.2 I have the same problem

Comment by Jordan Rose [ 24 May 2018 ]

See my comment above.

Generated at Mon Jul 16 15:44:31 CDT 2018 using JIRA 7.3.4#73015-sha1:a262b3457b3605f12635df4b0a0c3dc71d631a1e.