Comment by markus sähn (JIRA)

I can reproduce this and I see memory usages of ~5.5GB. It looks like that the memory is increasing exponential by the number of cores which are used. My example uses max 300MB when I use 1 core, but when I use 8 cores it is increasing to 5.6GB.

To reproduce it:

- Run "docker run -it --entrypoint /bin/bash swift:5.0.1-xenial"
- Run "docker stats" in another window
- Inside the container run "mkdir Bug && cd Bug && swift package init --name Bug --type executable"
- Then run: 
echo '// swift-tools-version:4.2
// The swift-tools-version declares the minimum version of Swift required to build this package.import PackageDescriptionlet package = Package(
    name: "Bug",
    dependencies: [
        .package(url: "https://github.com/lukaskubanek/OrderedDictionary.git", .exact("2.1.0")),
        .package(url: "https://github.com/PerfectlySoft/Perfect-NewRelic-linux.git", .upToNextMinor(from: "3.0.1")),
        .package(url: "https://github.com/Quick/Nimble.git", .upToNextMajor(from: "8.0.1")),
        .package(url: "https://github.com/evgenyneu/SigmaSwiftStatistics.git", .upToNextMajor(from: "9.0.2")),
        .package(url: "https://github.com/IBM-Swift/LoggerAPI.git", .upToNextMajor(from: "1.8.0")),
        .package(url: "https://github.com/typelift/SwiftCheck.git", .exact("0.12.0"))
    ],
    targets: [
        .target(
            name: "Bug",
            dependencies: ["OrderedDictionary", "PerfectNewRelic", "Nimble", "SigmaSwiftStatistics", "LoggerAPI", "SwiftCheck"]
        )
    ]
)
' > Package.swift
- Then run "swift build" and observe the output of "docker stats"

I wonder if there's a bad edge case here buffering output, if one of the dependencies has tons of warnings or something.

Comment by markus sähn (JIRA)

Any update on the ticket? It feels like nobody is having big swift projects so we are the only team with such issues?

CC @belkadan/@aciidb0mb3r I have recently heard quite a few reports of people running out of memory on CI and being killed. We really should look at this. This might also be SwiftPM spawning a very large number of swiftc processes?

Thanks papierschiff (JIRA User) for the repro instructions.

SwiftPM uses `ProcessInfo.processInfo.activeProcessorCount` to set the `num-threads` on swiftc invocations and then llbuild also has its own parallelism. I think there is a `--jobs` option to limit that.

See my attached screenshot and screen recording. I see the container taking up to about 1.6 GB of memory even with this comparatively small project. There are about 9 swift processes each having more than 200 MB resident which then adds up...

@aciidb0mb3r

$ docker run -it --rm --privileged swift:5.0.2
root@5fa18204d503:/# swift
import Foundation
Welcome to Swift version 5.0.2 (swift-5.0.2-RELEASE).
Type :help for assistance.
  1> import Foundation
  2> ProcessInfo.processInfo.activeProcessorCount
$R0: Int = 4

So it thinks it's 4 CPUs but I get 9 compilers running and -j isn't a thing in 5.0

$ swift --version
Swift version 5.0.2 (swift-5.0.2-RELEASE)
Target: x86_64-unknown-linux-gnu
johannes:~/devel/swift (jw-bench-find-string)
$ jw-docker-swift-5.0 swift build -j 1
error: unknown option -j; use --help to list available options