[SwiftPM] TargetSourcesBuilder is slow when source files are listed

It took over 12 minutes to start building in the example described at https://forums.swift.org/t/scaling-swift-package-manager-builds/38535.

$ git clone git@github.com:firebase/firebase-ios-sdk.git

$ git checkout 0a290a898

$ swift build

The workaround is in the next commit - ab7e4bad1