Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-3224] Can CoreFoundation build be made to work without DEPLOYMENT_RUNTIME_SWIFT? #4499

Closed
swift-ci opened this issue Nov 16, 2016 · 19 comments

Comments

@swift-ci
Copy link
Contributor

Previous ID SR-3224
Radar None
Original Reporter copumpkin (JIRA User)
Type Improvement
Status Resolved
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s Foundation
Labels Improvement
Assignee None
Priority Medium

md5: 8c4880b91b345b3ef0aaeb9fd4d50418

Issue Description:

This feels like a weird thing to post to the swift bug tracker, but please bear with me!

I'm having some trouble understanding the scope of the CoreFoundation folder inside swift-corelibs-foundation. It seems to be strictly broader than the CF being released on opensource.apple.com, and Apple appears to have by and large given up on releasing the source to CF on that site (the 10.11 releases have been "coming soon" for over a year now). Until I looked more carefully, I had largely assumed that the CoreFoundation in swift-corelibs-foundation was intended to replace and supersede it, as a more accurate, less redacted, and "almost identical to the full CoreFoundation" version of it.

Indeed, it has some preprocessor magic that clearly appears to support both use cases, both as a statically linked library under Swift's new Foundation, and as its own thing, in the form of DEPLOYMENT_RUNTIME_SWIFT. Unfortunately, if I undefine DEPLOYMENT_RUNTIME_SWIFT in the CoreFoundation build, the build just breaks, so I'm not sure anyone is currently exercising that code path. Is that just not something anyone has gotten to yet, or is this version of CoreFoundation intended only for use with Swift, even given the preprocessor directives? If that's the case, is there a plan to resume releasing newer versions of CF to opensource.apple.com?

Anyway, with all the preface, this ticket is basically asking to make a build without DEPLOYMENT_RUNTIME_SWIFT work. I'm marking it as an improvement rather than a bug due to my uncertainty about the intended scope of the project, but feel free to change it to a bug if more appropriate.

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

Ping? This is quite painful, as I was actively compiling CF on the public opensource.apple.com releases, but now that those have stopped, I'm kind of stuck in this weird limbo.

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

desperate struggle to get some attention @phausler CodaFi (JIRA User)

@phausler
Copy link
Member

phausler commented Feb 4, 2017

So the define of DEPLOYMENT_RUNTIME_SWIFT is intended for when CoreFoundation is being built with the Swift runtime present.

To give a bit more background of what is going on: when we develop CoreFoundation in the closed source version we are using nearly the same sources as what is currently in the swift-corelibs-foundation. There are of course some differentials; primarily of course the runtime we are using is the Objective-C runtime. This means that we of course cannot use swift specific runtime features. For example when a bridged call-out to NSString is made from CF to do the CFStringRef to NSString subclass bridging that happens for Swift, that is of course not the same as when it happens for Objective-C. The opposite is also true; that a bridging that happens for the Objective-C runtime of course isn't going to work for a Swift based runtime.

Take a look at this line:
https://github.com/apple/swift-corelibs-foundation/blob/master/CoreFoundation/String.subproj/CFString.c#L2124

CF_SWIFT_FUNCDISPATCHV(__kCFStringTypeID, const char *, (CFSwiftRef)str, NSString._fastCStringContents, true);

versus

CF_OBJC_FUNCDISPATCHV(__kCFStringTypeID, const char *, (NSString *)str, _fastCStringContents:true);

The CF_SWIFT_FUNCDISPATCHV macro expands out to

if (CF_IS_SWIFT(__kCFStringTypeID, str)) {
    return (const char *)__CFSwiftBridge.NSString._fastCStringContents((CFSwiftRef)str, true);
}

Now if you are looking to build CF without either being supported, both of the macros CF_SWIFT_FUNCDISPATCHV and CF_OBJC_FUNCDISPATCHV will be defined away to nothing. Since if you have no additional runtime support there of course cannot be any subclasses.

To build CF for linux outside of the context of Swift you should follow these rough steps:

checkout the swift-corelibs-foundation repo
cd to the CoreFoundation directory
run ../configure
then run ninja

There are a few additional knobs and dials to tweak (dispatch and such) but that should build a static library of libCoreFoundation.a (some assembly might be required to export the headers properly still iirc).

Note of course this is will have a different structural layout and implementation that when the Swift runtime and it CANNOT be used in conjunction with swift-corelibs-foundation's Swift.

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

Thanks for the information! The context here is that I'm trying to build it for macOS, not to replace the system CoreFoundation, but in a manner independent from the system one. So I'm not actually trying to build it for Linux, and thus get a lot of the Mac-specific preprocessor behavior.

I'm mostly trying to get back to the rough behavior we had with https://opensource.apple.com/source/CF/, which didn't have the Objective-C integration or the Swift integration and worked fine like that on both Linux and macOS. Unfortunately it hasn't been updated since 10.10, presumably because the swift version exists. Unfortunately it doesn't really replace the old CF, and it doesn't seem like anything much does. Am I just screwed?

@phausler
Copy link
Member

phausler commented Feb 4, 2017

I think we can probably re-purpose this bug to a request to improve the README.md to include more detailed instructions on how to build libCoreFoundation.a as a stand-alone thing.

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

That would be great, but keep in mind that it's not just about the standalone .a file. I have that building just fine right now, but unfortunately it still links to things like swift_retain which I don't (necessarily) have available. The .a file builds just fine but then complains when linking it later. I've been working on building it as a dylib but that exposes the dependency on the swift runtime much more explicitly.

@phausler
Copy link
Member

phausler commented Feb 4, 2017

that would definitely be a bug that it is calling swift_retain. I will take a peek at it on Monday.

From a brief scan of the sources it looks like all instances of swift_retain are gated under DEPLOYMENT_RUNTIME_SWIFT macros. Which should not be defined when using that build procedure.

What are the build steps you are using?
Do you have a build log and linker failure log?

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

Cool, thanks!

Here's some more detail:

  1. I'm building on macOS, with a custom toolchain (think e.g., MacPorts, fink, or opendarwin; in this case it's Nix)

  2. This line tells the build to do a standalone build with the Swift runtime. If I take it out, I get a bunch of failures because I believe the code assumes that if you're on macOS that DEPLOYMENT_RUNTIME_SWIFT is set.

  3. I've been building this by adjusting CoreFoundation/build.py to use DynamicLibrary instead of StaticLibrary, and then injecting several linker-specific flags to make the build work. I roughly need to set cf.LDFLAGS to include -lxml2 -licuX, and set up a bunch of the unicode/character set bitmap symbols.

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

Sure, here's my build log. I'll try to put up a simple recipe to get similar results in a bit.

[1/164] Cp Stream.subproj/CFStream.h
[2/164] Cp Base.subproj/SwiftRuntime/TargetConditionals.h
[3/164] Cp String.subproj/CFStringEncodingExt.h
[4/164] Cp Base.subproj/SwiftRuntime/CoreFoundation.h
[5/164] Cp Collections.subproj/CFBinaryHeap.h
[6/164] Cp RunLoop.subproj/CFMessagePort.h
[7/164] Cp PlugIn.subproj/CFBundle.h
[8/164] Cp Locale.subproj/CFCalendar.h
[9/164] Cp Collections.subproj/CFBitVector.h
[10/164] Cp Base.subproj/CFAvailability.h
[11/164] Cp Collections.subproj/CFTree.h
[12/164] Cp NumberDate.subproj/CFTimeZone.h
[13/164] Cp Error.subproj/CFError.h
[14/164] Cp Collections.subproj/CFBag.h
[15/164] Cp Parsing.subproj/CFXMLParser.h
[16/164] Cp NumberDate.subproj/CFDate.h
[17/164] Cp PlugIn.subproj/CFPlugIn.h
[18/164] Cp PlugIn.subproj/CFPlugInCOM.h
[19/164] Cp String.subproj/CFString.h
[20/164] Cp Collections.subproj/CFSet.h
[21/164] Cp Base.subproj/CFUUID.h
[22/164] Cp AppServices.subproj/CFUserNotification.h
[23/164] Cp Collections.subproj/CFDictionary.h
[24/164] Cp Base.subproj/CFByteOrder.h
[25/164] Cp Base.subproj/CFBase.h
[26/164] Cp Preferences.subproj/CFPreferences.h
[27/164] Cp Locale.subproj/CFLocale.h
[28/164] Cp URL.subproj/CFURLAccess.h
[29/164] Cp RunLoop.subproj/CFSocket.h
[30/164] Cp Parsing.subproj/CFPropertyList.h
[31/164] Cp Collections.subproj/CFArray.h
[32/164] Cp RunLoop.subproj/CFRunLoop.h
[33/164] Cp RunLoop.subproj/CFMachPort.h
[34/164] Cp Locale.subproj/CFDateFormatter.h
[35/164] Cp PlugIn.subproj/CFBundle_BinaryTypes.h
[36/164] Cp Base.subproj/CFUtilities.h
[37/164] Cp Parsing.subproj/CFXMLNode.h
[38/164] Cp URL.subproj/CFURLComponents.h
[39/164] Cp Locale.subproj/CFNumberFormatter.h
[40/164] Cp String.subproj/CFCharacterSet.h
[41/164] Cp Base.subproj/ForFoundationOnly.h
[42/164] Cp NumberDate.subproj/CFNumber.h
[43/164] Cp Collections.subproj/CFData.h
[44/164] Cp String.subproj/CFAttributedString.h
[45/164] Cp Base.subproj/CFAsmMacros.h
[46/164] Cp Base.subproj/ForSwiftFoundationOnly.h
[47/164] Cp Error.subproj/CFError_Private.h
[48/164] Cp URL.subproj/CFURLPriv.h
[49/164] Cp String.subproj/CFBurstTrie.h
[50/164] Cp Base.subproj/CFLogUtilities.h
[51/164] Cp PlugIn.subproj/CFBundlePriv.h
[52/164] Cp StringEncodings.subproj/CFStringEncodingConverter.h
[53/164] Cp Stream.subproj/CFStreamAbstract.h
[54/164] Cp Base.subproj/CFInternal.h
[55/164] Cp Parsing.subproj/CFXMLInterface.h
[56/164] Cp Parsing.subproj/CFXMLInputStream.h
[57/164] Cp PlugIn.subproj/CFPlugIn_Factory.h
[58/164] Cp String.subproj/CFStringLocalizedFormattingInternal.h
[59/164] Cp PlugIn.subproj/CFBundle_Internal.h
[60/164] Cp StringEncodings.subproj/CFStringEncodingConverterPriv.h
[61/164] Cp Stream.subproj/CFStreamInternal.h
[62/164] Cp Collections.subproj/CFBasicHash.h
[63/164] Cp StringEncodings.subproj/CFStringEncodingDatabase.h
[64/164] Cp StringEncodings.subproj/CFUnicodeDecomposition.h
[65/164] Cp Base.subproj/CFPriv.h
[66/164] Cp Locale.subproj/CFICULogging.h
[67/164] Cp Locale.subproj/CFLocaleInternal.h
[68/164] Cp URL.subproj/CFURL.h
[69/164] Cp StringEncodings.subproj/CFUnicodePrecomposition.h
[70/164] Cp NumberDate.subproj/CFBigNumber.h
[71/164] Cp StringEncodings.subproj/CFUniCharPriv.h
[72/164] Cp URL.subproj/CFURL.inc.h
[73/164] Cp StringEncodings.subproj/CFUniChar.h
[74/164] Cp Collections.subproj/CFStorage.h
[75/164] Cp StringEncodings.subproj/CFStringEncodingConverterExt.h
[76/164] Cp String.subproj/CFCharacterSetPriv.h
[77/164] Cp Base.subproj/CFRuntime.h
[78/164] Cp String.subproj/CFStringDefaultEncoding.h
[79/164] Cp StringEncodings.subproj/CFICUConverters.h
[80/164] Cp Stream.subproj/CFStreamPriv.h
[81/164] Cp String.subproj/CFRegularExpression.h
[82/164] Cp String.subproj/CFRunArray.h
[83/164] Cp Base.subproj/module.modulemap
[84/164] CompileC: Base.subproj/CFSortFunctions.c
[85/164] CompileC: Base.subproj/CFPlatform.c
[86/164] CompileC: Collections.subproj/CFDictionary.c
[87/164] CompileC: Base.subproj/CFBase.c
[88/164] CompileC: Collections.subproj/CFBitVector.c
[89/164] CompileC: Base.subproj/CFFileUtilities.c
[90/164] CompileC: Base.subproj/CFSystemDirectories.c
[91/164] CompileC: Base.subproj/CFRuntime.c
[92/164] CompileC: Base.subproj/CFUUID.c
[93/164] CompileC: Base.subproj/CFUtilities.c
[94/164] CompileC: Collections.subproj/CFBinaryHeap.c
[95/164] CompileC: Collections.subproj/CFArray.c
[96/164] CompileC: Collections.subproj/CFBag.c
[97/164] CompileC: Collections.subproj/CFSet.c
[98/164] CompileC: Collections.subproj/CFData.c
[99/164] CompileC: Collections.subproj/CFTree.c
[100/164] CompileC: Error.subproj/CFError.c
[101/164] CompileC: Collections.subproj/CFStorage.c
[102/164] CompileC: Locale.subproj/CFCalendar.c
[103/164] CompileC: RunLoop.subproj/CFSocket.c
[104/164] CompileC: Locale.subproj/CFLocale.c
[105/164] CompileC: Locale.subproj/CFLocaleKeys.c
[106/164] CompileC: Locale.subproj/CFLocaleIdentifier.c
[107/164] CompileC: NumberDate.subproj/CFDate.c
[108/164] CompileC: Locale.subproj/CFDateFormatter.c
[109/164] CompileC: NumberDate.subproj/CFBigNumber.c
[110/164] CompileC: Locale.subproj/CFNumberFormatter.c
[111/164] CompileC: NumberDate.subproj/CFTimeZone.c
[112/164] CompileC: NumberDate.subproj/CFNumber.c
[113/164] CompileC: Parsing.subproj/CFOldStylePList.c
[114/164] CompileC: Collections.subproj/CFBasicHash.c
[115/164] CompileC: Parsing.subproj/CFXMLNode.c
[116/164] CompileC: Parsing.subproj/CFXMLTree.c
[117/164] CompileC: StringEncodings.subproj/CFStringEncodingConverter.c
[118/164] CompileC: Parsing.subproj/CFXMLInputStream.c
[119/164] CompileC: Parsing.subproj/CFBinaryPList.c
[120/164] CompileC: Parsing.subproj/CFXMLInterface.c
[121/164] CompileC: PlugIn.subproj/CFBundle_Binary.c
[122/164] CompileC: PlugIn.subproj/CFBundle_InfoPlist.c
[123/164] CompileC: PlugIn.subproj/CFBundle.c
[124/164] CompileC: PlugIn.subproj/CFBundle_Locale.c
[125/164] CompileC: Parsing.subproj/CFPropertyList.c
[126/164] CompileC: PlugIn.subproj/CFBundle_Strings.c
[127/164] CompileC: PlugIn.subproj/CFPlugIn.c
[128/164] CompileC: PlugIn.subproj/CFBundle_Resources.c
[129/164] CompileC: Parsing.subproj/CFXMLParser.c
[130/164] CompileC: PlugIn.subproj/CFPlugIn_Factory.c
[131/164] CompileC: PlugIn.subproj/CFPlugIn_Instance.c
[132/164] CompileC: PlugIn.subproj/CFPlugIn_PlugIn.c
[133/164] CompileC: Preferences.subproj/CFPreferences.c
[134/164] CompileC: Stream.subproj/CFConcreteStreams.c
[135/164] CompileC: Preferences.subproj/CFApplicationPreferences.c
[136/164] CompileC: PlugIn.subproj/CFBundle_Grok.c
[137/164] CompileC: Stream.subproj/CFSocketStream.c
[138/164] CompileC: StringEncodings.subproj/CFPlatformConverters.c
[139/164] CompileC: StringEncodings.subproj/CFICUConverters.c
[140/164] CompileC: Stream.subproj/CFStream.c
[141/164] CompileC: RunLoop.subproj/CFRunLoop.c
[142/164] CompileC: String.subproj/CFBurstTrie.c
[143/164] CompileC: String.subproj/CFStringScanner.c
[144/164] CompileC: String.subproj/CFStringEncodings.c
[145/164] CompileC: String.subproj/CFStringTransform.c
[146/164] CompileC: StringEncodings.subproj/CFStringEncodingDatabase.c
[147/164] CompileC: StringEncodings.subproj/CFBuiltinConverters.c
[148/164] CompileC: String.subproj/CFStringUtilities.c
[149/164] CompileC: StringEncodings.subproj/CFUnicodeDecomposition.c
[150/164] CompileC: String.subproj/CFRunArray.c
[151/164] CompileC: StringEncodings.subproj/CFUniChar.c
[152/164] CompileC: StringEncodings.subproj/CFUnicodePrecomposition.c
[153/164] CompileC: URL.subproj/CFURLComponents_URIParser.c
[154/164] Assemble: String.subproj/CFCharacterSetData.S
[155/164] CompileC: String.subproj/CFCharacterSet.c
[156/164] Assemble: String.subproj/CFUnicodeData.S
[157/164] Assemble: String.subproj/CFUniCharPropertyDatabase.S
[158/164] CompileC: String.subproj/CFRegularExpression.c
[159/164] CompileC: URL.subproj/CFURLAccess.c
[160/164] CompileC: String.subproj/CFAttributedString.c
[161/164] CompileC: URL.subproj/CFURLComponents.c
[162/164] CompileC: URL.subproj/CFURL.c
[163/164] CompileC: String.subproj/CFString.c
[164/164] Link: ../Build/CoreFoundation/libCoreFoundation.dylib
FAILED: ../Build/CoreFoundation/libCoreFoundation.dylib 
mkdir -p `dirname ../Build/CoreFoundation/libCoreFoundation.dylib`; clang --target=x86_64-apple-darwin  -L/lib/swift/macosx --sysroot=unused -L../bootstrap/x86_64-apple-darwin/usr/lib -shared -lxml2 -licui18n -licuuc -twolevel_namespace -Wl,-alias_list,Base.subproj/DarwinSymbolAliases -sectcreate __UNICODE __csbitmaps CharacterSets/CFCharacterSetBitmaps.bitmap -sectcreate __UNICODE __properties CharacterSets/CFUniCharPropertyDatabase.data -sectcreate __UNICODE __data CharacterSets/CFUnicodeData-L.mapping -segprot __UNICODE r r    ../Build/CoreFoundation/Base.subproj/CFBase.c.o ../Build/CoreFoundation/Base.subproj/CFFileUtilities.c.o ../Build/CoreFoundation/Base.subproj/CFPlatform.c.o ../Build/CoreFoundation/Base.subproj/CFRuntime.c.o ../Build/CoreFoundation/Base.subproj/CFSortFunctions.c.o ../Build/CoreFoundation/Base.subproj/CFSystemDirectories.c.o ../Build/CoreFoundation/Base.subproj/CFUtilities.c.o ../Build/CoreFoundation/Base.subproj/CFUUID.c.o ../Build/CoreFoundation/Collections.subproj/CFArray.c.o ../Build/CoreFoundation/Collections.subproj/CFBag.c.o ../Build/CoreFoundation/Collections.subproj/CFBasicHash.c.o ../Build/CoreFoundation/Collections.subproj/CFBinaryHeap.c.o ../Build/CoreFoundation/Collections.subproj/CFBitVector.c.o ../Build/CoreFoundation/Collections.subproj/CFData.c.o ../Build/CoreFoundation/Collections.subproj/CFDictionary.c.o ../Build/CoreFoundation/Collections.subproj/CFSet.c.o ../Build/CoreFoundation/Collections.subproj/CFStorage.c.o ../Build/CoreFoundation/Collections.subproj/CFTree.c.o ../Build/CoreFoundation/Error.subproj/CFError.c.o ../Build/CoreFoundation/Locale.subproj/CFCalendar.c.o ../Build/CoreFoundation/Locale.subproj/CFDateFormatter.c.o ../Build/CoreFoundation/Locale.subproj/CFLocale.c.o ../Build/CoreFoundation/Locale.subproj/CFLocaleIdentifier.c.o ../Build/CoreFoundation/Locale.subproj/CFLocaleKeys.c.o ../Build/CoreFoundation/Locale.subproj/CFNumberFormatter.c.o ../Build/CoreFoundation/NumberDate.subproj/CFBigNumber.c.o ../Build/CoreFoundation/NumberDate.subproj/CFDate.c.o ../Build/CoreFoundation/NumberDate.subproj/CFNumber.c.o ../Build/CoreFoundation/NumberDate.subproj/CFTimeZone.c.o ../Build/CoreFoundation/Parsing.subproj/CFBinaryPList.c.o ../Build/CoreFoundation/Parsing.subproj/CFOldStylePList.c.o ../Build/CoreFoundation/Parsing.subproj/CFPropertyList.c.o ../Build/CoreFoundation/Parsing.subproj/CFXMLInputStream.c.o ../Build/CoreFoundation/Parsing.subproj/CFXMLNode.c.o ../Build/CoreFoundation/Parsing.subproj/CFXMLParser.c.o ../Build/CoreFoundation/Parsing.subproj/CFXMLTree.c.o ../Build/CoreFoundation/Parsing.subproj/CFXMLInterface.c.o ../Build/CoreFoundation/PlugIn.subproj/CFBundle.c.o ../Build/CoreFoundation/PlugIn.subproj/CFBundle_Binary.c.o ../Build/CoreFoundation/PlugIn.subproj/CFBundle_Grok.c.o ../Build/CoreFoundation/PlugIn.subproj/CFBundle_InfoPlist.c.o ../Build/CoreFoundation/PlugIn.subproj/CFBundle_Locale.c.o ../Build/CoreFoundation/PlugIn.subproj/CFBundle_Resources.c.o ../Build/CoreFoundation/PlugIn.subproj/CFBundle_Strings.c.o ../Build/CoreFoundation/PlugIn.subproj/CFPlugIn.c.o ../Build/CoreFoundation/PlugIn.subproj/CFPlugIn_Factory.c.o ../Build/CoreFoundation/PlugIn.subproj/CFPlugIn_Instance.c.o ../Build/CoreFoundation/PlugIn.subproj/CFPlugIn_PlugIn.c.o ../Build/CoreFoundation/Preferences.subproj/CFApplicationPreferences.c.o ../Build/CoreFoundation/Preferences.subproj/CFPreferences.c.o ../Build/CoreFoundation/RunLoop.subproj/CFRunLoop.c.o ../Build/CoreFoundation/RunLoop.subproj/CFSocket.c.o ../Build/CoreFoundation/Stream.subproj/CFConcreteStreams.c.o ../Build/CoreFoundation/Stream.subproj/CFSocketStream.c.o ../Build/CoreFoundation/Stream.subproj/CFStream.c.o ../Build/CoreFoundation/String.subproj/CFBurstTrie.c.o ../Build/CoreFoundation/String.subproj/CFCharacterSet.c.o ../Build/CoreFoundation/String.subproj/CFString.c.o ../Build/CoreFoundation/String.subproj/CFStringEncodings.c.o ../Build/CoreFoundation/String.subproj/CFStringScanner.c.o ../Build/CoreFoundation/String.subproj/CFStringUtilities.c.o ../Build/CoreFoundation/String.subproj/CFStringTransform.c.o ../Build/CoreFoundation/StringEncodings.subproj/CFBuiltinConverters.c.o ../Build/CoreFoundation/StringEncodings.subproj/CFICUConverters.c.o ../Build/CoreFoundation/StringEncodings.subproj/CFPlatformConverters.c.o ../Build/CoreFoundation/StringEncodings.subproj/CFStringEncodingConverter.c.o ../Build/CoreFoundation/StringEncodings.subproj/CFStringEncodingDatabase.c.o ../Build/CoreFoundation/StringEncodings.subproj/CFUniChar.c.o ../Build/CoreFoundation/StringEncodings.subproj/CFUnicodeDecomposition.c.o ../Build/CoreFoundation/StringEncodings.subproj/CFUnicodePrecomposition.c.o ../Build/CoreFoundation/URL.subproj/CFURL.c.o ../Build/CoreFoundation/URL.subproj/CFURLAccess.c.o ../Build/CoreFoundation/URL.subproj/CFURLComponents.c.o ../Build/CoreFoundation/URL.subproj/CFURLComponents_URIParser.c.o ../Build/CoreFoundation/String.subproj/CFCharacterSetData.S.o ../Build/CoreFoundation/String.subproj/CFUnicodeData.S.o ../Build/CoreFoundation/String.subproj/CFUniCharPropertyDatabase.S.o ../Build/CoreFoundation/String.subproj/CFRegularExpression.c.o ../Build/CoreFoundation/String.subproj/CFAttributedString.c.o ../Build/CoreFoundation/String.subproj/CFRunArray.c.o  -o ../Build/CoreFoundation/libCoreFoundation.dylib
ld: warning: directory not found for option '-L../bootstrap/x86_64-apple-darwin/usr/lib'
ld: warning: undefined base symbol '__TMC15SwiftFoundation19_NSCFConstantString' for alias '___CFConstantStringClassReference'
Undefined symbols for architecture x86_64:
  "__TF15SwiftFoundation19__CFInitializeSwiftFT_T_", referenced from:
      ___CFInitialize in CFRuntime.c.o
  "__TF15SwiftFoundation21__CFSwiftGetBaseClassFT_PMPs9AnyObject_", referenced from:
      ___CFInitialize in CFRuntime.c.o
  "___CFConstantStringClassReference", referenced from:
      ___CFAllocatorCopyDescription.str in CFBase.c.o
      ___CFNullCopyFormattingDescription.str in CFBase.c.o
      ___CFNullCopyDescription.str in CFBase.c.o
      __CFGetSlashStr.str in CFFileUtilities.c.o
      __CFAppendPathExtension2.str in CFFileUtilities.c.o
      _CFCopyUserName.str in CFPlatform.c.o
      ___CFConstantStringClassReferencePtr in CFRuntime.c.o
      ...
     (maybe you meant: ___CFConstantStringClassReferencePtr)
  "_swift_allocObject", referenced from:
      __CFRuntimeCreateInstance in CFRuntime.c.o
  "_swift_release", referenced from:
      __CFThreadSpecificDestructor in CFPlatform.c.o
      _CFRelease in CFRuntime.c.o
      ___CFTypeCollectionRelease in CFRuntime.c.o
      __CFSwiftRelease in CFRuntime.c.o
      ___CFFinalizeRunLoop in CFRunLoop.c.o
  "_swift_retain", referenced from:
      __CThreadSpecificSet in CFPlatform.c.o
      _CFRetain in CFRuntime.c.o
      ___CFTypeCollectionRetain in CFRuntime.c.o
      _CFMakeUncollectable in CFRuntime.c.o
      _objc_retainAutoreleasedReturnValue in CFRuntime.c.o
      __CFSwiftRetain in CFRuntime.c.o
ld: symbol(s) not found for architecture x86_64
clang-3.7: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
builder for ‘/nix/store/ksvhnr6w3bl2xphhf2101xfks3gwsnwn-swift-corefoundation.drvfailed with exit code 1

And this is the code I added to the build.py after switching it to a DynamicLibrary:

    cf.LDFLAGS = "-lxml2 -licui18n -licuuc -twolevel_namespace"
    cf.LDFLAGS += " -Wl,-alias_list,Base.subproj/DarwinSymbolAliases"
    cf.LDFLAGS += " -sectcreate __UNICODE __csbitmaps CharacterSets/CFCharacterSetBitmaps.bitmap"
    cf.LDFLAGS += " -sectcreate __UNICODE __properties CharacterSets/CFUniCharPropertyDatabase.data"
    cf.LDFLAGS += " -sectcreate __UNICODE __data CharacterSets/CFUnicodeData-L.mapping"
    cf.LDFLAGS += " -segprot __UNICODE r r "

@phausler
Copy link
Member

phausler commented Feb 4, 2017

Building for Mac OS X is going to be something that won't work that great, I would HIGHLY suggest to just use the real CoreFoundation. The swift-corelibs-foundation version is always going to be behind our active development and would be missing security fixes or general crash fixes or performance improvements. Moreover it may have some strange behavior when it comes to linkages or building with it in general.

The closest you could do is perhaps build from the Xcode target (with some modifications) but I don't see much of a potential of it being useful/safe beyond perhaps just for an educational experiment.

That line that you referenced is incorrect and should be removed.

Per building a dynamic library; I have been tinkering with the idea of being able to produce both kinds even for the swift versions too. (again something I will take a bit more of a look at when I get back into work)

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

I recognize the downsides to not using the system one, but there are a few reasons we want to build our own. I've already experienced the pleasures of linking against both, but it generally works fine. The goal here is strong determinism and in some restricted senses, to build a relative of the old OpenDarwin/PureDarwin. We're doing our best (even though it's remarkably, unbelievably painful) to build Apple open source releases. CF used to be one of the more pleasant ones to build 🙂 For example, we've been running tons of code on top of the CF built from 10.9, even under macOS, so we were quite sad to see it stop being released. Or if nothing else it's been "coming soon" for a couple of years now 🙁

So I'm not sure about your point about the -DDEPLOYMENT_RUNTIME_SWIFT line being incorrect: I've tried removing it, but the build just breaks horribly on macOS. I'd be quite happy to get the Linux-like runtime behavior on Darwin, but I can't even get the build to get that far.

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

Is the plan to eventually replace the underlying runtime for CoreFoundation with swift, rather than today's one based on libobjc?

@phausler
Copy link
Member

phausler commented Feb 4, 2017

The opensource.apple.com link for CF really should be updated to point to the GitHub repo. That has been an outstanding task assigned for a while now unfortunately. However I don't see the advantage of building CF for Mac when the real McCoy is on the system already. They are cut from the same sources. The only differential is that the CF on the system interoperates with objc and has a few extra things like CFUserNotification or CFPreferences or CFNotification etc.

I think if your goal is to produce code that can be built for both Linux and Mac, on Linux hosts use the libCoreFoundtion produced from the swift-corelibs-foundation repo and on Mac hosts use the one already provided by the system. If that approach doesn't work we should fix the Linux hosts to be more on par with the Mac counterpart.

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

Among other things, we're trying to boot a home-built xnu from the public sources, and distribute it. We can't really distribute Apple's proprietary built CoreFoundation.framework.

So in some ways it's a legally necessary thing; just because we're running on Darwin doesn't mean that we actually have a CoreFoundation we can use. But even on a "proper" official macOS, we keep exact reproducible builds and can only guarantee those if we build this sort of thing.

Anyway, as I said, we've (in Nix) been building and actively using the open source CF for two or three years now, so we're well acquainted with both the pros and cons of doing so. It's honestly been pretty good so far. The main driver for me to update is that a whole bunch of other projects have started using CF nullability macros that didn't exist back in the public 10.9 and 10.10 releases, so that's what's pushed me to looking into updating to 10.11+, which pushed me over here.

It might not make much sense 🙂, but it ends up being quite a beautiful thing to build up the entire toolchain and runtime environment (including CF) from scratch.

@phausler
Copy link
Member

phausler commented Feb 4, 2017

Ah ha! ok that makes a ton more sense, I was thinking you were wanting to deploy this on Mac OS X hosts.

Ok, I think what we should do is figure out a way to identify this as a port to a new os (which happens to be a Darwin flavored BSD). So it might be good to understand what is available in this environment and what is not. Is the (Apple) objc runtime available? Is libdispatch (a.k.a swift-corelibs-libdispatch) available? what about mach (presuming so since it is xnu based).

If let say you are attempting to use GNUStep; that probably won't interoperate well with CoreFoundation. But other efforts out there for Foundation might. If that is a goal then the #defines as I was illustrating before will of course be a task for development for you to complete. (but not impossible)

I definitely would claim that most likely TARGET_OS_MAC etc should NOT be defined for your platform, and TARGET_OS_LINUX shouldn't be either, perhaps TARGET_OS_XNU? We will likely also need to have a DEPLOYMENT_TARGET macro definition as well (but it should fall somewhere in between DEPLOYMENT_TARGET_LINUX and DEPLOYMENT_TARGET_MAC as per usage wise goes).

Most of those changes I will have to rely on external contributors to fill in the gaps since my job doesn't really entail supporting those platforms. Similarly for testing and such again the community will have to verify things. But I can take a look to see what we can do to make a laundry list of TODOs and I will take a look on Monday to see if I can suggest points at tweaking the build scripts a bit since it does indicate one of the platforms we are actively developing support for is not working as expected.

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

Yeah, your idea to treat it as a separate platform sounds best, and I definitely don't expect any official support for an unusual platform like this 🙂 the unusual thing is that our platform does currently run on macOS, but we're in the process of trying to put together something that will boot xnu and run autonomously (mostly for the benefit of running headless build/CI servers for macOS binaries, amusingly 🙂).

TARGET_OS_DARWIN? Anyway, I've built the public objc4 sources in the past but never really used the output, so I don't know how well it works. I'll need to look into libdispatch (doesn't that integrate with XPC, which is all closed source?) and yes, xnu is full of mach so we definitely have that.

@phausler
Copy link
Member

phausler commented Feb 4, 2017

TARGET_OS_DARWIN would perhaps be a bit confusing since we have to maintain the CF sources across both closed and open source contexts.

libdispatch does not have XPC (that is a totally different library and non of the open source CF uses it). Technically CF can be built without libdispatch but I would really suggest to get that since it will make it much easier to deal with. There is the emulation of dispatch_once (which now works more similarly to the real dispatch_once), but I would much rather let the wizards over on the dispatch team be responsible for that. The one we have is just a work-around and I would like to eventually remove it since it adds to our #ifdef maze.

Do you have docker or vm images for this beast?

@swift-ci
Copy link
Contributor Author

swift-ci commented Feb 4, 2017

Comment by Dan Peebles (JIRA)

Fair enough about Darwin 🙂 Good to hear about XPC. As much as I'd love to see the source of that one (because launchd used to be public but then got moved into XPC and is thus no longer public)

Anyway, nope, no images, still in the process of figuring out how to build it unfortunately 🙂

The past couple of years have been setting up a lot of the ecosystem to run (almost) completely independently of the host macOS. So today we basically have something that behaves like a MacPorts/fink/homebrew but (save for dyld and libSystem) is completely independent of the host. That means we have our own compiler toolchain that doesn't touch Xcode or the command-line tools and don't use any of the BSD userland binaries that ship with macOS, and is built entirely by combining non-Apple open source projects with as much of opensource.apple.com as we found ourselves needing and able to use. If you're curious, you can try it out at https://nixos.org/nix (because it's independent of the host, it's easy to get rid of and lives entirely under /nix).

Anyway, I'll definitely look into getting libdispatch to build. We currently have a fake build of it in Nix that only extracts the headers from opensource.apple.com, but we don't bother with the actual build. Who knows what gremlins I'll encounter when I try to build it, but we'll see![]( Thanks for the help/advice)

@swift-ci
Copy link
Contributor Author

Comment by Dan Peebles (JIRA)

This actually seems to work now. Not sure who to thank, but thank whoever it is 🙂

cc @parkera @millenomi for a glimpse into my past efforts here

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 5, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants