Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-29] Support custom directory layouts #5339

Closed
krzyzanowskim opened this issue Dec 4, 2015 · 28 comments
Closed

[SR-29] Support custom directory layouts #5339

krzyzanowskim opened this issue Dec 4, 2015 · 28 comments

Comments

@krzyzanowskim
Copy link
Contributor

Previous ID SR-29
Radar rdar://problem/28039974
Original Reporter @krzyzanowskim
Type New Feature
Status Resolved
Resolution Done

Attachment: Download

Additional Detail from JIRA
Votes 3
Component/s Package Manager
Labels New Feature
Assignee None
Priority Medium

md5: e62834d3875d27814a7aba76facdedb5

relates to:

  • SR-3405 the package has an unsupported layout

Issue Description:

The Swift Package Manager relies on folder names very much. I found it impossible to create Package with folders layout not blessed by Swift Package Manager. This is issue for current project where Xcode has it's own layout of directories - incompatible with the approach of Swift Package Manager.

I tried create Package and failed. This is what I have build: https://github.com/krzyzanowskim/CryptoSwift/tree/develop/SwiftPackageManager it builds but I can't use it anywhere because it complain about folders.

and yes, I hate it 😉

@lilyball
Copy link
Mannequin

lilyball mannequin commented Dec 4, 2015

Xcode does not care about the folder structure on disk. Many (probably most) Xcode projects put all source files at the top level (or inside a single top-level folder) even when they have a well-defined file groupings in the project. You should be able to take an existing project and change the folder structure to match what Swift Package Manager expects without causing a problem for Xcode (although it can be a bit awkward to update your project when moving a lot of files around on disk).

@krzyzanowskim
Copy link
Contributor Author

Depends on project change may be quite big. Bear in mind that SPM compile subfolders as separate modules, it's not very common to have Sources organised in Submodules this way - this affect all "imports" and is incompatible with Xcode build system - unless developer create separate targets. Of course one can say that SPM is for Linux only, it's just not highlighted for other developers.

@mxcl
Copy link
Contributor

mxcl commented Dec 4, 2015

We intend to allow you to specify custom targets, so this ticket is now the ticket that follows that work.

The convention approach is meant to speed and ease development, but not prohibit other ways of working.

@ddunbar
Copy link
Member

ddunbar commented Dec 4, 2015

There are several things going on here.

The first thing is that the Package.swift has to be at the top-level in order for it to be recognized as a package when cloning as a dependency (we don't look anywhere else for the package manifest file). I tweaked your existing solution to do this here:
https://github.com/ddunbar/CryptoSwift
(on develop branch).

The second thing is that you do currently have to tag a version for it to be found by downstream packages. I tagged "0.1.2" although obviously this isn't the right thing to do from a develop branch. We are still figuring out how to support development versioning from dependencies...

The last thing is that even if you do all this, and then write a client like this:

import PackageDescription

let package = Package(
name: "CryptoSwiftTest",
dependencies: [
.Package(url: "https://github.com/ddunbar/CryptoSwift.git", majorVersion: 0)
]
)

it doesn't work because we don't currently pull down all of the tags, and the 0.1.2 tag won't be pulled by default because it is on an ancillary branch and not reachable from master.

You can probably get things working for experimental purposes by creating a separate repo that had the tag reachable from master, but we will work on resolving each of these issues...

@mxcl
Copy link
Contributor

mxcl commented Dec 4, 2015

I have made some fixes, though a few more are required:

1. Ensure we fetch all tags: this ensures we fetch `0.1.2` even though it is not connected to the cloned branch
2. It is valid for a Package.swift to not exist at HEAD: this ensures that the fetch process completes even if HEAD does not have a Package.swift provided the tag that is picked does.

@mxcl
Copy link
Contributor

mxcl commented Dec 4, 2015

Success (with pending patch):

Works with Sources subdirectory though it will also work without that now. However it tries to build the tests and fails. Which is a separate issue. But we should look into some solution, eg: allow a custom root sources to be specified, or, to allow exclusions, or to consider directories with `Tests` suffixes to be test directories.

@mxcl
Copy link
Contributor

mxcl commented Dec 4, 2015

K, against HEAD the following Package.swift works for me:

import PackageDescription

let package = Package(
   dependencies: [
       .Package(url: "https://github.com/ddunbar/CryptoSwift.git", majorVersion: 0)
   ]
)

I would like to propose the following:

import PackageDescription

let package = Package(
   sourceRoot: "CryptoSwift",
   dependencies: [
       .Package(url: "https://github.com/ddunbar/CryptoSwift.git", majorVersion: 0)
   ]
)

As a first step towards allowing varied layouts.

@ddunbar
Copy link
Member

ddunbar commented Dec 4, 2015

To clarify, I guess the proposal is that sourceRoot defines the root structure under which everything is presumed to follow the package manager conventions?

The one thing I am a little worried about with this approach (as opposed to, say, an "exclusions" mechanism), is that if someones project almost fits into the layout, but not quite, it will be harder for them to adapt. But perhaps that should be a handled in another manner that fewer projects need.

@mxcl
Copy link
Contributor

mxcl commented Dec 5, 2015

Yes I think this would be the first of many additions that allow tweaks. This one IMO allows a lot more projects to easily become a package without too much difficulty for them.

And yes you are correct, it is essentially saying the `Sources` directory is called `Foo`.

Perhaps you are right and it would be better to not have this and instead just add an exclusions mechanism, something like:

let package = Package(
   dependencies: [
       .Package(url: "https://github.com/ddunbar/CryptoSwift.git", majorVersion: 0)
   ]
)

package.targets.remove(name: "foo")

@ddunbar
Copy link
Member

ddunbar commented Dec 5, 2015

I think this case is one that should find into the "structured Package() initializer" declaration subset and not require extra statements. I like "sourceRoot:" better than that, but I am mixed on that versus, say:

let package = Package(
   dependencies: [
       .Package(url: "https://github.com/ddunbar/CryptoSwift.git", majorVersion: 0)
   ],
   exclude: ["Examples"]
)

I'd like it if we had a good way for evaluating this choice based on the layout of actual projects...

@krzyzanowskim
Copy link
Contributor Author

+1 for Daniel syntax. Speaking of "exclude", I'd love to disable "generate module out of every subdirectory". Is it possible?

@swift-ci
Copy link
Contributor

swift-ci commented Dec 7, 2015

Comment by Kostiantyn Koval (JIRA)

Modules are generated only for subdirectories in Source. Every inner subdirectory in is just a directory.

Hello/Sources/A/A.swift
Hello/Sources/B/C/C.swift
Hello/Sources/B/D/D.swift
// 2 Targets, A.a, B.a

You would be able to exclude some with exclude.

 exclude: ["Examples"]

How would you like it to behave? Would you like to say both A and B folders are one target ?

@krzyzanowskim
Copy link
Contributor Author

I don't know, maybe "nonTargets", or simply drop assumption that what inside is target and move to explicit definition of targets and modules

@mxcl
Copy link
Contributor

mxcl commented Dec 7, 2015

The convention-based approach is designed to accelerate development for people who want to use it, but we understand not everyone enjoys such approaches and would like to make it possible to manually specify targets as you wish.

@swift-ci
Copy link
Contributor

swift-ci commented Dec 9, 2015

Comment by Kostiantyn Koval (JIRA)

I did sone work on this issue and added exclude directories support
#57

@mxcl
Copy link
Contributor

mxcl commented Dec 11, 2015

We've merged the exclude support.

@swift-ci
Copy link
Contributor

Comment by Kostiantyn Koval (JIRA)

Is there anything else that should be done in this issue or can we mark it as resolved?
As I see @krzyzanowskim made CryptoSwift working with swiftpm. https://github.com/krzyzanowskim/CryptoSwift/#swift-package-manager

@krzyzanowskim
Copy link
Contributor Author

I've gave up and reorganised folders structure in CryptoSwift, which is not the way I should do to have support for Package Manager. For that reason this "issue" is still valid.

@swift-ci
Copy link
Contributor

Comment by Kostiantyn Koval (JIRA)

Ok, the swiftpm follows convention approach for Source folders layout + it has `exclude` folder functionality. How else can we make it easier for people to add swiftpm support for projects with custom folders layout? Any ideas and suggestion?

@krzyzanowskim
Copy link
Contributor Author

CocoaPods approach is quite convenient, where it's not PM who assume folder layout but it's configuration that describe locations for required parts. I don't think "exclude" should be the way to go, I'd rather suggest remove hard requirement for "Sources" etc and allow for configuration to handle that part.

@mxcl
Copy link
Contributor

mxcl commented Jan 25, 2016

We acknowledge it is necessary to add full configuration support for Targets. This project is still only 4 months old however, so we'll get to it in due course.

@abertelrud
Copy link
Contributor

Update: The `swift package init` goes some way toward this, since it creates the default structure. That said, I very much agree that we should allow the package manifest to specify paths of things like the `Sources` and `Tests` directories explicitly if it chooses to (if it doesn't, the defaults will be the current conventions).

@swift-ci
Copy link
Contributor

Comment by Darren Mo (JIRA)

Custom directory layouts would be useful for a project like Thrift, which has many language implementations that are organized like this: lib/__LANG__.

There is a pull request for a pure Swift 3 version of Thrift. But the author, apocolipse (JIRA User), has to work around this Swift PM limitation by creating a separate repository that puts the Sources directory at the root of the repo.

With support for custom directory layout, he could just specify in Package.swift that the sources directory is at lib/swift/Sources.

@swift-ci
Copy link
Contributor

Comment by Chris Simpson (JIRA)

As fumoboy007 (JIRA User) mentioned, projects like Thrift which have multiple language libraries in 1 hierarchy would benefit from configurable directory layouts.

Currently projects like Cocoapods and Carthage allow this, Cocoapods requires the podspec at the root of the repository with a setting pointing to where sources live,
Carthage requires an xcodeproj at the root of the repository with a shared framework scheme (allowing the xcodeproj to point to source locations accordingly).

Specifically we would need the ability to point to where the SPM SourceRoot is, not just where `Sources` lives, due to the need to also know where the accompanying `Tests` dir lives.

@rballard
Copy link
Contributor

This was discussed in our Swift 4 SwiftPM Roadmap as:
– Overriding Package Conventions

The package manager provides certain conventions for the on-disk layout of a package which allow us to identify multiple modules, infer product types (executable vs library), associate tests with the module they test, etc. If you wish to create a Swift package from a source tree which does not conform to these conventions, there is currently no way to do so. We'd like for users to be able to override the conventions and explicitly specify how their source tree is laid out using directives in the Package.swift manifest.

We are also considering removing some of the existing conventions so that there is one standardized conventional layout instead of the multiple supported layouts available today; if we do so, manifest convention overrides would allow existing packages that are using the alternate layouts to continue using those layouts.

@rballard
Copy link
Contributor

rballard commented Apr 4, 2017

@ankitspd
Copy link
Member

Stardust bot update:
Repository: https://github.com/apple/swift-package-manager
Branch: master
Commit: c75f5b4

@swift-ci
Copy link
Contributor

Comment by Darren Mo (JIRA)

Thanks, @aciidb0mb3r, @rballard, and everyone else!

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 4, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants