Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-6053] [llbuild] Add support for IPC-based extension of BuildSystem + client library #792

Open
ddunbar opened this issue Oct 4, 2017 · 2 comments

Comments

@ddunbar
Copy link
Member

ddunbar commented Oct 4, 2017

Previous ID SR-6053
Radar None
Original Reporter @ddunbar
Type New Feature
Additional Detail from JIRA
Votes 1
Component/s llbuild
Labels New Feature
Assignee None
Priority Medium

md5: 57ae6f0cdc4c06e14d2ae85590a1e6b3

Issue Description:

Currently, llbuild is intended to be used in a way where all build tasks are directly exposed to the build engine running within a single process.

This is problematic when working with existing build system designs which depend on running out-of-process build tools. The traditional way we have thought about solving this was by turning each tool individually into library-based tool (potentially with its own IPC boundary) which we could then interact with more directly. However, this is difficult and disruptive.

As an alternative, we should extend llbuild to expose an IPC protocol to subprocesses (e.g., an environment variable connected to a named pipe or Unix domain socket), and a new library which clients can link against.

This library would expose the following functionality:

  • Clients could register new sets of rule domains they could "match".

  • Clients could request new work be performed on their behalf (potentially matching those keys).

  • Clients could communicate scheduling information (such as when they are blocked waiting on another task, or when they are launching a new thread).

Under the covers, this would work by shuttling this traffic over the IPC pipe to the active build engine, which would then translate that into native APIs. Similary, when rule requests appear they would be forwarded to any actively running clients which had registered support for that rule domain (we could probably use a simple prefix matching scheme for routing).

The benefit of this approach is that it is quite easy to integrate with existing tools, and in such a way that the tool transparently uses llbuild if available, but if not simply does the work itself.

For example, consider a tool which does some computation, where one part is easily cached and may be shared with many things in the build:

func foo(name: String, input: Data) {
   let z = computeSomeProperty(input)
   ... do more work ...
}

then, assume the tool can unambiguous identify this computation (give it a name), we could write something like:

import llbuild

func init() {
  // Register rule callbacks.
  llbuild.registerRule(prefix: "ComputeSomeProperty-") { (rule, engine) in
    let name = ... extract name from rule ...
    engine.requiresInput(name) { input in
      let output = computeSomeProperty(data)
      engine.provideOutput(output)
    }
  }
}

func foo(name: String, input: Data) {
  // This will block (and notify the engine we are blocked) until available.
  let output = llbuild.computeRule("ComputeSomeProperty-\(name)")
  ... do more work ...
}

When run in a build, the engine will now only schedule this task to run once, after that it will be cached (in memory) and stored in the database. Any subsequent requests for the same value will be vended directly. We can make the library convenient to use so that if llbuild isn't connected, we just do the work as usual with minimal overhead.

@ddunbar
Copy link
Member Author

ddunbar commented Mar 31, 2020

While it never landed, for posterity I did at one point start working on this here: https://github.com/ddunbar/swift-llbuild/tree/SR-6053

@ddunbar
Copy link
Member Author

ddunbar commented Mar 31, 2020

This also has some (loose) similarities to the Linda programming language: https://en.wikipedia.org/wiki/Linda\_(coordination_language)

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant