[build2] Feature Request: Build2 Work Tree

Klaim - Joël Lamotte mjklaim at gmail.com
Fri Nov 23 01:43:03 UTC 2018

I decided to spend a few sleep hour to finish redact my thinking below
as I think it's better discussed early on, so maybe it's not super
well redacted. Please tell me what you think.
If you agree with what I think would work here, the other main
question is: would it be part of build2 or some project by someone
else (I guess me).

Also feedback from other users would be useful as I can only relie on
my xp so far.

(note: this is a markdown file copy-pasted)

A Joël Lamotte


Feature Request: Build2 Work Tree
2018-11-23 A. Joël Lamotte <mjklaim at gmail.com>

0 - Summary

I suggest that a new set of commands would be useful if available
through `build2` that would help a developer working (as in
developping, not just using) on a set of related projects to manage
these projects together as a whole.

1 - Issues

The inspiration comes from several projects I have worked on or am
working on currently, in particular:

  - My current job's projects: robotics, involves tons of libraries
specialized in one aspect, and tons of processes projects using these
libraries and communicating remotely (more than 150
projects/repositories handled by more than 10 teams).
  - My open-source project (halted currently) : basically a toolchain
for another domain than programming (in phylosophy it looks like
build2 actually. It's at least 5 projects/repositories.
  - A game project: multiplayer and partially persistent and
extendable, 15 libraries, 3 executables at the moment. Most can be
handled as one repository, but there is also an online service and
some other related projects.

In all these cases and in some previous complex projets, the best
organization recommended if I had used build2 to make them would have
been to make each specific tool/library it's own build2 project, then
group these in repositories separated as fine-grained as possible (the
usual would be a library and related executable are isolated in their
repositories). This is not the setup I used so far because of lack of
a good depency manager designed for people working on these projects,
but I agree it would be ideal.

Most of the time, someone in the team of one of these big projects
will need to always work with several repositories at the same time.
This is not because they need to make changes through the whole scope,
but it's because when they change something in one project, want to
update the other projects that have been changed by teammates but that
are not released yet, they want to be quickly able to setup the
repositories in the versions they need for each project and start
testing them, patching them etc.

Here are some of the usual operations I experienced that are currently
hard to do with `build2` because I have to do it manually or through
some easily breakable/not scalable scripts:

 - 1.1. A newcomer setup a directory, let's call it a "work tree",
with all the projects related to the team fetched, configured and
 - 1.2. Attempt to change the current branch of all the projects to
the same branch name (all to `master`, or all to `develop` would be
the most common);
 - 1.3. Define a new configuration, then make all the projects of the
directory use and build with that configuration (should also be easy
with several configurations added);
 - 1.4. Launch all the tests of all the projects (one build2 project at a time);
 - 1.5. Add/clone new repositories to the directory and make them all
build with all the configurations already existing;
 - 1.6. Define the same package repository(ies) for all the
configurations of the directory (so that the dependencies all come
from the same repositories);
 - 1.7. When a dependency is actually one of the projects,
automatically makes all the configurations use the local version
instead of the one retrieved from the package repositories;
 - 1.8. Change a dependency version in all configurations (implicitely
changing that version for all the projects currently being worked on);

Basically, the issue is that if we have M configurations and N
projects and K branches to work with at any time, it can become hard
to apply any of these operations quickly. At least, it's never just
one command. M and N are moving values while developing so it can be

Currently, `bdep` and `b` are not sufficient to do any of these in one
command because they work with one project at a time, and `bpkg` works
with one configuration directory at a time too. It have to be done
manually or through scripts.

2 - Exploring Current Solutions

### 2.1 Git Sub-Modules to handle Work Trees.


I believe this is the solution used by `build2` team but they are not
in a situation where each project can have several teams add to their
repositories and non-linearly evolve.

My experience is that this can work for some time but quickly becomes
problematic. In any way, it it was sufficient, I wouldn't need
something like `build2`.

### 2.2. Scripts driving `build2` tools.

Unfortunately, this means:

 - making specific scripts for each super-projects, instead of just
defining them and let the tool handle the potential changes;
 - they are brittle as they have to be maintained and rely on
logic/turing-completeness, while we just want to change a whole state
per operation;

### 2.3. Magic commands that I am not aware of.

Please teach me these if they already exist.

3 - Exploring Solutions To Develop

For my game project for now I use 1 repository to handle all the projects.
For my open source project, there was several repositories and a
super-repository with submodules. It became quickly unmanageable, in
particular because each project need to have it's own version number,
which is where a dependency manager becomes useful (and there was none
reliable for C++ at the time). The project is halted but I want to
resurect it in a year with build2 (as it seems to be an appropriate
tool for handling it).

In the case of my current job, we use `qiBuild`
(http://doc.aldebaran.com/qibuild/index.html), a tool that was
developped internally and open-sources and available through `pip`. It
have it's (design) problems, but it have some interesting features.
One of it's tools (because like `build2`, it is made of several
programs with different levels of usage, in theory at least), there is
`qisrc` which helps work with big projects.

Notably, `qisrc` (and therefore `qibuild`, the configure+builder)
allows to describe a work tree through a XML file.
Then `qibuild` allows in one project to define configurations (it
relies on CMake so the definition of "configuration" is similar but
different), which are created for each project in the work tree that
the current project depends on. So if project A depends on B, C and D,
and B and D are in the work tree, when creating configuration
"msvc-14.0-Debug" we end up one one configuration directory per
project, one in each project (it can also create one configuration
directory but I think it's buggy). Then it makes sure that building A
will build B and D, and use C which is already available as a package.

`qibuild` is definitely not perfect, in particular as a dependency
manager and because it relies to much on ad-hoc CMake scripts and
special cases. However it works well with setting up a worktree with a
few commands and let you focus on other issues (when it works).

The main author left the company some years ago and decided to not
continue it's support (we support it but we don't have anyone
speicalized in it). He switched to `conan`, combined with a tool of
his that does the same thing than `qisrc` but simpler:
That tool combined with conan and some other internal tools similar to
qibuild help them handle the issues listed in section 1.

I rely on experience with `qibuild` to suggest the following potential
solutions (that could be, or not, cumulated or combined):

### 3.1. Extend `bdep` and `bpkg` to handle multiple.

`bdep` (and `b`) works on one repository and `bpkg` works on one configuration.
Making most `bdep` and `bpkg` commands optionally applicable to
several project directories at the same time would almost solve
issues 1.3, 1.4, 1.5, 1.7 and 1.8.

For example, currently:

    PS E:\Projects\build2-libs> bdep test ./*
    error: E:\Projects\build2-libs\ is not a (sub)directory of a
package or project
    PS E:\Projects\build2-libs> bdep test -d ./*
    error: .\*\ is not a (sub)directory of a package or project

Although it would not solve all issues if these commands were working,
it would make sense that operations that work with `bdep`, for
example, is basically the same command when applied to several
projects instead of one.

### 3.2. `worktree` - A file format to describe a "build2 work tree".

This is just a temptative name for such a file.

There would be several advantages to have a file format to describe a
work tree as follow:

 - Defines repositories ("origin") that need to be cloned (and worked
on). For each repository:
   - a relative path where to clone;
   - a "origin" repository url (including branch, tag or commit identifier).
 - Defines configurations to automatically setup for all projects. For
each configuration:
   - a relative path where to create the configuration;
   - a set of `config.option = value` to be applied to that configuration;
 - Defines a common set of configuration variables, applied on all
 - Defines a common set of package repositories;

Having such file would help drive current tools commands (3.1) or
making a new tool (3.3).

### 3.3. `bwt` - A new tool to handle "build2 work trees".

This is just a temptative name for such a tool.

The core idea is to either:

 3.3.1. Provide a command for each operation listed in section 1.
 3.3.2. Extend current tools to handle multiple repositories (3.1),
then add this tool for handling the issues that are higher level or
too different (for example 1.1).

In all cases, such a tool would need to rely on the existence of a
worktree file.
That worktree file would need to exist in the root directory of the
worktree. It should be possible to make it automatically downloaded
from some URI, but it needs to be then copied locally. The user can
then change it and just make `bwt` update the work tree according to
it, or use `bwt` to modify it.

It might be possible, in the case of 3.3.2, that such a tool would
only be able to update the work tree according to the worktree file.
The other operations would then be easily done through extended `bdep`
and `bpkg`.

`bwt` would mostly drive `bdep`, `bpkg` and the source control tool(s).

More information about the users mailing list