[build2] Feature Request: Build2 Work Tree

[Klaim - Joël Lamotte]

On Wed, 28 Nov 2018 at 12:11, Boris Kolpackov <boris at codesynthesis.com> wrote:
>
> This is my first pass to clarify some of the misconceptions and try
> to understand what's this all about (after a couple of reads, I am
> still very fuzzy).

Thanks for your time, no problem.
I think I missed a few words like "### 3.1. Extend `bdep` and `bpkg`
to handle multiple." I meant "multiple directories for an operation".

> Klaim - Joėl Lamotte <mjklaim at gmail.com> writes:
>
> > In all these cases and in some previous complex projets, the best
> > organization recommended if I had used build2 to make them would have
> > been to make each specific tool/library it's own build2 project, then
> > group these in repositories separated as fine-grained as possible (the
> > usual would be a library and related executable are isolated in their
> > repositories).
>
> This is not quite what we recommend. There is nothing wrong with
> grouping related packages in a single project repository and this
> approach is supported by build2:
>
> https://build2.org/build2-toolchain/doc/build2-toolchain-intro.xhtml#guide-dev-multi

Indeed. The case I'm pointing is the most complex setup (although not
rare in my xp):
 - each project (solving a category of issues) have one repository,
with one or more packages (lib/exe), and can be used as is or
worked-on independently;
 - teams work on a set of projects that need to be developed together,
but still need to be usable separately, so they can't put everything
in the same repository*;
 - as part of their development process, developers need to be able to
quickly add/remove/edit source repositories to their current work
environment, often for than one project at a time (this is often
because of debugging, developing patches as suggestion to other teams
and/or when working on "trans-team" projects);
In which case manipulating a worktree is super slow and error-prone by
hand, in particular for new recruits and when having to switch setup
regularly.

I can give more specific details on the situation in the company I
work for, it that help, as it is the most extreme/complex example I
have of this situation.

The general issue is to be able to quickly setup/change/remove a
number of these repositories, ready to build and be developed,
assuming a specific set of configurations will be used.

*: such situation might indicate that the team should work on one
giant repository like Google does, but there are situations where it's
not working either.

>
> >  - 1.1. A newcomer setup a directory, let's call it a "work tree",
> > with all the projects related to the team fetched, configured and
> > built;
>
> So basically clone a bunch of repositories and initialize them with
> bdep-init, creating a bunch of configurations in the process.

Exactly.

>
> >  - 1.2. Attempt to change the current branch of all the projects to
> > the same branch name (all to `master`, or all to `develop` would be
> > the most common);
>
> Ok, sounds like purely VCS operation.

I think so, though experience with `qisrc` shows that some situations
might require additional behavior.

>
> >  - 1.3. Define a new configuration, then make all the projects of the
> > directory use and build with that configuration (should also be easy
> > with several configurations added);
>
> Ok, so basically a bdep-init -C call followed by a bunch of bdep-init -A.

I think that's correct.

> >  - 1.4. Launch all the tests of all the projects (one build2 project
> >  at a time);
>
> Ok, so a bunch of bdep-test calls.

Yes.

> >  - 1.5. Add/clone new repositories to the directory and make them all
> > build with all the configurations already existing;
>
> Ok, a bunch of bdep-init -A calls (the same project but multiple
> configurations).

Yes.

> >  - 1.6. Define the same package repository(ies) for all the
> > configurations of the directory (so that the dependencies all come
> > from the same repositories);
>
> Not sure what this means. Are you talking about a manual bpkg-rep-add
> for each build configuration?

The way I think about it is that it would be like: one
"repositories.manifest" shared between all repositories/projects
(maybe in addition to the ones in the projects).
If that'st strictly equivalent to just doing `bpkg-rep-add` for each
listed repository, then yes that would be it.
This is mainly to be able to specify one package repository or set of
package repositories that would be common for, say, a whole company
(because owned, built and checked internally).
In the case of the company I work for, modifying 150+ repositories
would rely on one package repository, but it did happen in the past
that we had to change the url.

> >  - 1.7. When a dependency is actually one of the projects,
> > automatically makes all the configurations use the local version
> > instead of the one retrieved from the package repositories;
>
> This is achieved by simply initializing  it (bdep-init) in the
> respective configurations. So this is effectively the same as 1.5.

I'll reformulate to clarify:
If a library X is initially downloaded as package from a remote
package into a configuration, then acquiring/cloning the sources of X
then doing that operation would remove the package and use the
sources.

If `bdep-init -A` already does that in one step, then yes it would be
it (I didn't check that specific case yet; I'm mainly listing it
because we had to implement it in qibuild in the past, it was
problematic before we did).

>
> >  - 1.8. Change a dependency version in all configurations (implicitely
> > changing that version for all the projects currently being worked on);
>
> Ok, so bdep-sync in the dependency upgrade/downgrade mode for all the
> build configurations.

I think that would work, though maybe there would need to be some kind
of failure handling.

> So far it sounds like all of these boil down to a for-loop either
> over the list of configuration or the list of projects. Am I about
> right?

Yes, I think you are correct and I did expect such imaginary tool to
be implemented like that, in essence. With some higher-level logic in
some situations (mostly error handling and guidance).

Note that I am merely pointing the need, not implying build2's team
should make it. Tools are built over git for example, not by people
making git, and that makes sense as these tools often implement higher
level logic than git.

I'm opening the discussion to exchange on what could be done, by who etc.
For example I could try making such tool myself (though it would not
be done quickly) but you might want something like that as part of
build2 as, in my opinion, that's kind of an extension of the mindset
behind bdep: an extension to what it does, as most operations I listed
could be done as a version of it's current operations but applied to
more than one source repository/ configuration.

(Also, it might be an interesting data point for you on how some
companies I know work(ed).)

I dont think it's an urgent issue. I'm interested in this because it
would be immediately as a blocker if, for example, I wanted to replace
_today_ our current use of `qibuild`+`CMake` by `build2` in the
company I am working for.

>
> > ### 2.1 Git Sub-Modules to handle Work Trees.
> >
> > I believe this is the solution used by `build2` team [...]
>
> No, we don't use git submodules for that.

I stand corrected. :)

A. Joël Lamotte