[build2] Build system reorganization (heads-up to distribution packagers)

[Matthew Krupcale]

On Wed, Oct 2, 2019 at 9:22 AM Boris Kolpackov <boris at codesynthesis.com> wrote:
>
> We are a couple of weeks away from the 0.12.0 release and I wanted to
> give heads-up to distribution packagers (and anyone else interested)
> about the major reorganization in the build system (the `build2`
> package).

Glad to hear that, and thanks for the heads-up.

> A bit of background on build system modules: they are libraries
> that can be either linked statically, linked dynamically, or
> loaded dynamically (with dlopen() or equivalent). We call it
> the "library first" model meaning that they are normal libraries
> that happen to additionally be usable via dynamic loading. As a
> result, they should be installed into `$prefix/lib/` rather than
> `$prefix/libexec/`.

Sounds good. This was my plan as well, and I was thinking of packaging
each module as a separate sub-package (with the proper
requires/dependencies). However, I wonder what might be considered
required modules by libbuild2 to have some sort of minimum viable
build system. Since the modules themselves require libbuild2, any
modules required by libbuild2 should be part of the libbuild2
sub-package to avoid cyclic dependencies.

Do you have a list of modules which are considered required by
libbuild2? Or would you consider all of the bundled modules required
by libbuild2 and as such I should not make sub-packages for the
modules?

> One thing that you may notice is the strange-looking `0.12-0.12`
> suffix in the module libraries as well as the extra intermediate
> symlink with just one `0.12`:

> This is because module libraries contain two versions: the build
> system core (`libbuild2`) interface version as well as the module
> version. Because the above modules are bundled, the two version are
> the same. But if it were an external module, then they would normally
> be different, for example:

I'm wondering if this may become too complicated, especially with
platform-specific library versioning also in the mix. How will these
two filename-based versioning systems work together?

> The intermediate symlink (`libbuild2-hello-0.12.so`) is what we call
> a "load name" (as opposed to `libbuild2-hello.so` which is the "link
> name"). It is the name that the build system core will use to
> dynamically load modules ensuring that they are compatible. It is
> also a way for us to have multiple versions of the build system core
> and corresponding modules installed in the same location (e.g.,
> `$prefix/lib`).

I wonder if it might be better to use symbol versioning[1,2] for this
(at least on Linux--I don't know how Mac/Windows deal with this).

Consider a scenario in which the module developer links against a
different version of libbuild2 than that against which the user's
build2 is linked.

With filename versioning:
  1. Someone writes a (possibly proprietary) build system module "a"
linked against libbuild2-0.12.so, producing
libbuild2-a-0.12{,-0.1}.so.
  2. Module "a" is distributed along with libbuild2-0.12.so.
  3. User loads module "a" in some build2 project with build2 linked
against libbuild2-0.13.so, dynamically loading both libbuild2-0.12.so
and libbuild2-0.13.so in the process. Depending on the particular
changes between these two library versions, this may fail
catastrophically.

With symbol versioning:
  1. Someone writes a (possibly proprietary) build system module "a"
linked against libbuild2.so.1 with symbol versions 0.12, producing
libbuild2-a.so.1.
  2. Module "a" is distributed along with libbuild2.so.1 with symbol
versions 0.12.
  3. User loads module "a" in some build2 project with build2 linked
against libbuild2.so.1 with symbol versions 0.12 and 0.13, dynamically
loading only libbuild2.so.1 with symbol versions 0.12 and 0.13.
Provided the symbol versions were used in a compatible manner (e.g. a
symbol previously versioned 0.12 must be kept), this should work just
fine.

The same can be said about a module linked against newer libbuild2
than the user's build2 distribution, as well as an alternative
scenario in which two build system modules are built using different
libbuild2 versions, possibly even different from the user's libbuild2
version.

Even if multiple versions of libbuild2-0.{x,y,z}.so are distributed in
`$prefix/lib`, there's no guarantee that they can be simultaneously
loaded by a single executable with filename-based versioning, whereas
the multiple versions can coexist in the same executable with symbol
versioning. Furthermore, then build module developers only need to
worry about their module library version (e.g. 0.1) rather than the
libbuild2 version (at least as far as filenames go--they still need to
be sure module users have libbuild2 with the symbol versions used by
their module). This avoids an explosion of
$libbuild2_interface_version-$libbuild2_module_version library
combinations, as the user will only need to load one libbuild2 library
(the one with the latest symbol versions necessary), unless it becomes
necessary to bump the SONAME due to ABI-breaking changes not handled
by symbol versioning.

Symbol versioning is probably a bit more difficult (for the library
developer) to implement than filename-based $major.$minor versioning
or even platform-specific filename-based versioning, but it is capable
of handling these situations of "multiple versions" of an interface
without having to deal with multiple library files. This generally
makes the library users' (both module writers and module users) lives
easier.

Thoughts?

Matthew

[1] https://sourceware.org/binutils/docs/ld/VERSION.html
[2] Section 3 of https://www.akkadia.org/drepper/dsohowto.pdf