[build2] build2 0.17.0 pre-release notes for distribution package maintainers

[Boris Kolpackov]

Hi,

We aim to release build2 0.17.0 early next weeks and I would like to
give a heads-up to distribution package maintainers (and anyone else
interested) about build-related changes in case they want to test
things out using the staged version of the toolchain[0] and report
any issues or concerns before the final release.

Prior to 0.17.0 the build2 toolchain had the following external
dependencies:

1. libpkg-config (substitutable with system libpkgconf)
2. libsqlite3
3. libodb and libodb-sqlite

These four libraries were bundled with the build2-toolchain amalgamation,
were fetched from cppget.org when building the toolchain from packages
(e.g., with the install/build script unless --local was specified), and
you were able to substitute them with system-installed versions (using
the --system install/build script options).

This has changed as follows for the 0.17.0 release:

1. The libpkg-config source code is now bundled with libbutl (build2
   toolchain utility library) and is no longer built from a package.
   It is also no longer possible to use system libpkgconf instead.

   In other words, pkg-config support is now an integral part of libbutl
   and you no longer need to deal with it as an external dependency.

   The key motivation for this change is that libpkgconf again broke
   backwards-compatibility. Plus, it has limitations that will start
   affecting support for C++20 modules, which are now becoming usable.
   For details/background, see[2].

2. The libsqlite3 source code is now bundled with libbutl and is no
   longer built from a package. However, it is possible to instead
   use system version with the config.libbutl.system_libsqlite2=true
   configuration variable.

   However, I would strongly advise against doing this for the following
   reasons:

   a) SQLite places a compile-time limit on the number of databases
      that can be attached to a connection with the default being 10
      (see SQLITE_MAX_ATTACHED). All the major distributions (Debian,
      Fedora, Homebrew) do not increase this default.

      In build2 (specifically, in bpkg), we attach databases for all the
      linked configurations involved in a package manager operation and
      it's not difficult to exceed this default limit. The typical
      scenario would be a number of target configurations linked via a
      shared host configuration.

   b) Our use of SQLite in bpkg is performance-sensitive and we fine-tune
      our build. For example, we build SQLite single-threaded and without
      shared cache support, which gives us an estimated 10% performance
      boost. On the other hand, all the major distributions (Debian, Fedora,
      Homebrew) build SQLite without these optimizations.

   Also note that our use of SQLite is not security-sensitive. Specifically,
   a vulnerability in SQLite is unlikely to be exploitable remotely via bpkg
   or bdep.

3. The libodb and libodb-sqlite source code is now bundled with libbutl
   and is no longer built from packages.

   I don't believe this should be a major change from the current state
   of things since we use a beta version of ODB that is not packaged for
   any distribution (so it was already effectively bundled).

   We are planning to continue with this approach long-term (that is,
   even after the final version of ODB is released) because ODB uses
   build2 as the build system and depending on packaged ODB would create
   a chicken-and-egg problem for distributions (that is, to package
   build2 one needs ODB and to package ODB one needs build2).

   Overall, we feel this is the best approach, at least in the short to
   medium term (we can always re-evaluate this decision later).

I don't believe these changes should cause any technical difficulties when
adjusting your build2 package build for 0.17.0. Specifically, if you are
happy to go with everything bundled (as is recommended), then all you
will need to do is remove any explicit references to the above-mentioned
libraries. Note that there will be two new library binaries in libbutl:
libbutl-pkg-config.{a,so} and libbutl-odb.{a,so}.

Let me know if you have any questions or concerns.

[0] https://build2.org/community.xhtml#stage
[1] https://github.com/build2/build2/issues/381