[build2-announce] build2 0.16.0 released

Boris Kolpackov boris at codesynthesis.com
Tue Jul 4 12:32:16 UTC 2023

We have released build2 toolchain version 0.16.0.

The two main areas of focus in this release are the system package manager
integration and more advanced functionality for more complex projects. The
system package manager support includes both consumption and production of
binary distribution packages, including uploading them from the CI builds.
This support required a large amount of ground work which produced
functionality useful in its own right, such as relocatable installations,
installation filtering, and installation manifests. The more advanced
functionality includes built-in support for the Objective-C/C++ and
Assembler with C Preprocessor (.S) compilation, dynamic target extraction
in addition to prerequisites, and support for buildfile importation.

There are also quite a few smaller features in this release, such as new
functions, commands, and builtins, as well as a large amount of maintenance
work, including the low verbosity diagnostics overhaul and support for
diagnostics buffering. We have also continued with our performance
optimizations in the build system and the package manager which should
especially help projects with a large number of dependencies.

A note on backwards compatibility: this release cannot be upgraded to from
0.15.0 and has to be installed from scratch.

The corresponding NEWS file entries are included below with a more detailed
discussion of the major new features available in the release notes:


This release has been tested on Windows, Linux (x86_64 and aarch64), Mac OS
(x86_64 and M1), and FreeBSD with GCC, Clang (both vanilla and Apple's),
MinGW GCC, and MSVC (14, 15, 16, and 17). A big thank you to everyone who
helped test the development snapshots and the release candidate.

The following new build configurations have been added to the CI service:



  macos_12-clang_14.0       (Xcode 14.2,   Apple Clang 14.0.0)
  macos_13-clang_14.0       (Xcode 14.3.1, Apple Clang 14.0.3)
  macos_13-clang_15.0       (Xcode 15.0,   Apple Clang 15.0.0)

  windows_10-msvc_17.6                  (LTSC)

Note that the Linux configurations now support building for both x86_64 and
aarch64 (ARM64).

There are also two new build classes: sys and bindist. The sys class contains
configurations that support the system package manager integration for package
consumption. They can be used to test against system-installed versions of
packages or satisfy dependencies that are not yet available from source. The
bindist class contains configurations that support the generation and upload
of binary distribution packages. Currently, this class contains the following


All in all, there are now 91 build configurations. For the complete list, see:


Installation instructions:


Download directory:


SHA256 checksums:

4c457f8365b83000e9ffb6de3924f47d3aee3722e0101996fdfab8adcfb80237 *build2-install-0.16.0.sh
1ae62f46fa2c23abd734d8497abae1909481d48ea74f924a7335887a17a42e21 *build2-install-msvc-0.16.0.bat
282cb2a409dcafb1e841664290b4ff55255852eba15b81b4e03f4e6c61a034a9 *build2-install-clang-0.16.0.bat
6c5fef712ebb90055e3d7745dade4125e779a55ba823dedfd063dff7aba3e8b8 *build2-install-mingw-0.16.0.bat

222690e4b22cd6e7afe900da495475310087b7c9bee8beee71a6d7f5635bbc88 *build2-toolchain-0.16.0.tar.gz
23793f682a17b1d95c80bbd849244735ed59a3e27361529aa4865d2776ff8adc *build2-toolchain-0.16.0.tar.xz

273afae47c738085fe35fd3ddf9eca830645fd8478d8c2f7a8d10c293197770d *build2-baseutils-0.16.0-x86_64-windows.zip
4a6f16a4d8f055c2caceee7437d47b84008c6f5031813d9a49b18f2686b1c12b *build2-mingw-0.16.0-x86_64-windows.tar.xz

Repository certificate fingerprint for cppget.org:


NEWS file entries:

build2 version 0.16.0

  * Support for Objective-C/C++ compilation.

    Specifically, the c and cxx modules now provide the c.objc and cxx.objcxx
    submodules which can be loaded in order to register the m{}/mm{} target
    types and enable Objective-C/C++ compilation in the c and cxx compile
    rules. Note that c.objc and cxx.objcxx must be loaded after the c and cxx
    modules, respectively, and while the m{}/mm{} target types are registered
    unconditionally, compilation is only enabled if the C/C++ compiler
    supports Objective-C/C++ for this target platform. Typical usage:

    # root.build
    using cxx
    using cxx.objcxx

    # buildfile
    lib{hello}: {hxx cxx}{*}
    lib{hello}: mm{*}: include = ($cxx.target.class == 'macos')

    Note also that while there is support for linking Objective-C/C++
    executables and libraries, this is done using the C/C++ compiler driver
    and no attempt to automatically link any necessary Objective-C runtime
    (such as -lobjc) is made. For details, refer to "Objective-C Compilation"
    and "Objective-C++ Compilation" in the manual.

  * Support for Assembler with C Preprocessor (.S) compilation.

    Specifically, the c module now provides the c.as-cpp submodule which can
    be loaded in order to register the S{} target type and enable Assembler
    with C Preprocessor compilation in the c compile rule. For details, refer
    to "Assembler with C Preprocessor Compilation" in the manual.

  * Support for buildfile importation.

    A project can now export buildfiles that can then be imported by other
    projects. This mechanism is primarily useful for exporting target type
    definitions and ad hoc rules.

    Specifically, a project can now place *.build files into its build/export/
    subdirectory (or *.build2 and build2/export/ in the alternative naming
    scheme). Such files can then be imported by other projects as buildfile{}
    targets. For example:

    import thrift%buildfile{thrift-cxx}

    While for other target types the semantics of import is to load the
    project's export stub and return the exported target, for buildfile{} the
    semantics is to source the imported buildfile at the point of importation.

    Note that care must be taken when authoring exported buildfiles since they
    will be sourced by other projects in unpredictable circumstances. In
    particular, the import directive by default does not prevent sourcing the
    same buildfile multiple times (neither in the same project nor in the same
    scope). As a result, if certain parts must only be sourced once per
    project (such as target type definitions), then they must be factored into
    a separate buildfile (in build/export/) that is imported by the "main"
    exported buildfile with the `once` attribute. For example, the above
    thrift-cxx.build may contain:

    import [once] thrift%buildfile{thrift-cxx-target-type}

    See also "install Module" in the manual for details on the exported
    buildfile installation.

  * Support for defining explicit (as opposed to ad hoc) target groups.

    A user-defined explicit target group must be derived from the group base
    target type. If desired, it can be marked as "see-through", meaning that
    when it is listed as a prerequisite of a target, the matching rule will
    "see" its members, rather than the group itself. For example:

    define [see_through] thrift_cxx: group

    define thrift: file
    thrift{*}: extension = thrift

    exe{hello}:       cxx{hello} thrift_cxx{data}
    thrift_cxx{data}: thrift{data}

    Explicit group members can be specified statically, injected by an ad hoc
    rule, or extracted dynamically by the depdb-dyndep builtin (see the next
    NEWS item). For example:

    thrift_cxx{data}<{hxx cxx}{data_constants}>: thrift{data}        # Static.

    thrift_cxx{~'/(.+)/'}<{hxx cxx}{^'/\1_types/'}>: thrift{~'/\1/'} # Inject.
       depdb dyndep --dyn-target ...                                 # Dynamic.

  * Support for dynamic target extraction in addition to prerequisites.

    This functionality is enabled with the depdb-dyndep --dyn-target option.
    If the recipe target is an explicit group (see the previous NEWS item),
    then the dynamically extracted targets are added as its members.
    Otherwise, the listed targets are added as ad hoc group members. In both
    cases the dynamically extracted target is ignored if it is already
    specified as a static member or injected by a rule. Note that this
    functionality is not available in the --byproduct mode. See the
    depdb-dyndep builtin options description for details.

  * New `lines` depdb-dyndep dependency format in addition to `make`.

    The `lines` format lists targets and/or prerequisites one per line. See
    the depdb-dyndep builtin options description for details.

  * Low verbosity diagnostics rework.

    The low verbosity (level 1) rule diagnostics format has been adjusted to
    include the output target where appropriate. The implementation has also
    been redesigned to go through the uniform print_diag() API, including for
    the `diag` pseudo-builtin in ad hoc recipes.

    Specifically, the `diag` builtin now expects its arguments to be in one of
    the following two forms (which correspond to the two print_diag() forms):

    diag <prog> <l-target> <comb> <r-target>...
    diag <prog> <r-target>...

    If the `diag` builtin is not specified, the default diagnostics is now
    equivalent to:

    For update:

    diag <prog> ($<[0]) -> $>

    And for other operations:

    diag <prog> $>

    For details, see the print_diag() API description in diagnostics.hxx. See
    also GitHub issue #40 for additional background/details.

  * Buffering of diagnostics from child processes.

    By default, unless running serially or --no-diag-buffer is specified,
    diagnostics issued by child processes (compilers, etc) is buffered and
    printed all at once after each child exits in order to prevent
    interleaving. See also the new --[no-]diag-color options.

  * New $path.posix_string() and $path.posix_representation() functions.

    These functions are similar to $path.string() and $path.representation()
    except that they always return the string/representation of a path in the
    POSIX notation, that is, using forward slashes.

  * New $regex.filter[_out]_{match,search}(<vals>, <pat>) functions.

    The match versions return elements of a list that match (filter) or do not
    match (filter_out) the regular expression. The search versions do the same
    except for the search instead of match regex semantics.

  * New $find(<sequence>, <value>), $find_index(<sequence>, <value>) functions.

    The $find() function returns true if the sequence contains the specified
    value. The $find_index() function returns the index of the first element
    in the sequence that is equal to the specified value or $size(<sequence>)
    if none is found. For string sequences, it's possible to request case-
    insensitive comparison with a flag, for example:

    if ($find ($values, 'foo', icase))

  * New $integer_sequence(<begin>, <end>[, <step>]) function.

    This function returns the list of uint64 integers starting from <begin>
    (including) to <end> (excluding) with the specified <step> or 1 if
    unspecified. For example:

    hdr = foo.hxx bar.hxx baz.hxx
    src = foo.cxx bar.cxx baz.cxx

    assert ($size($hdr) == $size($src)) "hdr and src expected to be parallel"

    for i: $integer_sequence(0, $size($hdr))
      h = ($hdr[$i])
      s = ($src[$i])

  * New $is_a(<name>, <target-type>), $filter[_out](<names>, <target-types>)

    $is_a() returns true if the <name>'s target type is-a <target-type>. Note
    that this is a dynamic type check that takes into account target type

    $filter[_out]() return names with target types which are-a (filter) or
    not are-a (filter_out) one of <target-types>.

    In particular, these functions are useful for filtering prerequisite
    targets ($<) in ad hoc recipes and rules.

  * Support for the hex notation for the uint64 type.

    Specifically, now we can do:

    x = [uint64] 0x0000ffff

    cxx.poptions += "-DOFFSET=$x"                 # -DOFFSET=65535
    cxx.poptions += "-DOFFSET=$string($x, 16)"    # -DOFFSET=0xffff
    cxx.poptions += "-DOFFSET=$string($x, 16, 8)" # -DOFFSET=0x0000ffff

    Note that there is no hex notation support for the int64 (signed) type.

  * Support for the `for` and `while` loops in Buildscript recipes and

    For example:

    for v: $values

    cat values.txt | for -n v

    while (!$regex.match(...))

    See "Command-For" and "Command-While" in the Testscript manual for

  * New `find` builtin in Buildscript recipes and Testscript.

    For example:

    find gen/ -type f -name '*.?xx' | for -n f

    See "find" in the Testscript manual for details.

  * Improvements to escape sequence support.

    In the double-quoted strings we now only do effective escaping of the
    special [$("\] characters, line continuations, plus [)] for symmetry.

    There is now support for "escape sequence expansion" in the $\X form where
    \X can be any of the C/C++ simple escape sequences (\n, \t, etc) plus \0
    (which in C/C++ is an octal escape sequence). For example:

    info "foo$\n$\tbar$\n$\tbaz"

    Will print:

    buildfile:1:1: info: foo

  * New include_arch installation location and the corresponding
    config.install.include_arch configuration variable.

    This location is meant for architecture-specific files, such as
    configuration headers. By default it's the same as the standard include
    location but can be configured by the user to a different value (for
    example, /usr/include/x86_64-linux-gnu/) for platforms that support
    multiple architectures from the same installation location. This is how
    one would normally use it from a buildfile:

    # The generated configuration header may contain target architecture-
    # specific information so install it into include_arch/ instead of
    # include/.
    h{*}:      install = include/libhello/
    h{config}: install = include_arch/libhello/

  * Support for installation filtering.

    While project authors determine what gets installed at the buildfile
    level, the users of the project can now further filter the installation
    using the config.install.filter variable. For details, see "Installation
    Filtering" in the manual.

  * Support for relocatable installations.

    A relocatable installation can be moved to a directory other than its
    original installation location. To request a relocatable installation, set
    the config.install.relocatable variable to true. For details, see
    "Relocatable Installation" in the manual.

  * Support for installation manifest.

    During the install operation, the config.install.manifest variable can be
    set to a file path (or `-`) in order to write the information about all
    the filesystem entries being installed into the specified file (or
    stdout). The format of the installation manifest is "JSON lines". For
    details, see the config.install.manifest variable documentation in the
    install module.

  * Ability to remap paths in source distributions.

    The dist target-specific variable can now specify a path besides true or
    false. This path is the "imaginary" source location which is used to
    derive the corresponding distribution location. This location can be
    either a directory path (to remap with the same file name) or a file path
    (to remap with a different name). If the path is relative, then it is
    treated relative to the target directory. Note that to make things less
    error-prone, simple paths without any directory separators are not allowed
    (use ./<name> instead).

    Note that if multiple targets end up with the same source location, the
    behavior is undefined and no diagnostics is issued. Note also that such
    remapping has naturally no effect in the bootstrap distribution mode.

  * The in.substitution variable has been renamed to in.mode.

    The original name is still recognized for backwards compatibility.

  * Ability to specify `in` rule substitutions as key-value pairs.

    See "in Module" in the manual for details.

  * New public/private variables model.

    Now unqualified variables are project-private and can be typed, meaning
    that a value assigned to a variable with such a name anywhere within the
    project will have this type. For example:

    [uint64] priority   = [null]
    [uint64] stack_size = [null]

    priority = 1     # Ok.
    stack_size = abc # Error.

    Besides the type, variable attributes can specify visibility (project by
    default) and overridability (false by default). For example:

      [uint64, visibility=target] priority   = [null]
      [uint64, visibility=target] stack_size = [null]

    thread{foo}: priority = 1 # Ok.
    priority = 1              # Error.

  * Support for post hoc prerequisites.

    Unlike normal and ad hoc prerequisites, a post hoc prerequisite is built
    after the target, not before. It may also form a dependency cycle together
    with normal/ad hoc prerequisites. In other words, all this form of
    dependency guarantees is that a post hoc prerequisite will be built if its
    dependent target is built.

    A canonical example where this can be useful is a library with a plugin:
    the plugin depends on the library while the library would like to make
    sure the plugin is built whenever the library is built so that programs
    that link the library can be executed without having to specify explicit
    dependency on the plugin (at least for the dynamic linking case):

    lib{hello}: ...
    lib{hello-plugin}: ... lib{hello}
    libs{hello}: libs{hello-plugin}: include = posthoc

    Note that there is no guarantee that post hoc prerequisites will be built
    before the dependents of the target "see" it as built. Rather, it is
    guaranteed that post hoc prerequisites will be built before the end of the
    overall build (more precisely, before the current operation completes).
    As a result, post hoc prerequisites should not be relied upon if the
    result (for example, a source code generator) is expected to be used
    during build (more precisely, within the same operation).

    Note also that the post hoc semantics is not the same as order-only in
    GNU make. In fact, it is an even more "relaxed" form of dependency.
    Specifically, while order-only prerequisite is guaranteed to be built
    before the target, post hoc prerequisite is only guaranteed to be built
    before the end of the overall build.

  * Support for dumping build system state in the JSON format.

    The new --dump-format option can be used to select the desired format.
    Its valid values are `buildfile` and `json-v0.1`. For details on the JSON
    dump format see "Appendix A - JSON Dump Format" in the manual.

  * Change to the --dump option semantics.

    This option now recognizes two additional values: `match-pre` and
    `match-post` to dump the state of pre/post-operations. The `match` value
    now only triggers dumping of the main operation.

  * New --dump-scope and --dump-target options to limit --dump output.

  * New --load-only option in addition to --match-only.

    This option has the effect of loading all the subdirectory buildfiles that
    are not explicitly included and is primarily useful in combination with

  * Quoted/display target names in the JSON structured result are now
    consistent with the JSON dump.

    Specifically, before we had `target` (display) and `quoted_target` and now
    we have `target` (quoted) and `display_target`. Note that this is a
    backwards-incompatible change.

  * The dist meta-operation no longer invokes the install program.

    This results in a substantial speedup, especially on Windows. The use of
    install (or another install-like program) can still be forced with
    explicit config.dist.cmd=install.

  * Clang -Wunqualified-std-cast-call warning was remapped to -Wextra.

    Clang 15 introduced the -Wunqualified-std-cast-call warning which warns
    about unqualified calls to std::move() and std::forward() (because they
    can be "hijacked" via ADL). Surprisingly, this warning is enabled by
    default, as opposed to with -Wextra or at least -Wall. It has also proven
    to be quite disruptive, causing a large number of warnings in a large
    number of packages. So we have "remapped" it to -Wextra for now and in the
    future may "relax" it to -Wall and potentially to being enabled by
    default. See GitHub issue #259 for background and details.

bpkg version 0.16.0

  * System package manager query/installation support for Debian and Fedora
    (and alike).

    The pkg-build command will now query (unless --sys-no-query is specified)
    the system package manager on Debian (and alike, such as Ubuntu) and
    Fedora (and alike, such as RHEL) for versions of packages that are
    specified as coming from the system (the sys: scheme). For example, if
    running the following command on one of these distributions:

    bpkg build hello ?sys:libsqlite3

    Then pkg-build will query the system package manager for the installed
    version of libsqlite3 and fail if none is present.

    Additionally, if --sys-install is specified, pkg-build will attempt to
    install such packages if not present but available from the system package

    Other relevant options include --sys-yes, --sys-no-fetch, --sys-no-stub,
    and --sys-sudo. See bpkg-pkg-build(1) for details.

    See also the `*-{name, version, to-downstream-version}` package manifest
    values in the manual for details on the bpkg to distribution package name
    and version mapping.

  * Binary distribution package generation support for Debian and Fedora (and

    The new pkg-bindist command can be used to automatically generate binary
    distribution packages from bpkg packages for Debian (and alike, such as
    Ubuntu), Fedora (and alike, such as RHEL), and for other operating systems
    as installation archives. For Debian and Fedora, dependencies can be
    satisfied with system packages, bpkg packages, or bundled. See
    bpkg-pkg-bindist(1) for details.

  * Package build configuration support (*-build-config manifest values).

    A package can now customize in its manifest the build configuration used
    by automated build bots. This includes specifying configuration variable
    values, forcing specific versions of dependencies, satisfying dependencies
    with system packages, and enabling/disabling build bot steps. For example:

    # Test with extras enabled.
    extras-build-config: config.libfoo.extra=true

    # Test with system-installed libsqlite3.
    system-builds: sys
    system-build-config: ?sys:libsqlite3

    # Enable Debian binary distribution generation and upload.
    bindist-debian-builds: bindist
    bindist-debian-build-include: linux_debian*-**
    bindist-debian-build-include: linux_ubuntu*-**
    bindist-debian-build-exclude: **

    See the `*-build-config` package manifest values in the manual for

  * New package-description and package-description-{file,type} package
    manifest values.

    Compared to the description* values, these can be used to provide a bpkg
    package-specific description, such as the recommended usage, configuration
    variables, etc. See the `description`, `package-description` package
    manifest values in the manual for details.

  * New changes-type package manifest value and type auto-detection, similar
    to description.

    See the `changes` package manifest value in the manual for details.

  * New --deorphan pkg-build option.

    This option can be used to replace orphaned packages (packages that no
    longer have the corresponding package available in the repository it came
    from) with the closest available package versions that satisfy all the

  * New --mask-repository* pkg-build options.

    These options allow pretending for the duration of the pkg-build command
    execution that the specified repository was removed as if by performing
    the rep-remove command.

  * New --dependent-exit pkg-drop option.

    This option causes the pkg-drop command to silently exit with the
    specified error code if attempting to drop dependent packages.

  * New --git-capabilities common option to override auto-detected git

    We now also assume the git repository protocol is smart if the HTTP
    response code is 401 (requires authentication).

  * curl is now used instead of wget as the default fetch program.

    We used to prefer wget 1.16 because it has --show-progress which results
    in nicer progress. But experience shows that wget is quite unreliable plus
    with bdep always using curl, it would be strange to use both curl and wget
    (and expecting the user to setup proxy, authentication, etc., for both).

bdep version 0.16.0

  * The bdep-deinit command can now deinitialize a package that still has

    In this case, the package is replaced with the closest available version
    from a repository. See also the new --no-fetch option in bdep-deinit(1).

  * New bdep-ci command --target-config and --package-config options to match
    the *-package-config package manifest values.

    Note that the existing --interactive|-i, --builds, and --build-config
    option values have been adjusted as well. See bdep-ci(1) for details.

  * The bdep-new command may now add `type` and `language` values to the newly
    created package manifests.

    These values can help determine the package type (executable, library) and
    language and are used, for example, by bpkg-pkg-bindist to produce correct
    binary distribution packages.

  * New `branch` suboption for the git vcs option in the bdep-new command.

    This suboption can be used to specify the initial branch name in the newly
    created repository.

  * New --sys-* options in the bdep-init and bdep-sync commands.

    This is a subset of the bpkg-pkg-build --sys-* options for controlling
    interaction with the system package manager. See the "System package
    manager query/installation support for Debian and Fedora" NEWS item in
    bpkg for background.

More information about the announce mailing list