[build2] build2 0.15.0 released

Boris Kolpackov boris at codesynthesis.com
Thu Jul 28 14:24:05 UTC 2022


We have released build2 toolchain version 0.15.0.

The overall theme of this release is more advanced functionality that
is becoming necessary as we start to use build2 to handle more complex
projects. Specifically, the package manager now supports a number of
more advanced dependency declaration features, including conditional
dependencies, dependency alternatives, and dependency configuration.
On the build system side we now have rule hints, support for dynamic
dependencies in ad hoc recipes, and the ability to save user metadata
in C/C++ libraries.

Another new development is the creation of the HOWTO repository with
practical advice on using build2 to achieve common tasks. At the time
of the release the repository contained 10 articles:

  https://github.com/build2/HOWTO/#readme

A note on backwards compatibility: this release cannot be upgraded to from
0.14.0 and has to be installed from scratch.

The corresponding NEWS file entries are included below with a more detailed
discussion of the major new features available in the release notes:

  https://build2.org/release/0.15.0.xhtml

This release adds one new standard pre-installed build system module,
autoconf (GNU Autoconf emulation for processing config.h.in files):

  https://github.com/build2/libbuild2-autoconf/

For background on standard pre-installed modules see:

  https://lists.build2.org/archives/users/2021-January/000862.html

All the information-querying commands in the package and project managers
as well as the structured result output and the info meta-operation in the
build system can now produce JSON output. This should make it easier to
integrate build2 with other tools, IDEs, etc (see the NEWS files below
for details).

This release also includes a large number of performance optimizations in
the build system which should especially help projects that make use of
heavily inter-dependent libraries (such as Boost). For example, on some
tests the up-to-date check times went down from 1.2s to 0.3s.

This release has been tested on Windows, Linux, Mac OS (including M1), and
FreeBSD with GCC, Clang (both vanilla and Apple's), MinGW GCC, and MSVC (14,
15, 16, and 17). There is also preliminary support for OpenBSD. A big thank
you to everyone who helped test the development snapshots and the release
candidate.

The following new build configurations have been added to the CI service:

  freebsd_13-clang_13.0

  linux_debian_11-gcc_12.1
  linux_debian_11-clang_14.0[_libc++]

  macos_12-clang_13.1          (Xcode 13.4 Clang 13.1.6)
  macos_12-gcc_12.1_homebrew

  windows_10-msvc_17.2
  windows_10-clang_14.0_llvm_msvc_17.2[_lld]
  windows_10-gcc_11.2_mingw_w64

  linux_debian_11-emcc_3.1.6   (Emscripten)

In addition, all the Linux configurations in the default class now include
system packages necessary for GUI development and run a mock X server via
Xvfb. All in all, there are now 66 build configurations. For the complete
list, see:

  https://ci.cppget.org/?build-configs

Installation instructions:

  https://build2.org/install.xhtml

Download directory:

  https://download.build2.org/0.15.0/

SHA256 checksums:

814c4f475b42749dab49c52549ef85899749c20e5c32276b281fd58dad54f47b *build2-install-0.15.0.sh
2556da043d11d56a58bb5b4a8279e6eddf87d81b70e72c0e808008d807195efb *build2-install-msvc-0.15.0.bat
40d266125e4c7fd35801ea5e8375a34d50b14cf8aaf36e21abf6dfafa081344d *build2-install-clang-0.15.0.bat
487b8aa29fc350413de0652ad7a04e605f8c0ca067d8a9765b4c213683af6e36 *build2-install-mingw-0.15.0.bat

b7df173dc1b7b48ab6e09d201112c0fcec7ba62b2b33789c5e17516e9850342c *build2-toolchain-0.15.0.tar.gz
5152f679daeb9627f9710c60ef88de1591c02097146268be2f5aea929d2837c4 *build2-toolchain-0.15.0.tar.xz

8aaaeea6b99c4c123be87f0c3906321c512418523530c3427d6802e81e62a150 *build2-baseutils-0.15.0-x86_64-windows.zip
a1adb7da12db9d88b0fbf528b29dc39b5568693fae0558d9866689263871063c *build2-mingw-0.15.0-x86_64-windows.tar.xz

Repository certificate fingerprint for cppget.org:

  70:64:FE:E4:E0:F3:60:F1:B4:51:E1:FA:12:5C:E0:B3:DB:DF:96:33:39:B9:2E:E5:C2:68:63:4C:A6:47:39:43

NEWS file entries:


build2 version 0.15.0
---------------------

  * Generated C/C++ headers and ad hoc sources are now updated during match.

    Specifically, all headers as well as ad hoc headers and sources are now
    treated by the cc::link_rule as if they had update=match unless explicit
    update=execute is specified (see below on the update operation-specific
    variable).

    This change should be transparent to most projects. For background and
    discussion of rare cases where you may wish to disable this, see:

    https://github.com/build2/HOWTO/blob/master/entries/handle-auto-generated-headers.md

  * Support for rule hints.

    A rule hint is a target attribute, for example:

    [rule_hint=cxx] exe{hello}: c{hello}

    Rule hints can be used to resolve ambiguity when multiple rules match the
    same target as well as to override an unambiguous match.

    In cc::link_rule we now support "linking" libraries without any sources or
    headers with a hint. This can be useful for creating "metadata libraries"
    whose only purpose is to convey metadata (options to use and/or libraries
    to link).

  * UTF-8 is now the default input/source character set for C/C++ compilation.

    Specifically, the cc module now passes the appropriate compiler option
    (/utf-8 for MSVC and -finput-charset=UTF-8 for GCC and Clang) unless a
    custom value is already specified (with /{source,execution}-charset for
    MSVC and -finput-charset for GCC and Clang).

    This change may trigger new compilation errors in your source code if
    it's not valid UTF-8 (such errors most commonly point into comments).
    For various ways to fix this, see:

    https://github.com/build2/HOWTO/blob/master/entries/convert-source-files-to-utf8.md

  * Project configuration variables are now non-nullable by default.

    A project configuration variable with the NULL default value is naturally
    assumed nullable, for example:

    config [string] config.libhello.fallback_name ?= [null]

    Otherwise, to make a project configuration nullable use the `null`
    variable attribute, for example:

    config [string, null] config.libhello.fallback_name ?= "World"

  * New $relative(<path>, <dir-path>) function.

  * New $root_directory(<path>) function.

  * New $size() function to get the size of string, path, dir_path.

  * New $size() function to get the size of a sequence (strings, paths, etc).

  * New $sort() function to sort a sequence (strings, paths, etc).

    The function has the following signature:

    $sort(<sequence> [, <flags>])

    The following flag is supported by all the overloads:

      dedup - in addition to sorting also remove duplicates

          Additionally, the strings overload also support the following flag:

      icase - sort ignoring case

    Note that on case-insensitive filesystems the paths and dir_paths
    overloads' order is case-insensitive.

  * New $config.origin() function for querying configuration value origin.

    Give a config.* variable name, this function returns one of `undefined`,
    `default`, `buildfile`, or `override`.

  * Recognition of -pthread as a special -l option in *.libs.

    For background, see:

    https://github.com/build2/HOWTO/blob/master/entries/link-pthread.md

  * The bin.whole (whole archive) value is now saved in generated pkg-config
    files.

  * Ability to customize header and library search paths in generated
    pkg-config files.

    Specifically, {cc,c,cxx}.pkgconfig.{include,lib} variables specify header
    (-I) and library (-L) search paths to use in the generated pkg-config
    files instead of the default install.{include,lib}. Relative paths are
    resolved as installation paths. For example:

    lib{Qt6Core}: cxx.pkgconfig.include = include/qt6/

  * Ability to save user metadata in C/C++ libraries, including in generated
    pkg-config files.

    For background and details, see:

    https://github.com/build2/HOWTO/blob/master/entries/convey-additional-information-with-exe-lib.md

  * Support for rule-specific search in immediate import.

    We can now nominate a rule to perform the rule-specific search (if
    required) using the rule_hint attribute. For example:

    import! [metadata, rule_hint=cxx.link] lib = libhello%lib{hello}

  * Support for dynamic dependencies in ad hoc recipes.

    Specifically, the `depdb` builtin now has the new `dyndep` command that
    can be used to extract dynamic dependencies from program output or a
    file. For example, from program output:

    obje{hello.o}: cxx{hello}
    {{
      s = $path($<[0])
      o = $path($>)

      poptions = $cxx.poptions $cc.poptions
      coptions = $cc.coptions $cxx.coptions

      depdb dyndep $poptions --what=header --default-type=h -- \
        $cxx.path $poptions $coptions $cxx.mode -M -MG $s

      diag c++ ($<[0])

      $cxx.path $poptions $coptions $cxx.mode -o $o -c $s
    }}

    Or, alternatively, from a file:

    t = $(o).t
    depdb dyndep $poptions --what=header --default-type=h --file $t -- \
      $cxx.path $poptions $coptions $cxx.mode -M -MG $s >$t

    The above depdb-dyndep commands will run the C++ compiler with the -M -MG
    options to extract the header dependency information, parse the resulting
    make dependency declaration (either from stdout or from file) and enter
    each header as a prerequisite of the obje{hello.o} target, as if they were
    listed explicitly. It will also save this list of headers in the auxiliary
    dependency database (hello.o.d file) in order to detect changes to these
    headers on subsequent updates. The --what option specifies what to call
    the dependencies being extracted in diagnostics. The --default-type option
    specifies the default target type to use for a dependency if its file name
    cannot be mapped to a target type.

    The above depdb-dyndep variant extracts the dependencies ahead of the
    compilation proper and will handle auto-generated headers (see the -MG
    option for details) provided we pass the header search paths where they
    could be generated with the -I options (passed as $poptions in the above
    example).

    If there can be no auto-generated dependencies or if they can all be
    listed explicitly as static prerequisites, then we can use a variant of
    the depdb-dyndep command that extracts the dependencies as a by-product of
    compilation. In this mode only the --file input is supported. For example
    (assuming hxx{config} is auto-generated):

    obje{hello.o}: cxx{hello} hxx{config}
    {{
      s = $path($<[0])
      o = $path($>)
      t = $(o).t

      poptions = $cxx.poptions $cc.poptions
      coptions = $cc.coptions $cxx.coptions

      depdb dyndep --byproduct --what=header --default-type=h --file $t

      diag c++ ($<[0])

      $cxx.path $poptions $coptions $cxx.mode -MD -MF $t -o $o -c $s
    }}

    Other options supported by the depdb-dyndep command:

    --format <name>

      Dependency format. Currently only the `make` dependency format is
      supported and is the default.

    --cwd <dir>

      Working directory used to complete relative dependency paths. This
      option is currently only valid in the --byproduct mode (in the normal
      mode relative paths indicate non-existent files).

    --adhoc

      Treat dynamically discovered prerequisites as ad hoc (so they don't end
      up in $<; only in the normal mode).

    --drop-cycles

      Drop prerequisites that are also targets. Only use this option if you
      are sure such cycles are harmless, that is, the output is not affected
      by such prerequisites' content.

    --update-{include,exclude} <tgt>|<pat>

      Prerequisite targets/patterns to include/exclude (from the static
      prerequisite set) for update during match (those excluded will be
      updated during execute). The order in which these options are specified
      is significant with the first target/pattern that matches determining
      the result. If only the --update-include options are specified, then
      only the explicitly included prerequisites will be updated. Otherwise,
      all prerequisites that are not explicitly excluded will be updated. If
      none of these options is specified, then all the static prerequisites
      are updated during match. Note also that these options do not apply to
      ad hoc prerequisites which are always updated during match.

    The common use-case for the --update-exclude option is to omit updating
    a library which is only needed to extract exported preprocessor options.
    Here is a typical pattern:

    import libs = libhello%lib{hello}

    libue{hello-meta}: $libs

    obje{hello.o}: cxx{hello} libue{hello-meta}
    {{
      s = $path($<[0])
      o = $path($>)

      lib_poptions = $cxx.lib_poptions(libue{hello-meta}, obje)
      depdb hash $lib_poptions

      poptions  = $cxx.poptions $cc.poptions $lib_poptions
      coptions  = $cc.coptions $cxx.coptions

      depdb dyndep $poptions --what=header --default-type=h \
        --update-exclude libue{hello-meta} -- \
          $cxx.path $poptions $coptions $cxx.mode -M -MG $s

      diag c++ ($<[0])

      $cxx.path $poptions $coptions $cxx.mode -o $o -c $s
    }}

    As another example, sometimes we need to extract the "common interface"
    preprocessor options that are independent of the the library type (static
    or shared). For example, the Qt moc compiler needs to "see" the C/C++
    preprocessor options from imported libraries if they could affect its
    input. Here is how we can implement this:

    import libs = libhello%lib{hello}

    libul{hello-meta}: $libs

    cxx{hello-moc}: hxx{hello} libul{hello-meta} $moc
    {{
      s = $path($<[0])
      o = $path($>[0])
      t = $(o).t

      lib_poptions = $cxx.lib_poptions(libul{hello-meta})
      depdb hash $lib_poptions

      depdb dyndep --byproduct --drop-cycles --what=header --default-type=h \
        --update-exclude libul{hello-meta} --file $t

      diag moc ($<[0])

      $moc $cc.poptions $cxx.poptions $lib_poptions \
        -f $leaf($s) --output-dep-file --dep-file-path $t -o $o $s
    }}

    Planned future improvements include support for the `lines` (list of
    files, one per line) input format in addition to `make` and support for
    dynamic targets in addition to prerequisites.

  * Support for specifying custom ad hoc pattern rule names.

    Besides improving diagnostics, this allows us to use such a name in the
    rule hints, for example:

    [rule_name=hello.link] exe{~'/(.*)/'}: obje{~'/\1/'}
    {{
      $cxx.path -o $path($>) $path($<[0])
    }}

    [rule_hint=hello] exe{hello}: obje{hello}

    obje{hello}: c{hello-c}

  * Ability to disfigure specific configuration variables.

    The new config.config.disfigure variable can be used to specify the list
    of variables to ignore when loading config.build (and any files specified
    in config.config.load), letting them to take on the default values. For
    example:

    $ b configure config.config.disfigure=config.hello.fancy

    Besides names, variables can also be specified as patterns in the
    config.<prefix>.(*|**)[<suffix>] form where `*` matches single
    component names (i.e., `foo` but not `foo.bar`), and `**` matches
    single and multi-component names. Currently only single wildcard (`*` or
    `**`) is supported.  Additionally, a pattern in the config.<prefix>(*|**)
    form (i.e., without `.` after <prefix>) matches config.<prefix>.(*|**)
    plus config.<prefix> itself (but not config.<prefix>foo).

    For example, to disfigure all the project configuration variables (while
    preserving all the module configuration variables; note quoting to prevent
    pattern expansion):

    $ b config.config.disfigure="'config.hello**'"

  * Ability to omit loading config.build.

    If the new config.config.unload variable is set to true, then omit loading
    the project's configuration from the config.build file. Note that the
    configuration is still loaded from config.config.load if specified. Note
    also that similar to config.config.load, only overrides specified on this
    project's root scope and global scope are considered.

  * Ability to match libul{} targets.

    The bin.libul rule picks, matches, and unmatches (if possible) a member
    for the purpose of making its metadata (for example, library's poptions,
    if it's one of the cc libraries) available.

  * Ability to get common interface options via ${c,cxx}.lib_poptions().

    Specifically, the output target type may now be omitted for utility
    libraries (libul{} and libu[eas]{}). In this case, only "common interface"
    options will be returned for lib{} dependencies. This is primarily useful
    for obtaining poptions to be passed to tools other than C/C++ compilers
    (for example, Qt moc).

  * Ability to control -I translation to -isystem or /external:I in
    ${c,cxx}.lib_poptions().

    See the function documentation for details.

  * New `update` operation-specific variable.

    This variable is similar to the already existing `clean` and `test`
    variables but besides the standard `true` and `false` values, it can also
    be set to `unmatch` (match but do not update) and `match` (update during
    match) and `execute` (update during execute, as is normally; this value is
    primarily useful if the rule has the `match` semantics by default).

    Note that the unmatch (match but do not update) and match (update during
    match) values are only supported by certain rules (and potentially only
    for certain prerequisite types).

    Additionally:

    - All the operation-specific variables are now checked for `false` as an
      override for the prerequisite-specific `include` variable. This can now
      be used to disable a prerequisite for update, for example:

      ./: exe{test}: update = false

    - Ad hoc Buildscript recipes now support update=unmatch|match.

    - The cc::link_rule now supports the `match` value for headers and ad hoc
      prerequisites. This can be used to make sure all the library headers are
      updated before matching any of its (or dependent's) object files.

  * New build.mode global scope variable.

    This variable signals the mode the build system may be running in. The two
    core modes are `no-external-modules` (bootstrapping of external modules is
    disabled, see --no-external-modules for details) and `normal` (normal
    execution). Other build system drivers may invent additional modes (for
    example, the bpkg `skeleton` mode; see "Package Build System Skeleton" in
    the package manager manual for details).

  * New cmdline value type for canned command lines.

    The Testscript and Buildscript languages now use the special cmdline value
    type for canned command lines. Specifically, the re-lexing after expansion
    now only happens if the expended value is of the cmdline type. See
    "Lexical Structure" in the Testscript manual for details.

  * The bash build system module now installs bash modules into
    bin/<project>.bash/ instead of bin/<project>/ to avoid clashes.

  * New --trace-{match,execute} options.

    These options can be used to understand which dependency chain causes
    matching or execution of a particular target. See b(1) for details.

  * JSON format support for the --structured-result option and the info meta
    operation.

    See b(1) for details.

  * Switch to using libpkg-config instead of libpkfconf for loading pkg-config
    files.


bpkg version 0.15.0
-------------------

  * New dependency declaration features:

    - Dependency groups, for example:

      depends: { libboost-any libboost-log libboost-uuid } ~1.77.0


    - Conditional dependencies, for example:

      depends: libposix-getopt ^1.0.0 ? ($cxx.target.class == 'windows')


    - Dependency alternatives, for example:

      depends: libmysqlclient >= 5.0.3 | libmariadb ^10.2.2


    - Reflected configuration variables, for example:

      depends: libposix-getopt ^1.0.0             \
               ? ($cxx.target.class == 'windows') \
	       config.hello.external_regex=true

      Or:

      depends: libmysqlclient >= 5.0.3 config.hello.db='mysql' | \
               libmariadb      ^10.2.2 config.hello.db='mariadb'


    - Dependency configuration, for example:

      depends:
      \
      libmariadb ^10.2.2
      {
        require
        {
          config.libmariadb.cache = true

          if ($cxx.target.class != 'windows')
            config.libmariadb.tls = true
        }
      }
      \

      Or:

      depends:
      \
      libmariadb ^10.2.2
      {
        prefer
        {
          config.libmariadb.cache = true

          config.libmariadb.buffer = ($config.libmariadb.buffer < 4096 \
                                      ? 4096                           \
                                      : $config.libmariadb.buffer)
        }

        accept ($config.libmariadb.buffer >= 4096)
      }
      \

    See the `depends` package manifest value in the manual for details.

    The implementation of these features led to bpkg becoming a special build
    system driver with repository metadata now containing the minimal subset
    of build system files for each package (called the package build system
    skeleton). See "Package Build System Skeleton" in the manual for details.

  * Support for JSON output in the bpkg-pkg-status command.

    See the --stdout-format option in bpkg-pkg-status(1) for details.

  * New --all, --all-pattern bpkg-pkg-drop options.

    These options can be used to drop all the held packages (in case of
    --all-pattern, limited to those that match a wildcard pattern).

  * New --keep-tmp common option.

    This option instructs bpkg not to remove its temporary directory at the
    end of the command execution and print its path if the verbosity level is
    2 or higher. This option is primarily useful for troubleshooting.


bdep version 0.15.0
-------------------

  * Support for JSON output in the bdep-status, bdep-config-list commands.

    See the --stdout-format option in bdep-status(1) and bdep-config(1)
    for details.

  * Support for submitting to a CI server packages with a non-standard
    version.

    See bdep-ci(1) for details.

  * Support for publishing packages with a non-standard version.

    See bdep-publish(1) for details.



More information about the users mailing list