[build2] build2 0.14.0 released
Boris Kolpackov
boris at codesynthesis.com
Tue Oct 19 11:50:37 UTC 2021
We have released build2 toolchain version 0.14.0. The main focus of
this release is support for build-time dependencies and the host/target
configuration split that it necessitates. This support required a large
amount of ground work which produced functionality useful in its own
right, such as hermetic builds and configuration linking. Another notable
new feature is ad hoc regex pattern rules.
A note on backwards compatibility: this release cannot be upgraded to from
0.13.0 and has to be installed from scratch.
The corresponding NEWS file entries are included below with a more detailed
discussion of the major new features available in the release notes:
https://build2.org/release/0.14.0.xhtml
Note also that with the release we have introduced a notion of "standard
pre-installed build system modules" that the user can assume are available
out of the box in a regular build2 toolchain installation. In particular,
the "official" install scripts will build and install such modules unless
explicitly requested not to. See the earlier announcement for details:
https://lists.build2.org/archives/users/2021-January/000862.html
This release has been tested on Windows, Linux, Mac OS (including M1), and
FreeBSD with GCC, Clang (both vanilla and Apple's), MinGW GCC, and MSVC (14,
15, and 16). A big thank you to everyone who helped test the development
snapshots and the release candidate.
The following new build configurations have been added to the CI service:
freebsd_13.0-clang_11.0
linux_debian_10-gcc_10.2
linux_debian_10-gcc_11.2
linux_debian_10-clang_11.0[_libc++]
linux_debian_10-clang_12.0[_libc++]
linux_debian_11-clang_13.0[_libc++]
macos_11-clang_12.0 (Xcode 12.5.1 Clang 12.0.5)
macos_11-clang_13.0 (Xcode 13 Clang 13.0.0)
macos_11-gcc_11.2_homebrew
windows_10-msvc_16.9
windows_10-msvc_16.11
windows_10-clang_12.0_msvc_msvc_16.11
windows_10-clang_13.0_llvm_msvc_16.11[_lld]
windows_10-gcc_10.2_mingw_w64
linux_debian_10-emcc_2.0.25 (Emscripten)
In addition, there are now optimized NDEBUG builds for select configurations.
All in all, there are now 58 build configurations. For the complete list, see:
https://ci.cppget.org/?build-configs
Installation instructions:
https://build2.org/install.xhtml
Download directory:
https://download.build2.org/0.14.0/
SHA256 checksums:
f2e0795fda1bdc6b6ea4d2fc5917469725c20962bb1f6672c8d2462d76b3a7db *build2-install-0.14.0.sh
3f54d6464924eee53874cdd377c66b793a1ebd9998d1cc136d9004bfb6ad1ce4 *build2-install-msvc-0.14.0.bat
3a697a30853beec9b5b81730b564391d33f088b5bbc0d8b9cfe755b586a2b622 *build2-install-clang-0.14.0.bat
3641b2511fc6029dd48316639c27d10c48b138b5d6524353858ff754d128a200 *build2-install-mingw-0.14.0.bat
36781e9fce483e431dcf1d4d73dbef267b1712fb8679dbe968334aae6242237f *build2-toolchain-0.14.0.tar.gz
18efc6b2d41498f7516e7a8a5c91023f6182c867d423792398390dd0c004cfdd *build2-toolchain-0.14.0.tar.xz
8f1ce283175aa6e527ceac03b97a4f78b59684495bf0b10a3a3a440e7e4ffbab *build2-baseutils-0.14.0-x86_64-windows.zip
f894814c878298eecf87c0d6a722beb2a45189064a2a8a7db5b150db7583ba94 *build2-mingw-0.14.0-x86_64-windows.tar.xz
Repository certificate fingerprint for cppget.org:
70:64:FE:E4:E0:F3:60:F1:B4:51:E1:FA:12:5C:E0:B3:DB:DF:96:33:39:B9:2E:E5:C2:68:63:4C:A6:47:39:43
NEWS file entries:
build2 version 0.14.0
---------------------
* Support for hermetic build configurations.
Hermetic build configurations save environment variables that affect the
project along with other project configuration in the config.build file.
These saved environment variables are then used instead of the current
environment when performing operations on the project, thus making sure
the project "sees" exactly the same environment as during configuration.
The built-in ~host and ~build2 configurations are now hermetic.
Hermetic configuration support is built on top of the lower-level
config.config.environment configuration variable which allows us to save a
custom set of environment variables/values.
As part of this work we now also track changes to the environment in non-
hermetic configurations and automatically rebuild affected targets.
See "Hermetic Build Configurations" in the manual for details.
* Support for ad hoc regex pattern rules.
An ad hoc pattern rule consists of a pattern that mimics a dependency
declaration followed by one or more recipes. For example:
exe{~'/(.*)/'}: cxx{~'/\1/'}
{{
$cxx.path -o $path($>) $path($<[0])
}}
If a pattern matches a dependency declaration of a target, then the recipe
is used to perform the corresponding operation on this target. For
example, the following dependency declaration matches the above pattern
which means the rule's recipe will be used to update this target:
exe{hello}: cxx{hello}
While the following declarations do not match the above pattern:
exe{hello}: c{hello} # Type mismatch.
exe{hello}: cxx{howdy} # Name mismatch.
On the left hand side of `:` in the pattern we can have a single target or
an ad hoc target group. The single target or the first (primary) ad hoc
group member must be a regex pattern (~). The rest of the ad hoc group
members can be patterns or substitutions (^). For example:
<exe{~'/(.*)/'} file{^'/\1.map/'}>: cxx{~'/\1/'}
{{
$cxx.path -o $path($>[0]) "-Wl,-Map=$path($>[1])" $path($<[0])
}}
On the right hand side of `:` in the pattern we have prerequisites which
can be patterns, substitutions, or non-patterns. For example:
<exe{~'/(.*)/'} file{^'/\1.map/'}>: cxx{~'/\1/'} hxx{^'/\1/'} hxx{common}
{{
$cxx.path -o $path($>[0]) "-Wl,-Map=$path($>[1])" $path($<[0])
}}
Substitutions on the left hand side of `:` and substitutions and non-
patterns on the right hand side are added to the dependency declaration.
For example, given the above rule and dependency declaration, the
effective dependency is going to be:
<exe{hello} file{hello.map>: cxx{hello} hxx{hello} hxx{common}
Similar to ad hoc recipes, ad hoc rules can be written in Buildscript or
C++.
* Support for regex patterns in target type/pattern-specific variables.
This is in addition to the already supported path patterns. For example:
hxx{*}: x = y # path pattern
hxx{~/.*/}: x = y # regex pattern
* New pre-defined semantics for the config.<project>.develop variable.
This variable allows a project to distinguish between development and
consumption builds. While normally there is no distinction, sometimes a
project may need to provide additional functionality during development.
For example, a source code generator which uses its own generated code in
its implementation may need to provide a bootstrap step from the pre-
generated code. Normally, such a step is only needed during development.
If used, this variable should be explicitly defined by the project with
the bool type and the false default value. For example:
config [bool] config.hello.develop ?= false
See "Project Configuration" in the manual for details.
* Support for warning suppression from external C/C++ libraries.
This is implemented by defining a notion of a project's internal scope and
automatically translating header search path options (-I) exported by
libraries that are outside of the internal scope to appropriate "external
header search path" options (-isystem for GCC/Clang, /external:I for MSVC
16.10 and later). In the future this functionality will be extended to
side-building BMIs for external module interfaces and header units.
Note that this functionality is not without limitations and drawbacks and,
if needed, should be enabled explicitly. See the "Compilation Internal
Scope" section in the manual for details.
* C++20 modules support in GCC 11 using the module mapper.
This support covers all the major C++20 modules features including named
modules, module partitions (both interface and implementation), header
unit importation, and include translation. All of these features are also
supported in libraries, including consumption of installed libraries with
information about modules and importable headers conveyed in pkg-config
files. Module interface-only libraries are also supported.
Note that one area that is not yet well supported (due to module mapper
limitations) is auto-generated headers. Also note that as of version 11,
support for modules in GCC is still experimental and incomplete.
* Support for automatic DLL symbol exporting.
It is now possible to automatically generate a .def file that exports all
symbols from a Windows DLL. See "Automatic DLL Symbol Exporting" in the
manual for details.
* Initial Emscripten compiler support.
- Target: wasm32-emscripten (wasm32-unknown-emscripten).
- Compiler id: clang-emscripten (type clang, variant emscripten, class
gcc).
- Ability to build executables (.js plus .wasm) and static libraries (.a).
Set executable bit on the .js file (so it can be executed with a
suitable binfmt interpreter). Track the additional .worker.js file if
-pthread is specified.
- Default config.bin.lib for wasm32-emscripten is static instead of both.
- Full C++ exception support is enabled by default unless disabled
explicitly by the user with -s DISABLE_EXCEPTION_CATCHING=1|2.
- The bin module registers the wasm{} target type for wasm32-emscripten.
* New string functions: $string.trim(), $string.lcase(), $string.ucase().
* Support for test runners (config.test.runner).
Support for test timeouts (config.test.timeout).
See "test Module" in the manual for details.
* New <version> install directory substitution in addition to <project>.
New config.install.etc variable with the data_root/etc/ default.
See the "install Module" chapter in the manual for details.
* Support for fallback substitution in the in module (in.null variable).
See "in Module" in the manual for details.
* New export pseudo-builtin that allows adding/removing variables to/from
the current scope's commands execution environment.
See the Testscript manual for details.
* New ad hoc recipe depdb preamble.
The Buildscript language now provides a new pseudo-builtin, depdb, that
allows tracking of custom auxiliary dependency information. Invocations of
this builtin should come before any recipe commands and are collectively
called the depdb preamble. Non-pure functions can now only be called as
part of this preamble. For example:
file{output}: file{input} $foo
{{
diag foo $>
depdb env FOO # foo uses the FOO environment variable
$foo $path($<[0]) >$path($>)
}}
* New ${c,cxx}.deduplicate_export_libs() functions.
These functions deduplicate interface library dependencies by removing
libraries that are also interface dependencies of other libraries on the
specified list. This can lead to a significantly better build performance
for heavily interface-interdependent library families (for example, like
Boost). Typical usage:
import intf_libs = ...
import intf_libs += ...
...
import intf_libs += ...
intf_libs = $cxx.deduplicate_export_libs($intf_libs)
* New ${c,cxx}.find_system_{header,library}() functions.
These functions can be used to detect the presence of a header/library in
one of the system header/library search directories.
* New ${c,cxx}.lib_{poptions,libs,rpaths}() and $cxx.obj_modules() functions.
These functions can be used to query library metadata for options and
libraries that should be used when compiling/linking dependent targets,
similar to how cc::{compile,link}_rule do it. With this support it should
be possible to more or less re-create their semantics in ad hoc recipes.
* Support for suppressing duplicates when extracting library options and
linking libraries in cc::{compile,link}_rule.
* The cxx.std=latest value has been mapped to c++2b for Clang 13 or later
and to /std:c++20 for MSVC 16.11 or later.
* Support for LTO parallelization during linking in GCC and Clang.
GCC >= 10 and Clang >= 4 support controlling the number of LTO threads
used during linking. The cc::link_rule now uses the build system scheduler
to automatically allocate up to the number of available threads to the GCC
or Clang linker processes when -flto=auto or -flto=thin is specified,
respectively.
* /Zc:__cplusplus is now passed by default starting from MSVC 15.7.
This can be overridden by passing a variant of this option as part of the
compiler mode options.
* Support for disabling clean through target-prerequisite relationships.
The current semantics is to clean any prerequisites that are in the same
project (root scope) as the target and it may seem more natural to rather
only clean prerequisites that are in the same base scope. While it's often
true for simple projects, in more complex cases it's not unusual to have
common intermediate build results (object files, utility libraries, etc)
residing in the parent and/or sibling directories. With such arrangements,
cleaning only in base may leave such intermediate build results laying
around since there is no reason to list them as prerequisites of any
directory aliases.
So we clean in the root scope by default but now any target-prerequisite
relationship can be marked not to trigger a clean with the clean=false
prerequisite-specific value. For example:
man1{cli}: exe{cli}: clean = false # Don't clean the man generation tool.
* exe{} targets are no longer installed through target-prerequisite
relationships of file-based targets.
Normally, an exe{} that is listed as a prerequisite of a file-based target
is there to be executed (for example, to generate that target) and not to
trigger its installation (such an exe{} would typically be installed via
the ./ alias). This default behavior, however, can be overridden with the
install=true prerequisite-specific value. For example:
exe{foo}: exe{bar}: install = true # foo runs bar
* Consistently install prerequisites from any scope by default.
It is also now possible to adjust this behavior with the global
!config.install.scope override. Valid values for this variable are:
project -- only from project
bundle -- from bundle amalgamation
strong -- from strong amalgamation
weak -- from weak amalgamation
global -- from all projects (default)
* Variable names/components that start with underscore as well as variables
in the build, import, and export namespaces are now reserved by the build
system core. For example:
_x = 1 # error
x._y = 1 # error
build.x = 1 # error
* New int64 (signed 64-bit integer) and int64s (vector of such integers)
variable types.
* Default options files can now contain global variable overrides.
* Support for multiple -e options (scripts) in the sed builtin.
* The bin.lib.version variable no longer needs to include leading `@` for
platform-independent versions.
* The actualize mode of $path.normalize() is now provided by a separate
$path.actualize() function.
* New --options-file build system driver option that allows specifying
additional options in a file.
* New notion of bundle amalgamation which is defined as the outermost named
strong (source-based) amalgamation.
* Support for unseparated scope-qualified variable assignment and expansion.
For example, now the following:
foo/x = y
info $(foo/x)
Is equivalent to:
foo/ x = y
info $(foo/ x)
While this makes scope-qualified syntax consistent with target-qualified,
it also means that variable names that contain directory separators are
now effectively reserved.
* New bootstrap distribution mode (!config.dist.bootstrap=true).
In this mode the dist meta-operation does not load the project (but does
bootstrap it) and adds all the source files into the distribution only
ignoring files and directories that start with a dot. This mode is
primarily meant for situations where the project cannot (yet) be loaded
due to missing dependencies.
* Support for external build system modules that require bootstrap (that is,
loaded in bootstrap.build). See also the new --no-external-modules option.
* New file cache for intermediate build results.
The file cache is used to store intermediate build results, for example,
partially-preprocessed C/C++ translation units (those .i/.ii files). The
cache implementation to use is controlled by the new --file-cache option.
Its valid values are noop (no caching or compression) and sync-lz4 (no
caching with synchronous LZ4 on-disk compression; this is the default).
* New BUILD2_DEF_OPT environment variable that can be used to suppress
loading of default options files.
* New BUILD2_DEF_OVR environment variable that can be used to propagate
global variable overrides to nested build system invocations.
bdep version 0.14.0
-------------------
* Support for configuration types, configuration linking, and build-time
dependencies.
Configurations can now be linked with each other to allow a package to be
built in one configuration while its dependencies -- in one or more linked
configurations. This can be used to create a "base" configuration with
common dependencies that are shared between multiple configurations.
Configurations now also have types with the three predefined types being
`target` (the default) `host` (used for build-time project packages and/or
dependencies), and `build2` (used for build system modules). We can now
also have multiple default configurations if they are of different types.
This mechanism together with configuration linking is used to provide
separate configurations for build-time dependencies, for example, tools
that need to be executed or build system modules that need to be loaded
during the build.
If during initialization or synchronization a build-time dependency is
encountered and there is no build configuration of a suitable type
associated with the project, then the user is prompted to automatically
create and associate such a configuration.
New relevant commands: bdep-config-{link,unlink}.
Command with relevant changes (new options, etc): bdep-config-create,
bdep-init, bdep-sync.
See bdep-config(1) and bdep-sync(1) for details on this functionality.
See also the corresponding functionality in bpkg on which this support
is based.
* Project packages are now by default initialized for development.
Specifically, a package is configured with config.<package>.develop=true
unless an alternative value is specified on the command line. For example:
$ bdep init @install config.hello.develop=false
To change the development mode of an already initialized package, use
bdep-sync:
$ bdep sync @install config.hello.develop=false
See the build system manual for details on the config.*.develop variable
semantics.
* Configuration of project packages is now preserved during synchronization.
To reconfigure a package from scratch, use the new --disfigure bdep-sync
option. For example:
$ bdep sync config.hello.develop=false # develop=false (new)
$ bdep sync # develop=false (preserved)
$ bdep sync --disfigure # develop=true (default)
* The bdep-new command can now create multi-language C/C++ projects.
* The bdep-new command can now create binless (header-only) C libraries.
* The bdep-new command now generates smarter glue buildfiles that read
packages.manifest instead of using a directory wildcard.
* New --immediate|-i and --recursive|-r bdep-{update,clean} options.
These options can be used to additionally update or clean immediate or all
dependencies of the project packages, respectively.
* New --build-config bdep-ci option.
This option provides an easier way to specify the desired build
configuration or configuration pattern.
* New --interactive|-i bdep-ci option.
This option can be used to request an interactive CI session. In this mode
the CI service pauses the testing at the specified breakpoint and provides
the login information to the execution environment.
* Support for argument grouping in bdep-{sync,init}.
See bdep-argument-grouping(1) for details on argument grouping.
* New BDEP_DEF_OPT environment variable that can be used to suppress loading
of default options files.
bpkg version 0.14.0
-------------------
* Support for configuration types, configuration linking, and build-time
dependencies.
Configurations can now be linked with each other to allow a package to be
built in one configuration while its dependencies -- in one or more linked
configurations. This can be used to create a "base" configuration with
common dependencies that are shared between multiple configurations.
Configurations now also have types with the three predefined types being
`target` (the default) `host` (used for build-time dependencies), and
`build2` (used for build system modules). This mechanism together with
configuration linking is used to provide separate configurations for
build-time dependencies, for example, tools that need to be executed or
build system modules that need to be loaded during the build.
If during dependency resolution a build-time dependency is encountered and
there is no build configuration of a suitable type linked with the target
configuration, then a private configuration of the needed type is
automatically created and linked.
New relevant commands: bpkg-cfg-{link,unlink,info}.
Command with relevant changes (new options, etc): bpkg-cfg-create,
bpkg-pkg-{build,status}.
See bpkg-cfg-create(1) for details on this functionality.
* Tests, examples, and benchmark packages can now also be run-time or build-
time.
See the `tests`, `examples`, and `benchmarks` package manifest values for
details.
* Two new pre-defined automated build bot requirements: `bootstrap` (package
is a build system module that requires bootstrapping) and `host` (package
is normally used as build-time dependency and should be built in a host
configuration).
See the `requires` package manifest value for details.
* Configuration of an external package is now preserved between upgrades and
downgrades.
The same behavior for normal packages is still a TODO. The old behavior
can be achieved with the new --disfigure bpkg-pkg-build option that forces
a from-scratch reconfiguration.
* New `min-bpkg-version` repositories manifest value that allows specifying
the minimum supported bpkg version.
See also the new --min-bpkg-version bpkg-rep-create option.
* Change of the `build-email` package manifest value semantics.
Now build result notifications are only sent if this value is explicitly
specified (before we used to fallback to `email`).
* New --immediate|-i and --recursive|-r bpkg-pkg-{update,clean} options.
These options can be used to additionally update or clean immediate or all
dependencies of a package, respectively.
* New --all-pattern bpkg-pkg-{update,clean,test,install,uninstall} option.
This option can be used to perform the command on all the held packages
that match a wildcard pattern.
* New --rebuild-checksum bpkg-pkg-build option.
This option can be used to avoid rebuilds if the result of a dependency
resolution for a package has not changed.
* New --noop-exit bpkg-pkg-build option.
This option can be used to request a distinct exit code if the build is
a noop (performs no new package builds, upgrades, etc).
* New --output-{root,purge} bpkg-pkg-checkout and --checkout-{root,purge}
bpkg-pkg-build options.
* New --keep-config bpkg-pkg-disfigure option.
* New BPKG_DEF_OPT environment variable that can be used to suppress loading
of default options files.
More information about the users
mailing list