[build2-announce] build2 0.16.0 released
Boris Kolpackov
boris at codesynthesis.com
Tue Jul 4 12:32:16 UTC 2023
We have released build2 toolchain version 0.16.0.
The two main areas of focus in this release are the system package manager
integration and more advanced functionality for more complex projects. The
system package manager support includes both consumption and production of
binary distribution packages, including uploading them from the CI builds.
This support required a large amount of ground work which produced
functionality useful in its own right, such as relocatable installations,
installation filtering, and installation manifests. The more advanced
functionality includes built-in support for the Objective-C/C++ and
Assembler with C Preprocessor (.S) compilation, dynamic target extraction
in addition to prerequisites, and support for buildfile importation.
There are also quite a few smaller features in this release, such as new
functions, commands, and builtins, as well as a large amount of maintenance
work, including the low verbosity diagnostics overhaul and support for
diagnostics buffering. We have also continued with our performance
optimizations in the build system and the package manager which should
especially help projects with a large number of dependencies.
A note on backwards compatibility: this release cannot be upgraded to from
0.15.0 and has to be installed from scratch.
The corresponding NEWS file entries are included below with a more detailed
discussion of the major new features available in the release notes:
https://build2.org/release/0.16.0.xhtml
This release has been tested on Windows, Linux (x86_64 and aarch64), Mac OS
(x86_64 and M1), and FreeBSD with GCC, Clang (both vanilla and Apple's),
MinGW GCC, and MSVC (14, 15, 16, and 17). A big thank you to everyone who
helped test the development snapshots and the release candidate.
The following new build configurations have been added to the CI service:
freebsd_12.4-clang_13.0
freebsd_13.2-clang_14.0
linux_debian_12-gcc_13
linux_debian_12-clang_15[_libc++]
linux_debian_12-clang_16[_libc++]
macos_12-clang_14.0 (Xcode 14.2, Apple Clang 14.0.0)
macos_13-clang_14.0 (Xcode 14.3.1, Apple Clang 14.0.3)
macos_13-clang_15.0 (Xcode 15.0, Apple Clang 15.0.0)
macos_13-gcc_13_homebrew
windows_10-msvc_17.5
windows_10-msvc_17.6 (LTSC)
windows_10-clang_15.0_msvc_msvc_17.6
windows_10-clang_16.0_llvm_msvc_17.6
windows_10-gcc_12.2_mingw_w64
Note that the Linux configurations now support building for both x86_64 and
aarch64 (ARM64).
There are also two new build classes: sys and bindist. The sys class contains
configurations that support the system package manager integration for package
consumption. They can be used to test against system-installed versions of
packages or satisfy dependencies that are not yet available from source. The
bindist class contains configurations that support the generation and upload
of binary distribution packages. Currently, this class contains the following
configurations:
linux_debian_11-gcc_10.2-bindist
linux_debian_12-gcc_12-bindist
linux_fedora_37-gcc_12.2-bindist
linux_fedora_38-gcc_13-bindist
All in all, there are now 91 build configurations. For the complete list, see:
https://ci.cppget.org/?build-configs
Installation instructions:
https://build2.org/install.xhtml
Download directory:
https://download.build2.org/0.16.0/
SHA256 checksums:
4c457f8365b83000e9ffb6de3924f47d3aee3722e0101996fdfab8adcfb80237 *build2-install-0.16.0.sh
1ae62f46fa2c23abd734d8497abae1909481d48ea74f924a7335887a17a42e21 *build2-install-msvc-0.16.0.bat
282cb2a409dcafb1e841664290b4ff55255852eba15b81b4e03f4e6c61a034a9 *build2-install-clang-0.16.0.bat
6c5fef712ebb90055e3d7745dade4125e779a55ba823dedfd063dff7aba3e8b8 *build2-install-mingw-0.16.0.bat
222690e4b22cd6e7afe900da495475310087b7c9bee8beee71a6d7f5635bbc88 *build2-toolchain-0.16.0.tar.gz
23793f682a17b1d95c80bbd849244735ed59a3e27361529aa4865d2776ff8adc *build2-toolchain-0.16.0.tar.xz
273afae47c738085fe35fd3ddf9eca830645fd8478d8c2f7a8d10c293197770d *build2-baseutils-0.16.0-x86_64-windows.zip
4a6f16a4d8f055c2caceee7437d47b84008c6f5031813d9a49b18f2686b1c12b *build2-mingw-0.16.0-x86_64-windows.tar.xz
Repository certificate fingerprint for cppget.org:
70:64:FE:E4:E0:F3:60:F1:B4:51:E1:FA:12:5C:E0:B3:DB:DF:96:33:39:B9:2E:E5:C2:68:63:4C:A6:47:39:43
NEWS file entries:
build2 version 0.16.0
---------------------
* Support for Objective-C/C++ compilation.
Specifically, the c and cxx modules now provide the c.objc and cxx.objcxx
submodules which can be loaded in order to register the m{}/mm{} target
types and enable Objective-C/C++ compilation in the c and cxx compile
rules. Note that c.objc and cxx.objcxx must be loaded after the c and cxx
modules, respectively, and while the m{}/mm{} target types are registered
unconditionally, compilation is only enabled if the C/C++ compiler
supports Objective-C/C++ for this target platform. Typical usage:
# root.build
#
using cxx
using cxx.objcxx
# buildfile
#
lib{hello}: {hxx cxx}{*}
lib{hello}: mm{*}: include = ($cxx.target.class == 'macos')
Note also that while there is support for linking Objective-C/C++
executables and libraries, this is done using the C/C++ compiler driver
and no attempt to automatically link any necessary Objective-C runtime
(such as -lobjc) is made. For details, refer to "Objective-C Compilation"
and "Objective-C++ Compilation" in the manual.
* Support for Assembler with C Preprocessor (.S) compilation.
Specifically, the c module now provides the c.as-cpp submodule which can
be loaded in order to register the S{} target type and enable Assembler
with C Preprocessor compilation in the c compile rule. For details, refer
to "Assembler with C Preprocessor Compilation" in the manual.
* Support for buildfile importation.
A project can now export buildfiles that can then be imported by other
projects. This mechanism is primarily useful for exporting target type
definitions and ad hoc rules.
Specifically, a project can now place *.build files into its build/export/
subdirectory (or *.build2 and build2/export/ in the alternative naming
scheme). Such files can then be imported by other projects as buildfile{}
targets. For example:
import thrift%buildfile{thrift-cxx}
While for other target types the semantics of import is to load the
project's export stub and return the exported target, for buildfile{} the
semantics is to source the imported buildfile at the point of importation.
Note that care must be taken when authoring exported buildfiles since they
will be sourced by other projects in unpredictable circumstances. In
particular, the import directive by default does not prevent sourcing the
same buildfile multiple times (neither in the same project nor in the same
scope). As a result, if certain parts must only be sourced once per
project (such as target type definitions), then they must be factored into
a separate buildfile (in build/export/) that is imported by the "main"
exported buildfile with the `once` attribute. For example, the above
thrift-cxx.build may contain:
import [once] thrift%buildfile{thrift-cxx-target-type}
See also "install Module" in the manual for details on the exported
buildfile installation.
* Support for defining explicit (as opposed to ad hoc) target groups.
A user-defined explicit target group must be derived from the group base
target type. If desired, it can be marked as "see-through", meaning that
when it is listed as a prerequisite of a target, the matching rule will
"see" its members, rather than the group itself. For example:
define [see_through] thrift_cxx: group
define thrift: file
thrift{*}: extension = thrift
exe{hello}: cxx{hello} thrift_cxx{data}
thrift_cxx{data}: thrift{data}
Explicit group members can be specified statically, injected by an ad hoc
rule, or extracted dynamically by the depdb-dyndep builtin (see the next
NEWS item). For example:
thrift_cxx{data}<{hxx cxx}{data_constants}>: thrift{data} # Static.
thrift_cxx{~'/(.+)/'}<{hxx cxx}{^'/\1_types/'}>: thrift{~'/\1/'} # Inject.
{{
depdb dyndep --dyn-target ... # Dynamic.
}}
* Support for dynamic target extraction in addition to prerequisites.
This functionality is enabled with the depdb-dyndep --dyn-target option.
If the recipe target is an explicit group (see the previous NEWS item),
then the dynamically extracted targets are added as its members.
Otherwise, the listed targets are added as ad hoc group members. In both
cases the dynamically extracted target is ignored if it is already
specified as a static member or injected by a rule. Note that this
functionality is not available in the --byproduct mode. See the
depdb-dyndep builtin options description for details.
* New `lines` depdb-dyndep dependency format in addition to `make`.
The `lines` format lists targets and/or prerequisites one per line. See
the depdb-dyndep builtin options description for details.
* Low verbosity diagnostics rework.
The low verbosity (level 1) rule diagnostics format has been adjusted to
include the output target where appropriate. The implementation has also
been redesigned to go through the uniform print_diag() API, including for
the `diag` pseudo-builtin in ad hoc recipes.
Specifically, the `diag` builtin now expects its arguments to be in one of
the following two forms (which correspond to the two print_diag() forms):
diag <prog> <l-target> <comb> <r-target>...
diag <prog> <r-target>...
If the `diag` builtin is not specified, the default diagnostics is now
equivalent to:
For update:
diag <prog> ($<[0]) -> $>
And for other operations:
diag <prog> $>
For details, see the print_diag() API description in diagnostics.hxx. See
also GitHub issue #40 for additional background/details.
* Buffering of diagnostics from child processes.
By default, unless running serially or --no-diag-buffer is specified,
diagnostics issued by child processes (compilers, etc) is buffered and
printed all at once after each child exits in order to prevent
interleaving. See also the new --[no-]diag-color options.
* New $path.posix_string() and $path.posix_representation() functions.
These functions are similar to $path.string() and $path.representation()
except that they always return the string/representation of a path in the
POSIX notation, that is, using forward slashes.
* New $regex.filter[_out]_{match,search}(<vals>, <pat>) functions.
The match versions return elements of a list that match (filter) or do not
match (filter_out) the regular expression. The search versions do the same
except for the search instead of match regex semantics.
* New $find(<sequence>, <value>), $find_index(<sequence>, <value>) functions.
The $find() function returns true if the sequence contains the specified
value. The $find_index() function returns the index of the first element
in the sequence that is equal to the specified value or $size(<sequence>)
if none is found. For string sequences, it's possible to request case-
insensitive comparison with a flag, for example:
if ($find ($values, 'foo', icase))
...
* New $integer_sequence(<begin>, <end>[, <step>]) function.
This function returns the list of uint64 integers starting from <begin>
(including) to <end> (excluding) with the specified <step> or 1 if
unspecified. For example:
hdr = foo.hxx bar.hxx baz.hxx
src = foo.cxx bar.cxx baz.cxx
assert ($size($hdr) == $size($src)) "hdr and src expected to be parallel"
for i: $integer_sequence(0, $size($hdr))
{
h = ($hdr[$i])
s = ($src[$i])
...
}
* New $is_a(<name>, <target-type>), $filter[_out](<names>, <target-types>)
functions.
$is_a() returns true if the <name>'s target type is-a <target-type>. Note
that this is a dynamic type check that takes into account target type
inheritance.
$filter[_out]() return names with target types which are-a (filter) or
not are-a (filter_out) one of <target-types>.
In particular, these functions are useful for filtering prerequisite
targets ($<) in ad hoc recipes and rules.
* Support for the hex notation for the uint64 type.
Specifically, now we can do:
x = [uint64] 0x0000ffff
cxx.poptions += "-DOFFSET=$x" # -DOFFSET=65535
cxx.poptions += "-DOFFSET=$string($x, 16)" # -DOFFSET=0xffff
cxx.poptions += "-DOFFSET=$string($x, 16, 8)" # -DOFFSET=0x0000ffff
Note that there is no hex notation support for the int64 (signed) type.
* Support for the `for` and `while` loops in Buildscript recipes and
Testscript.
For example:
for v: $values
...
end
cat values.txt | for -n v
...
end
while (!$regex.match(...))
...
end
See "Command-For" and "Command-While" in the Testscript manual for
details.
* New `find` builtin in Buildscript recipes and Testscript.
For example:
find gen/ -type f -name '*.?xx' | for -n f
...
end
See "find" in the Testscript manual for details.
* Improvements to escape sequence support.
In the double-quoted strings we now only do effective escaping of the
special [$("\] characters, line continuations, plus [)] for symmetry.
There is now support for "escape sequence expansion" in the $\X form where
\X can be any of the C/C++ simple escape sequences (\n, \t, etc) plus \0
(which in C/C++ is an octal escape sequence). For example:
info "foo$\n$\tbar$\n$\tbaz"
Will print:
buildfile:1:1: info: foo
bar
baz
* New include_arch installation location and the corresponding
config.install.include_arch configuration variable.
This location is meant for architecture-specific files, such as
configuration headers. By default it's the same as the standard include
location but can be configured by the user to a different value (for
example, /usr/include/x86_64-linux-gnu/) for platforms that support
multiple architectures from the same installation location. This is how
one would normally use it from a buildfile:
# The generated configuration header may contain target architecture-
# specific information so install it into include_arch/ instead of
# include/.
#
h{*}: install = include/libhello/
h{config}: install = include_arch/libhello/
* Support for installation filtering.
While project authors determine what gets installed at the buildfile
level, the users of the project can now further filter the installation
using the config.install.filter variable. For details, see "Installation
Filtering" in the manual.
* Support for relocatable installations.
A relocatable installation can be moved to a directory other than its
original installation location. To request a relocatable installation, set
the config.install.relocatable variable to true. For details, see
"Relocatable Installation" in the manual.
* Support for installation manifest.
During the install operation, the config.install.manifest variable can be
set to a file path (or `-`) in order to write the information about all
the filesystem entries being installed into the specified file (or
stdout). The format of the installation manifest is "JSON lines". For
details, see the config.install.manifest variable documentation in the
install module.
* Ability to remap paths in source distributions.
The dist target-specific variable can now specify a path besides true or
false. This path is the "imaginary" source location which is used to
derive the corresponding distribution location. This location can be
either a directory path (to remap with the same file name) or a file path
(to remap with a different name). If the path is relative, then it is
treated relative to the target directory. Note that to make things less
error-prone, simple paths without any directory separators are not allowed
(use ./<name> instead).
Note that if multiple targets end up with the same source location, the
behavior is undefined and no diagnostics is issued. Note also that such
remapping has naturally no effect in the bootstrap distribution mode.
* The in.substitution variable has been renamed to in.mode.
The original name is still recognized for backwards compatibility.
* Ability to specify `in` rule substitutions as key-value pairs.
See "in Module" in the manual for details.
* New public/private variables model.
Now unqualified variables are project-private and can be typed, meaning
that a value assigned to a variable with such a name anywhere within the
project will have this type. For example:
[uint64] priority = [null]
[uint64] stack_size = [null]
priority = 1 # Ok.
stack_size = abc # Error.
Besides the type, variable attributes can specify visibility (project by
default) and overridability (false by default). For example:
thread{*}:
{
[uint64, visibility=target] priority = [null]
[uint64, visibility=target] stack_size = [null]
}
thread{foo}: priority = 1 # Ok.
priority = 1 # Error.
* Support for post hoc prerequisites.
Unlike normal and ad hoc prerequisites, a post hoc prerequisite is built
after the target, not before. It may also form a dependency cycle together
with normal/ad hoc prerequisites. In other words, all this form of
dependency guarantees is that a post hoc prerequisite will be built if its
dependent target is built.
A canonical example where this can be useful is a library with a plugin:
the plugin depends on the library while the library would like to make
sure the plugin is built whenever the library is built so that programs
that link the library can be executed without having to specify explicit
dependency on the plugin (at least for the dynamic linking case):
lib{hello}: ...
lib{hello-plugin}: ... lib{hello}
libs{hello}: libs{hello-plugin}: include = posthoc
Note that there is no guarantee that post hoc prerequisites will be built
before the dependents of the target "see" it as built. Rather, it is
guaranteed that post hoc prerequisites will be built before the end of the
overall build (more precisely, before the current operation completes).
As a result, post hoc prerequisites should not be relied upon if the
result (for example, a source code generator) is expected to be used
during build (more precisely, within the same operation).
Note also that the post hoc semantics is not the same as order-only in
GNU make. In fact, it is an even more "relaxed" form of dependency.
Specifically, while order-only prerequisite is guaranteed to be built
before the target, post hoc prerequisite is only guaranteed to be built
before the end of the overall build.
* Support for dumping build system state in the JSON format.
The new --dump-format option can be used to select the desired format.
Its valid values are `buildfile` and `json-v0.1`. For details on the JSON
dump format see "Appendix A - JSON Dump Format" in the manual.
* Change to the --dump option semantics.
This option now recognizes two additional values: `match-pre` and
`match-post` to dump the state of pre/post-operations. The `match` value
now only triggers dumping of the main operation.
* New --dump-scope and --dump-target options to limit --dump output.
* New --load-only option in addition to --match-only.
This option has the effect of loading all the subdirectory buildfiles that
are not explicitly included and is primarily useful in combination with
--dump.
* Quoted/display target names in the JSON structured result are now
consistent with the JSON dump.
Specifically, before we had `target` (display) and `quoted_target` and now
we have `target` (quoted) and `display_target`. Note that this is a
backwards-incompatible change.
* The dist meta-operation no longer invokes the install program.
This results in a substantial speedup, especially on Windows. The use of
install (or another install-like program) can still be forced with
explicit config.dist.cmd=install.
* Clang -Wunqualified-std-cast-call warning was remapped to -Wextra.
Clang 15 introduced the -Wunqualified-std-cast-call warning which warns
about unqualified calls to std::move() and std::forward() (because they
can be "hijacked" via ADL). Surprisingly, this warning is enabled by
default, as opposed to with -Wextra or at least -Wall. It has also proven
to be quite disruptive, causing a large number of warnings in a large
number of packages. So we have "remapped" it to -Wextra for now and in the
future may "relax" it to -Wall and potentially to being enabled by
default. See GitHub issue #259 for background and details.
bpkg version 0.16.0
-------------------
* System package manager query/installation support for Debian and Fedora
(and alike).
The pkg-build command will now query (unless --sys-no-query is specified)
the system package manager on Debian (and alike, such as Ubuntu) and
Fedora (and alike, such as RHEL) for versions of packages that are
specified as coming from the system (the sys: scheme). For example, if
running the following command on one of these distributions:
bpkg build hello ?sys:libsqlite3
Then pkg-build will query the system package manager for the installed
version of libsqlite3 and fail if none is present.
Additionally, if --sys-install is specified, pkg-build will attempt to
install such packages if not present but available from the system package
repository.
Other relevant options include --sys-yes, --sys-no-fetch, --sys-no-stub,
and --sys-sudo. See bpkg-pkg-build(1) for details.
See also the `*-{name, version, to-downstream-version}` package manifest
values in the manual for details on the bpkg to distribution package name
and version mapping.
* Binary distribution package generation support for Debian and Fedora (and
alike).
The new pkg-bindist command can be used to automatically generate binary
distribution packages from bpkg packages for Debian (and alike, such as
Ubuntu), Fedora (and alike, such as RHEL), and for other operating systems
as installation archives. For Debian and Fedora, dependencies can be
satisfied with system packages, bpkg packages, or bundled. See
bpkg-pkg-bindist(1) for details.
* Package build configuration support (*-build-config manifest values).
A package can now customize in its manifest the build configuration used
by automated build bots. This includes specifying configuration variable
values, forcing specific versions of dependencies, satisfying dependencies
with system packages, and enabling/disabling build bot steps. For example:
# Test with extras enabled.
#
extras-build-config: config.libfoo.extra=true
# Test with system-installed libsqlite3.
#
system-builds: sys
system-build-config: ?sys:libsqlite3
# Enable Debian binary distribution generation and upload.
#
bindist-debian-builds: bindist
bindist-debian-build-include: linux_debian*-**
bindist-debian-build-include: linux_ubuntu*-**
bindist-debian-build-exclude: **
bindist-debian-build-config:
\
+bpkg.bindist.debian:
+bbot.bindist.upload:
\
See the `*-build-config` package manifest values in the manual for
details.
* New package-description and package-description-{file,type} package
manifest values.
Compared to the description* values, these can be used to provide a bpkg
package-specific description, such as the recommended usage, configuration
variables, etc. See the `description`, `package-description` package
manifest values in the manual for details.
* New changes-type package manifest value and type auto-detection, similar
to description.
See the `changes` package manifest value in the manual for details.
* New --deorphan pkg-build option.
This option can be used to replace orphaned packages (packages that no
longer have the corresponding package available in the repository it came
from) with the closest available package versions that satisfy all the
constraints.
* New --mask-repository* pkg-build options.
These options allow pretending for the duration of the pkg-build command
execution that the specified repository was removed as if by performing
the rep-remove command.
* New --dependent-exit pkg-drop option.
This option causes the pkg-drop command to silently exit with the
specified error code if attempting to drop dependent packages.
* New --git-capabilities common option to override auto-detected git
capabilities.
We now also assume the git repository protocol is smart if the HTTP
response code is 401 (requires authentication).
* curl is now used instead of wget as the default fetch program.
We used to prefer wget 1.16 because it has --show-progress which results
in nicer progress. But experience shows that wget is quite unreliable plus
with bdep always using curl, it would be strange to use both curl and wget
(and expecting the user to setup proxy, authentication, etc., for both).
bdep version 0.16.0
-------------------
* The bdep-deinit command can now deinitialize a package that still has
dependents.
In this case, the package is replaced with the closest available version
from a repository. See also the new --no-fetch option in bdep-deinit(1).
* New bdep-ci command --target-config and --package-config options to match
the *-package-config package manifest values.
Note that the existing --interactive|-i, --builds, and --build-config
option values have been adjusted as well. See bdep-ci(1) for details.
* The bdep-new command may now add `type` and `language` values to the newly
created package manifests.
These values can help determine the package type (executable, library) and
language and are used, for example, by bpkg-pkg-bindist to produce correct
binary distribution packages.
* New `branch` suboption for the git vcs option in the bdep-new command.
This suboption can be used to specify the initial branch name in the newly
created repository.
* New --sys-* options in the bdep-init and bdep-sync commands.
This is a subset of the bpkg-pkg-build --sys-* options for controlling
interaction with the system package manager. See the "System package
manager query/installation support for Debian and Fedora" NEWS item in
bpkg for background.
More information about the announce
mailing list