From boris at codesynthesis.com Thu Jul 2 12:46:32 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu, 2 Jul 2020 14:46:32 +0200 Subject: [build2] Installation of legal documentation (LICENSE, etc) In-Reply-To: References: <1426409928.351866.1590094726377@office.mailbox.org> Message-ID: Boris Kolpackov writes: > Matthew Krupcale writes: > > > Perhaps it would make sense to have a separate license{} target type > > which has its install path specified by e.g. config.install.license. > > That's a good idea, though I like the more general term 'legal', i.e., > legal{} and config.install.legal. > > Will try to add it before the release. Ok, this has been implemented[1] in the latest staged version so now you should be able to just add: config.install.legal=/usr/share/licenses// And get the desired result (and, yes, you really can write ``[2]). We are about a week from starting the release sequence so if you could try this out and confirm everything works as expected, that would be great. [1] https://git.build2.org/cgit/build2/commit/?id=7daa7916182025d737f439da5fe5d67fe1a2fb8c [2] https://git.build2.org/cgit/build2/commit/?id=83fc9f4287d24826abc8ea61e1524edf23dcd449 From mkrupcale at matthewkrupcale.com Fri Jul 3 00:22:00 2020 From: mkrupcale at matthewkrupcale.com (Matthew Krupcale) Date: Thu, 2 Jul 2020 20:22:00 -0400 Subject: [build2] Installation of legal documentation (LICENSE, etc) In-Reply-To: References: <1426409928.351866.1590094726377@office.mailbox.org> Message-ID: On Thu, Jul 2, 2020 at 8:46 AM Boris Kolpackov wrote: > > Ok, this has been implemented[1] in the latest staged version Thanks for implementing that. It simplifies build2 packages a bit on Fedora (and I suppose SUSE). > (and, yes, you really can write ``[2]). Is there a reason for using angle brackets to do the command-line variable substitution? Since it's on the shell, I can see how $project might be confusing. On the other hand, since < and > also have shell semantics (i.e. redirection), the argument must be shell quoted as well as indicated in [2]. So maybe quoting like b config.install.libexec='exec_root/lib/$project/' wouldn't be so bad. It would then be consistent with the buildfile variable syntax. > We are about a week from starting the release sequence Great, looking forward to it. > so if you could try this out and confirm everything works as expected, that would be great. Done[1], everything seems in order. [1] https://copr.fedorainfracloud.org/coprs/mkrupcale/build2/build/1516659/ > [2] https://git.build2.org/cgit/build2/commit/?id=83fc9f4287d24826abc8ea61e1524edf23dcd449 From mkrupcale at matthewkrupcale.com Wed Jul 15 15:02:37 2020 From: mkrupcale at matthewkrupcale.com (Matthew Krupcale) Date: Wed, 15 Jul 2020 11:02:37 -0400 Subject: [build2] Installation of legal documentation (LICENSE, etc) In-Reply-To: References: <1426409928.351866.1590094726377@office.mailbox.org> Message-ID: On Thu, Jul 2, 2020 at 8:22 PM Matthew Krupcale wrote: > > On Thu, Jul 2, 2020 at 8:46 AM Boris Kolpackov wrote: > > > > (and, yes, you really can write ``[2]). > > Is there a reason for using angle brackets to do the command-line > variable substitution? Since it's on the shell, I can see how $project > might be confusing. On the other hand, since < and > also have shell > semantics (i.e. redirection), the argument must be shell quoted as > well as indicated in [2]. > > So maybe quoting like > > b config.install.libexec='exec_root/lib/$project/' > > wouldn't be so bad. It would then be consistent with the buildfile > variable syntax. > I see you've updated the NEWS files to prepare for release now. Have you given any additional thought to using $project vs. CLI syntax? Best, Matthew From boris at codesynthesis.com Wed Jul 15 16:29:44 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed, 15 Jul 2020 18:29:44 +0200 Subject: [build2] Installation of legal documentation (LICENSE, etc) In-Reply-To: References: <1426409928.351866.1590094726377@office.mailbox.org> Message-ID: Sorry for the delay in getting back to you on this (and I am still going to comment on that re-link issue you've submitted, probably after the release though). Matthew Krupcale writes: > Is there a reason for using angle brackets to do the command-line > variable substitution? Since it's on the shell, I can see how $project > might be confusing. On the other hand, since < and > also have shell > semantics (i.e. redirection), the argument must be shell quoted as > well as indicated in [2]. > > So maybe quoting like > > b config.install.libexec='exec_root/lib/$project/' > > wouldn't be so bad. It would then be consistent with the buildfile > variable syntax. The fact that it would be consistent with the buildfile syntax is actually a problem since these values are persisted in config.build. If we were to use `$` then those values would have to be quoted which makes the whole thing quite hairy (and also confusing since it now looks like those are expanded during config.build loading, which is not the case). In fact, initially it was @project@ but that also would have required quoting. So in the end we've settled on . I think such an unusual syntax is a good choice since it's a very ad hoc kind of processing. The fact that the `<>` characters have shell meaning is actually a good thing since they are unlikely to be used as filesystem names (and are illegal in Windows paths). If you want to specify a config.install.* path that contains `<`, it will have to be escaped: b config.install.root='/tmp/</' Makes sense? From mkrupcale at matthewkrupcale.com Wed Jul 15 21:04:23 2020 From: mkrupcale at matthewkrupcale.com (Matthew Krupcale) Date: Wed, 15 Jul 2020 17:04:23 -0400 Subject: [build2] Installation of legal documentation (LICENSE, etc) In-Reply-To: References: <1426409928.351866.1590094726377@office.mailbox.org> Message-ID: On Wed, Jul 15, 2020 at 12:29 PM Boris Kolpackov wrote: > > Sorry for the delay in getting back to you on this No worries, I just wasn't sure if you saw that question in the midst of the larger email. > (and I am still going to comment on that re-link issue you've submitted, probably > after the release though). Yeah, that can certainly wait for a later release. > The fact that it would be consistent with the buildfile syntax is > actually a problem since these values are persisted in config.build. > If we were to use `$` then those values would have to be quoted which > makes the whole thing quite hairy (and also confusing since it now > looks like those are expanded during config.build loading, which is > not the case). In fact, initially it was @project@ but that also > would have required quoting. Doesn't the get substituted in config.install.* at libbuild2/install/init.cxx:proc_var during the install module init? So would the config.build actually persist the un-expanded variable or the expanded variable? > If you want to specify a config.install.* path that contains `<`, it will have to be escaped: > > b config.install.root='/tmp/</' I see. This also looks a bit weird since you don't have to escape the '>' here, so it's asymmetric. I suppose you'd have to do an analogous escape if '$' were used for the variable, but no need for the trailing bracket. From boris at codesynthesis.com Thu Jul 16 16:03:41 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu, 16 Jul 2020 18:03:41 +0200 Subject: [build2] Installation of legal documentation (LICENSE, etc) In-Reply-To: References: Message-ID: Matthew Krupcale writes: > Doesn't the get substituted in config.install.* at > libbuild2/install/init.cxx:proc_var during the install module init? Yes, but the expanded value is only used to initialize the corresponding install.* variable. The expansion is not written back to config.install.*. The nice thing about this setup is that now we can have a single set of config.install.* values (with ) in the amalgamation's config.build. Before we had to override those that contain the project name in each subproject's config.build. > I see. This also looks a bit weird since you don't have to escape the > '>' here, so it's asymmetric. The likelihood that someone would want a path with `<` in it was deemed so low that we went with the simplest escaping scheme to implement, which happens to be this. From boris at codesynthesis.com Thu Jul 16 16:51:06 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu, 16 Jul 2020 18:51:06 +0200 Subject: [build2] Release candidate for build2 0.13.0 available for testing Message-ID: We are planning to release build2 0.13.0 early next week with the latest staged version serving as a release candidate: https://build2.org/community.xhtml#stage If you have projects that use build2, we would encourage you to try it out and report regressions or any other issues (installation, etc). In particular, we would appreciate testing on Linux distributions that use musl libc (such as Alpine Linux, Void Linux, etc). There is also preliminary support for NetBSD if anyone is interested to try this out. Please note that this release cannot be upgraded to from 0.12.0 and a from- scratch installation is required (you can have multiple build2 versions side-by-side; just pick a different installation location for 0.13.0). From markus.pawellek at mailbox.org Thu Jul 16 18:36:54 2020 From: markus.pawellek at mailbox.org (Markus Pawellek) Date: Thu, 16 Jul 2020 20:36:54 +0200 (CEST) Subject: [build2] build2 not Compiling on Raspberry Pi ARMv7 Architecture Message-ID: <919513607.104412.1594924614664@office.mailbox.org> Hello, I am using a Raspberry Pi Model 3B with Raspbian and GCC version 8.3.0 for some home projects and I wanted to use build2 as the build system for my code. But the compilation of build2 fails with the following error messages. In file included from libbuild2/utility.cxx:12: ./libbuild2/target.hxx:1502:4: error: #error timestamp is not lock-free on this architecture # error timestamp is not lock-free on this architecture In file included from ./libbuild2/file.hxx:12, from build2/b.cxx:29: ./libbuild2/target.hxx:1502:4: error: #error timestamp is not lock-free on this architecture # error timestamp is not lock-free on this architecture In file included from libbuild2/prerequisite.cxx:7: ./libbuild2/target.hxx:1502:4: error: #error timestamp is not lock-free on this architecture # error timestamp is not lock-free on this architecture Is there any chance to get it to work on this platform? Thank you in advance. Greetings Markus From boris at codesynthesis.com Thu Jul 16 19:30:11 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu, 16 Jul 2020 21:30:11 +0200 Subject: [build2] build2 not Compiling on Raspberry Pi ARMv7 Architecture In-Reply-To: <919513607.104412.1594924614664@office.mailbox.org> References: <919513607.104412.1594924614664@office.mailbox.org> Message-ID: Markus Pawellek writes: > I am using a Raspberry Pi Model 3B with Raspbian > and GCC version 8.3.0 for some home projects I am also pretty sure you are using 32-bit Raspbian. > and I wanted to use build2 as the build system for my code. > But the compilation of build2 fails with the following error messages. > > In file included from libbuild2/utility.cxx:12: > ./libbuild2/target.hxx:1502:4: error: #error timestamp is not lock-free on this architecture > # error timestamp is not lock-free on this architecture The problem here is that on 32-bit ARM atomic 64-bit integer operations are not lock-free. There are three ways to fix this: 1. Run in the 64-bit mode. I've heard there is now 64-bit Raspbian. 2. Force use of non-lock-free atomics by defining the LIBBUILD2_ATOMIC_NON_LOCK_FREE macro (you will need to perform a custom bootstrap to pass the relevant -D option in all the places). This will also most likely result in pretty poor performance. 3. We could implement a 32-bit based spin-lock for this case but I am wondering if it's worth the complexity. From markus.pawellek at mailbox.org Thu Jul 16 20:23:22 2020 From: markus.pawellek at mailbox.org (Markus Pawellek) Date: Thu, 16 Jul 2020 22:23:22 +0200 (CEST) Subject: [build2] Interest in Development of NASM Module Message-ID: <326652268.104938.1594931002936@office.mailbox.org> Hello, I am very interested in the development of a build2 module providing the possibility of compiling and linking NASM code. For this, I have already taken a look inside the libbuild2-hello and libbuild2-rust modules, but the first one is not really explaining how to add functionality and the second one is not compiling on my platform. Therefore I wanted to go even further and tried to understand the builtin cc, c, and cxx modules inside the build2 repository. But the code here was a little bit too much for me and hard to understand. What would be the best starting point and what would I have to do to get an overview of the build2 source code and to get an idea of build2 module development. Is there already an existing documentation concerning module development? Is it even reasonable to try to develop an external module for NASM, would it be better to think about a builtin module, or would it even be better to not include it at all? Thank you in advance. Greetings Markus From markus.pawellek at mailbox.org Thu Jul 16 20:53:59 2020 From: markus.pawellek at mailbox.org (Markus Pawellek) Date: Thu, 16 Jul 2020 22:53:59 +0200 (CEST) Subject: [build2] build2 not Compiling on Raspberry Pi ARMv7 Architecture In-Reply-To: References: <919513607.104412.1594924614664@office.mailbox.org> Message-ID: <626545930.105046.1594932839402@office.mailbox.org> Boris Kolpackov wrote: > Markus Pawellek writes: > > > I am using a Raspberry Pi Model 3B with Raspbian > > and GCC version 8.3.0 for some home projects > > I am also pretty sure you are using 32-bit Raspbian. You are completely right. > The problem here is that on 32-bit ARM atomic 64-bit integer operations > are not lock-free. There are three ways to fix this: > > 1. Run in the 64-bit mode. I've heard there is now 64-bit Raspbian. > > 2. Force use of non-lock-free atomics by defining the > LIBBUILD2_ATOMIC_NON_LOCK_FREE macro (you will need to perform > a custom bootstrap to pass the relevant -D option in all the places). > This will also most likely result in pretty poor performance. Thank you. I will take a look as soon as possible into those alternatives. I do not know very much about the actual implementation of build2 but with respect to the second alternative, even if it would suffer from a poor performance, it may not be a dramatic problem, because in my opinion on a Raspberry Pi one is typically dealing with small code projects. > 3. We could implement a 32-bit based spin-lock for this case but I am > wondering if it's worth the complexity. No, it is definitely not worth the complexity, I think. I only wanted to get some information concerning the error message and maybe a workaround. So for now, I have everything I need ;) From markus.pawellek at mailbox.org Thu Jul 16 21:45:13 2020 From: markus.pawellek at mailbox.org (Markus Pawellek) Date: Thu, 16 Jul 2020 23:45:13 +0200 (CEST) Subject: [build2] Custom Configuration Variables for the Config Module Message-ID: <148206719.105178.1594935914055@office.mailbox.org> Hello, currently and as an exercise, I am porting the build system of the GLFW library from CMake to build2 for Linux platforms. Due to simplicity, I have only done this for X11 platforms without the possibility of linking to the Vulkan library. But I would like to add support for Vulkan and Wayland as described in the official CMake files by the GLFW repository. Both of these alternatives should be choose-able from the outside via the config module of build2. For this, I wanted to provide custom configuration variables, like 'config.graphics.protocol = wayland / x11' and 'config.graphics.api = opengl / vulkan'. Using only the syntax of the buildfile and the functionality of build2 itself without the config module, I was able to do this by always providing the respective variables as command-line argument to 'b' as described in the documentation for non-persistent builds. Using the config module, manually adding custom variables to the 'config.build' file of my project was working. But the moment I changed the configuration with the command-line tools, all custom variables vanished. Is there already a better workaround to provide such custom variables? Otherwise, would it be possible to provide this functionality? In this case, maybe it would even be useful to have the same mechanism for custom configuration variables as for the already given variable pairs, such as 'config.cxx.poptions' and 'cxx.poptions'. Thank you in advance. - [GLFW Library](https://github.com/glfw/glfw) - [build2-glfw](https://github.com/lyrahgames/build2-glfw) Greetings Markus From markus.pawellek at mailbox.org Thu Jul 16 22:57:35 2020 From: markus.pawellek at mailbox.org (Markus Pawellek) Date: Fri, 17 Jul 2020 00:57:35 +0200 (CEST) Subject: [build2] Custom Configuration Variables for the Config Module In-Reply-To: <148206719.105178.1594935914055@office.mailbox.org> References: <148206719.105178.1594935914055@office.mailbox.org> Message-ID: <1763585766.105252.1594940255678@office.mailbox.org> Markus Pawellek wrote: > currently and as an exercise, I am porting the build system > of the GLFW library from CMake to build2 for Linux platforms. > Due to simplicity, I have only done this for X11 platforms > without the possibility of linking to the Vulkan library. > But I would like to add support for Vulkan and Wayland as > described in the official CMake files by the GLFW repository. > Both of these alternatives should be choose-able from the > outside via the config module of build2. > > For this, I wanted to provide custom configuration variables, > like 'config.graphics.protocol = wayland / x11' and > 'config.graphics.api = opengl / vulkan'. > Using only the syntax of the buildfile and the functionality > of build2 itself without the config module, > I was able to do this by always providing the respective > variables as command-line argument to 'b' as described in the > documentation for non-persistent builds. > Using the config module, manually adding custom variables to > the 'config.build' file of my project was working. > But the moment I changed the configuration with the command-line > tools, all custom variables vanished. > > Is there already a better workaround to provide such custom variables? > Otherwise, would it be possible to provide this functionality? > In this case, maybe it would even be useful to have the same mechanism > for custom configuration variables as for the already given variable pairs, > such as 'config.cxx.poptions' and 'cxx.poptions'. > Thank you in advance. Right now, I am little bit embarassed. Until now, I have not read the NEWS file or the currently existing issues on GitHub. My questions are already answered and a solution seems to be already implemented. So I am very sorry that I have bothered you with this message. Please forget it. From boris at codesynthesis.com Fri Jul 17 16:27:09 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri, 17 Jul 2020 18:27:09 +0200 Subject: [build2] Custom Configuration Variables for the Config Module In-Reply-To: <1763585766.105252.1594940255678@office.mailbox.org> References: <148206719.105178.1594935914055@office.mailbox.org> <1763585766.105252.1594940255678@office.mailbox.org> Message-ID: Markus Pawellek writes: > Right now, I am little bit embarassed. > Until now, I have not read the NEWS file or the currently existing > issues on GitHub. My questions are already answered and a solution > seems to be already implemented. So I am very sorry that I have > bothered you with this message. No worries, that's quite a happy coincidence that the functionality you need was just implemented ;-). While the documentation for this feature will be available in the usual place after the release next week, in the meantime here it is from the staged version: https://build2.org/stage/build2/doc/build2-build-system-manual.xhtml#proj-config I believe what's currently supported should allow you to achieve everything you've described. But let me know if you find anything missing. From boris at codesynthesis.com Mon Jul 20 14:14:31 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon, 20 Jul 2020 16:14:31 +0200 Subject: [build2] build2 0.13.0 released Message-ID: We have released build2 toolchain version 0.13.0. The main focus of this release is build system support for project-specific configurations and ad hoc recipes. Another notable new feature is support for the creation of alternative (e.g., with include/src split) source code layouts by the bdep-new command. We have also switched to the SPDX License Expression as the default license name scheme (old names in existing packages will still work). A note on backwards compatibility: this release cannot be upgraded to from 0.12.0 and has to be installed from scratch. Additionally, on UNIX-like operating systems the installation script now installs the toolchain libraries into a private subdirectory of lib/. This means that when installed into a shared location such as /usr/local/, the toolchain implementation details can no longer clash with anything that is already installed or might be installed in the future into the same location by the user. The corresponding NEWS file entries are included below with a more detailed discussion of the major new features available in the release notes: https://build2.org/release/0.13.0.xhtml This release has been tested on Windows, Linux, Mac OS, and FreeBSD with GCC, Clang (both vanilla and Apple's), MinGW GCC, and MSVC (14, 15, and 16). A big thank you to everyone who helped test the release candidate. There is also now: - Experimental support for NetBSD. - Gentoo ebuild file: https://packages.gentoo.org/packages/dev-util/build2 The following new build configurations have been added to the CI service: linux_debian_9-gcc_8.4 linux_debian_10-gcc_9.3 linux_debian_10-gcc_10.1 linux_debian_10-clang_10.0[_libc++] macos_10.15-clang_11.0 (Xcode 11.5 Clang 11.0.3) macos_10.15-gcc_9.3_homebrew windows_10-msvc_16.4 windows_10-msvc_16.5 windows_10-msvc_16.6 windows_10-clang_10.0_llvm_msvc_16.6[_lld] windows_10-gcc_9.2_mingw_w64 In addition, there are now optimized and static optimized builds for the latest version of every configuration. All in all, there are now 57 build configurations, up from 42 in the previous release. For the complete list, see: https://ci.cppget.org/?build-configs Installation instructions: https://build2.org/install.xhtml Download directory: https://download.build2.org/0.13.0/ SHA256 checksums: aeec2ac0e7080341159bbbc95d8264521ddf82bda01c5c253d63cae5b4558b15 *build2-install-0.13.0.sh bb20552ef26862ee9101c9e5f1677ddae911798bafc259b1ef06f64ba29ab163 *build2-install-msvc-0.13.0.bat a8c08441c82b9601e61634933fa1bfe1ef08f0d4e22e51b34d50a33ad81f0163 *build2-install-clang-0.13.0.bat 60f36c7f5135676d0772dd67ccfac012b8bcb9e9177c9305a889fd54782b6870 *build2-install-mingw-0.13.0.bat bafabcab4d5efdc489af59755a3ba7ce90834b479125cf21c34828857cc91506 *build2-toolchain-0.13.0.tar.xz 4be9fdd0c9511bd9ecf8399c4bc2eac46bd01e637e86ad0b76d7db482f4e5b11 *build2-toolchain-0.13.0.tar.gz 4382174132e1dc7ca79d7ab2d1406d14afbfc927b9814fe7e7285781eabc3022 *build2-baseutils-0.13.0-x86_64-windows.zip 00345f8e0ac5e93c9e2bee8cd6127fc922528a99ed4a543d7baa28a3a60b13e6 *build2-mingw-0.13.0-x86_64-windows.tar.xz Repository certificate fingerprint for cppget.org: 86:BA:D4:DE:2C:87:1A:EE:38:C7:F1:64:7F:65:77:02:15:79:F3:C4:83:C0:AB:5A:EA:F4:F7:8C:1D:63:30:C6 NEWS file entries: build2 version 0.13.0 * Support for project-specific configuration. A project can now use the config directive to define config..* variables, similar to the build system core and modules. For example: config [bool] config.libhello.fancy ?= false config [string] config.libhello.greeting ?= 'Hello' These variables can then be used in buildfiles and/or propagated to the source code using the command line, .in file substitution, etc. For example: if $config.libhello.fancy cxx.poptions += -DLIBHELLO_FANCY cxx.poptions += "-DLIBHELLO_GREETING=\"$config.libhello.greeting\"" See the "Project Configuration" chapter in the manual for details. * Support for ad hoc recipes. With ad hoc recipes it is now possible to provide custom implementations of operations (update, test, etc) for certain targets. For example, this is how we can pick a config header based on the platform: hxx{config}: hxx{config-linux}: include = ($cxx.target.class == 'linux') hxx{config}: hxx{config-win32}: include = ($cxx.target.class == 'windows') hxx{config}: hxx{config-macos}: include = ($cxx.target.class == 'macos') hxx{config}: {{ cp $path($<) $path($>) }} Another, more elaborate example that shows how to embed binary data into the source code with the help of the xxd(1) utility: import! xxd = xxd%exe{xxd} <{hxx cxx}{foo}>: file{foo.bin} $xxd {{ diag xxd ($<[0]) i = $path($<[0]) # Input. h = $path($>[0]) # Output header. s = $path($>[1]) # Output source. n = $name($<[0]) # Array name. # Get the position of the last byte (in hex). # $xxd -s -1 -l 1 $i | sed -n -e 's/^([0-9]+):.*$/\1/p' - | set pos if ($empty($pos)) exit "unable to extract input size from xxd output" end # Write header and source. # echo "#pragma once" >$h echo "extern const char $n[0x$pos + 1];" >>$h echo "extern const char $n[0x$pos + 1]= {" >$s $xxd -i <$i >>$s echo '};' >>$s }} Note that in both examples, the utilities (cp, echo, and sed) are builtins which means these recipes are portable. See the Testscript manual for the list of available builtins. Ad hoc recipes can also be used to customize a part of the update chain otherwise handled by rules. For example, in embedded systems development it is often required to perform a custom link step: obje{foo}: cxx{foo} obje{bar}: cxx{bar} : obje{foo bar} {{ diag ld ($>[0]) $cxx.path $cc.loptions $cxx.loptions $cxx.mode -o $path($>[0]) \ "-Wl,-Map=$path($>[1])" $path($<) $cxx.libs $cc.libs }} While the above examples are all for the update operation, ad hoc recipes can be used for other operations, such as test. For example: exe{hello}: cxx{hello} % test {{ diag test $> $> 'World' >>>?'Hello, World!' }} The above recipes are written in a shell-like language called Buildscript that has similar semantics to Testscript tests. Another language that can be used to write recipes is C++. For example: ./: {{ c++ 1 recipe apply (action, target& t) const override { text (recipe_loc) << "Hello, " << t; return noop_recipe; } }} Note that in this release support for ad hoc recipes is at the "technology preview" stage. In particular, there is no documentation and there might be some rough edges. * Support for project-local importation. An import without a project name is now treated as importation from the same project. For example, given the libhello project that exports the lib{hello} target, a buildfile for an executable in the same project instead of doing something like this: include ../libhello/ exe{hello}: ../libhello/lib{hello} Can now do: import lib = lib{hello} exe{hello}: $lib Note that the target in project-local importation must still be exported in the project's export stub. In other words, project-local importation goes through the same mechanism as normal import. See the "Target Importation" section in the manual for details. * Support for ad hoc importation and "glue buildfiles". If the target being imported has no project name and is either absolute or is a relative directory, then this is treated as ad hoc importation. Semantically it is similar to normal importation but with the location of the project being imported hard-coded into the buildfile. In particular, this type of import can be used to create a special "glue buildfile" that "pulls" together several projects, usually for convenience of development. One typical case that calls for such a glue buildfile is a multi-package project. To be able to invoke the build system directly in the project root, we can add a glue buildfile that imports and builds all the packages: import pkgs = */ ./: $pkgs See the "Target Importation" section in the manual for details. * Support for value subscripts. A value subscript is only recognized in evaluation contexts (due to ambiguity with wildcard patterns; consider: $x[123].txt) and should be unseparated from the previous token. For example: x = ($y[1]) x = (($f ? $y : $z)[1]) x = ($identity($y)[$z]) * New legal{} target type and config.install.legal variable. This allows separation of legal files (LICENSE, AUTHORS, etc) from other documentation. For example: ./: ... doc{README} legal{LICENSE} $ b install ... config.install.legal='share/licenses//' * Support for -substitutions in config.install.* values. The currently recognized variable names are and which are replaced with the project name and private subdirectory, respectively. This can be used along these lines: $ b config.install.libexec='exec_root/lib//' install The private installation subdirectory can be used to hide the implementation details of a project. This is primarily useful when installing an executable that depends on a bunch of libraries into a shared location, such as /usr/local/. For example: $ b config.install.private=foo install See the "Install Module" chapter in the manual for details. * New $regex.find_{match,search}() functions that operate on lists. * The $process.run*() functions now recognize a number of portable builtins. Refer to the Testscript manual for the list and details. * New $defined() and $visibility() functions. * New $target.process_path() function for exe{} targets analogous to $target.path(). * New $bin.link_member() function. Given a linker output target type (exe, lib[as], or libu[eas]) this function returns the target type of the lib{} group member (liba or libs) that will be picked when linking a lib{} group to this target type. * New scripting builtins: date, env. Refer to the Testscript manual for details. * New variable block applicability semantics in dependency chains. Previously the block used to apply to the set of prerequisites before the last colon. This turned out to be counterintuitive and not very useful since prerequisite-specific variables are less common than target- specific ones. The new rule is as follows: if the chain ends with a colon, then the block applies to the last set of prerequisites. Otherwise, it applies to the last set of targets. For example: ./: exe{test}: cxx{main} { test = true # Applies to the exe{test} target. } ./: exe{test}: libue{test}: { bin.whole = false # Applies to the libue{test} prerequisite. } * Test and install modules are now implicitly loaded for simple projects. While these can be useful on their own, this also makes the test and install operations available in simple projects, which is handy for "glue buildfiles" that "pull" (using ad hoc import) a bunch of other projects together. * The translated {c,cxx}.std options are now folded into the compiler mode options ({c,cxx}.mode). This makes them accessible from ad hoc recipes. The original mode/path are available in {c,cxx}.config.mode/path. * Generation of a common pkg-config .pc file in addition to static/shared- specific. The common .pc file is produced by ignoring any static/shared-specific poptions and splitting loptions/libs into Libs/Libs.private. It is "best effort", in a sense that it's not guaranteed to be sufficient in all cases, but it will probably cover the majority of cases, even on Windows, thanks to automatic dllimport'ing of functions. * The ~host configuration now only contains the cc and bin modules configuration. There is also the new ~build2 configuration that contains everything (except config.dist.*) and is meant to be used for build system modules. * Reworked tool importation support. Specifically, now config. (like config.cli) is handled by the import machinery (it is a shorter alias for config.import...exe that we already had). This also adds support for uniform tool metadata extraction that is handled by the import machinery. As a result, a tool that follows the "build2 way" can be imported with metadata by the buildfile and/or corresponding module without any tool-specific code or brittleness associated with parsing --version or similar outputs. See the cli tool/module for an example of how this all fits together. Finally, two new flavors of the import directive are now supported: import! triggers immediate importation skipping any rule-specific logic while import? is optional import (analogous to using?). Note that optional import is always immediate. There is also the import-specific metadata attribute which can be specified for these two import flavors in order to trigger metadata importation. For example: import? [metadata] cli = cli%exe{cli} if ($cli != [null]) info "cli version $($cli:cli.version)" * Backtick (`) and bit-or (|) are reserved in eval context for future use. Specifically, they are reserved for planned support of arithmetic eval contexts and evaluation pipelines, respectively. bpkg version 0.13.0 * The SPDX License Expression is now the default scheme for the 'license' package manifest value. See the "license" section in the manual for details. * New --pkg-proxy common option. This option specifies the proxy server to use when fetching package manifests and archives from remote pkg repositories. See the option documentation in bpkg-common(1) for details. * External test packages can now have their own build constraints. bdep version 0.13.0 * The SPDX License Expression in now the default scheme in the bdep-new --type|-t,license sub-option. Auto-detected licenses now also result in the SPDX License ID in the 'license' package manifest value. * New source layout customization mechanisms in bdep-new. In particular, the split include/src layout is now supported out of the box: $ bdep new -l c++ -t lib,split libhello See the SOURCE LAYOUT section in bdep-new(1) for details and a large number of examples. * The bdep-new command in the --package mode now creates the "glue buildfile" for the project. * The bdep-new command now generates the .gitattributes file with sensible defaults. * The bdep-new --subdirectory mode option has been renamed to --source. The corresponding default options file has also been renamed to bdep-new-source.options. * The bdep-new binless sub-option has been moved from --lang|-l to --type|t. brep version 0.13.0 * Support for the alternative package rebuild timeout. The alternative rebuild timeout can be used to "pull" the rebuild window to the specified time of day, for example, to optimize load and/or power consumption of the build infrastructure (off-work hours, solar, off-peak electricity tariffs, etc). * New brep-monitor utility for monitoring for and reporting on brep state issues. Currently it only reports build delays. See brep-monitor(1) for details. * Support for test-exclude task manifest value. bbot version 0.13.0 * Build logs are now UTF-8-sanitized by the worker. * Support for test-exclude task manifest value. * Bootstrap timeout has been increased from 30 to 60 minutes. From mkrupcale at matthewkrupcale.com Mon Jul 20 22:20:34 2020 From: mkrupcale at matthewkrupcale.com (Matthew Krupcale) Date: Mon, 20 Jul 2020 18:20:34 -0400 Subject: [build2] build2 0.13.0 released In-Reply-To: References: Message-ID: On Mon, Jul 20, 2020 at 10:14 AM Boris Kolpackov wrote: > > We have released build2 toolchain version 0.13.0. Congratulations on the latest release. Updates[1,2] are pending for Fedora 32 and 31. > The main focus of this release is build system support for project-specific configurations and > ad hoc recipes. Very nice, that is two of the strategic/foundational items on the roadmap[3]. I think project-specific configurations might have been one of the last things missing in my mind for a minimum viable production build system. As you point out in the documentation[4], it's not without drawbacks, but I think a lot of production codes have at least some level of build-time configurability either for dependencies or features, etc. > - Gentoo ebuild file: https://packages.gentoo.org/packages/dev-util/build2 Nice, looks like this was recently added. Glad to see more Linux distros picking it up officially. Still missing Debian/Ubuntu, SUSE, and others. > * Support for value subscripts. > > A value subscript is only recognized in evaluation contexts (due to > ambiguity with wildcard patterns; consider: $x[123].txt) and should be > unseparated from the previous token. For example: > > x = ($y[1]) > x = (($f ? $y : $z)[1]) > x = ($identity($y)[$z]) > > * Generation of a common pkg-config .pc file in addition to static/shared- > specific. This should make it easier for other systems to use build2-installed packages since they don't need to invoke pkg-config with the .shared or .static suffix. > * Reworked tool importation support. > > Specifically, now config. (like config.cli) is handled by the import > machinery (it is a shorter alias for config.import...exe that > we already had). > > This also adds support for uniform tool metadata extraction that is > handled by the import machinery. As a result, a tool that follows the > "build2 way" can be imported with metadata by the buildfile and/or > corresponding module without any tool-specific code or brittleness > associated with parsing --version or similar outputs. See the cli > tool/module for an example of how this all fits together. > > > * Backtick (`) and bit-or (|) are reserved in eval context for future use. > > Specifically, they are reserved for planned support of arithmetic eval > contexts and evaluation pipelines, respectively. [1] https://bodhi.fedoraproject.org/updates/FEDORA-2020-e60dab0868 [2] https://bodhi.fedoraproject.org/updates/FEDORA-2020-fd1c5ae7fa [3] https://github.com/build2/README/blob/master/roadmap.md [4] https://build2.org/build2/doc/build2-build-system-manual.xhtml#proj-config From boris at codesynthesis.com Tue Jul 21 06:44:25 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue, 21 Jul 2020 08:44:25 +0200 Subject: [build2] build2 not Compiling on Raspberry Pi ARMv7 Architecture In-Reply-To: <626545930.105046.1594932839402@office.mailbox.org> References: <919513607.104412.1594924614664@office.mailbox.org> <626545930.105046.1594932839402@office.mailbox.org> Message-ID: Markus Pawellek writes: > I do not know very much about the actual implementation of build2 > but with respect to the second alternative, even if it would suffer > from a poor performance, it may not be a dramatic problem, because > in my opinion on a Raspberry Pi one is typically dealing with small > code projects. One thing to keep in mind is that build2 builds itself and it's not exactly small. In fact, the reason we have those checks is that we don't actually know how bad is the performance hit. If it's not much, we could just remove them. So in this light it would have actually been useful to try to build build2 with both 32-bit Raspbian (with these checks disabled) and then with 64-bit and compare how long it takes. The way I would do it is get the build2-toolchain package (skipping the install script) and then do a timed local build: $ cd build2-toolchain $ time ./build.sh --local --make make -j 4 --install-dir /tmp/install g++ For the 32-bit build I would just edit build2/libbuild2/target.hxx and add `#define LIBBUILD2_ATOMIC_NON_LOCK_FREE` on top to disable the checks. If you do get to run this test, I would be interested to hear the result. From boris at codesynthesis.com Tue Jul 21 11:12:38 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue, 21 Jul 2020 13:12:38 +0200 Subject: [build2] Interest in Development of NASM Module In-Reply-To: <326652268.104938.1594931002936@office.mailbox.org> References: <326652268.104938.1594931002936@office.mailbox.org> Message-ID: Markus Pawellek writes: > I am very interested in the development of a build2 module > providing the possibility of compiling and linking NASM code. Sounds doable. Not having used NASM myself, I've just tried it out on the "Hello, World!" example from Wikipedia. I was even able to whip out a toy buildfile that uses the new ad hoc recipes feature to build it: using bin # For obje{}. define asm: file asm{*}: extension = asm obje{*}: extension = o exe{hello}: obje{hello} {{ diag ld $> ld -o $path($>) $path($<) }} obje{hello}: asm{hello} {{ diag nasm $< nasm -f elf64 -o $path($>) $path($<) }} To build on my Linux box I ran: $ b config.bin.target=x86_64-linux-gnu > For this, I have already taken a look inside the > libbuild2-hello and libbuild2-rust modules, > but the first one is not really explaining how to add > functionality and the second one is not compiling on my platform. Hm, strange, was it perhaps due to the version mismatch? If so, could you try it with the just-release 0.13.0? Generally, libbuild2-rust is not a bad place to start. It's not too complex but does something along the lines of what you want. > Therefore I wanted to go even further and tried to understand > the builtin cc, c, and cxx modules inside the build2 repository. > But the code here was a little bit too much for me and hard to > understand. Yeah, that stuff is not for the faint of heart. Probably not the best place to start. > What would be the best starting point and what would I have to do > to get an overview of the build2 source code and to get an idea of > build2 module development. I think libbuild2-rust is a good starting point. > Is there already an existing documentation concerning module development? Unfortunately, at this stage the best documentation is existing modules and the build system core headers (which are fairly well commented). > Is it even reasonable to try to develop an external module for NASM, > would it be better to think about a builtin module, or would it even > be better to not include it at all? That's actually a good question. At the outset, let me state that I am by no means an expert on assembler tooling and all I have is a general idea of what's going on. So if some of the things I say below are wrong, please do correct me. Firstly, given the above example with ad hoc recipes, do we even need a module? That's also a good question. I think the main reasons for wanting the module would be: 1. Not having to repeat the recipes for multiple files/in multiple projects. 2. Automatic dealing with platform specifics (for example, translating x86_64-linux-gnu to -f elf64, etc). 3. Automatic include dependency extraction (%include directive, etc). All of these are non-issues for simple projects and if ad hoc recipes are all you need (at least for now), that's fine as well. Now let's say you do want to write a module for NASM. So we are back to the question of where it should go. We already have the bin module (short for binutils) and we could add NASM support there. I am not sure, however, this is the best approach. The bin module is closely tied to the cc module and the plan is to automatically "derive" the assembler and linker from the compiler being used (we already do this on Window for the linker). This is all a bit fuzzy to me since I don't have much experience with using assemblers. I guess the question is whether it could be reasonable to want to use both the C/C++ compiler's assembler (gas, MASM) and NASM in different parts of the same build? If the answer is "yes" (and it feels like it is), then I think it's best to have a separate module for NASM. Here are a few other notes question that I've collected reading on NASM: - Should we package NASM itself? - OpenSSL requires NASM on Windows. - NASM has the -M* option family for make-style dependency extraction. - There is the %include directive and -i/-I search paths option. From boris at codesynthesis.com Wed Jul 22 12:39:52 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed, 22 Jul 2020 14:39:52 +0200 Subject: [build2] build2 0.13.0 released In-Reply-To: References: Message-ID: Matthew Krupcale writes: > Congratulations on the latest release. Updates[1,2] are pending for > Fedora 32 and 31. Thanks! > > The main focus of this release is build system support for project- > > specific configurations and ad hoc recipes. > > Very nice, that is two of the strategic/foundational items on the > roadmap[3]. I think project-specific configurations might have been > one of the last things missing in my mind for a minimum viable > production build system. Right, though I think ad hoc recipes are also in that bag: few non- trivial real-world projects don't have any custom build steps. In fact, I believe that with the addition of ad hoc recipes we now have feature parity with GNU make. Maybe ad hoc rules would be the last missing bit (planned for the next release). From bsutherland at soundhound.com Fri Jul 31 01:55:45 2020 From: bsutherland at soundhound.com (Bruce Sutherland) Date: Fri, 31 Jul 2020 10:55:45 +0900 Subject: [build2] Generating source code in build2 projects Message-ID: Hi all, As part of our build process, we generate some C++ source and header files from code in an internal language. Let's call the language Blub, say that the files have extension .blub, and the generator is called blubc. So I assume we need to configure blubc as a tool in build2, and define dependencies from .blub to .cpp and .h files. How should we do that with build2? The documentation mentions generated source code, but doesn't go into much detail. Can anyone point me to an example of a project with a similar configuration? Thanks for your help! Bruce From boris at codesynthesis.com Fri Jul 31 04:42:09 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri, 31 Jul 2020 06:42:09 +0200 Subject: [build2] Generating source code in build2 projects In-Reply-To: References: Message-ID: Bruce Sutherland writes: > So I assume we need to configure blubc as a tool in build2, and define > dependencies from .blub to .cpp and .h files. > > How should we do that with build2? The documentation mentions > generated source code, but doesn't go into much detail. Can anyone > point me to an example of a project with a similar configuration? Currently there are two way to do this: you can provide a rule for compiling any .blub to .h/.cpp or you can provide an ad hoc recipe for compiling a specific foo.blub to foo.h/foo.cpp. To provide a rule, currently, you will have to write a build system module that implements it (see, for example, libbuild2-rust[1]). In the next release we are also planning to add support for ad hoc rules that can be written directly in buildfiles (similar to ad hoc recipes below). An ad hoc recipe you can write directly in the buildfile and I would suggest that you start with that since writing a rule/module is fairly involved. The release notes for 0.13.0[2] have a number of examples that should get you started (in particular, see the xxd example). There is also a NASM example[3] I posted a few days ago that shows how to define a custom target type (e.g., for .blub). Note also that this is still an area of active development so there might be rough edges, missing pieces, etc. So we are quite interested in usage feedback and also don't mind doing a bit of hand-holding. So let me know how it goes or if there are any issues. In particular, if you have a lot of .blub, then writing ad hoc recipes could get tedious and that's where support for ad hoc rules could be helpful. [1] https://github.com/build2/libbuild2-rust [2] https://build2.org/release/0.13.0.xhtml#adhoc-recipe [3] https://lists.build2.org/archives/users/2020-July/000823.html