From mkrupcale at matthewkrupcale.com Sat Aug 1 16:45:09 2020 From: mkrupcale at matthewkrupcale.com (Matthew Krupcale) Date: Sat, 1 Aug 2020 12:45:09 -0400 Subject: [build2] LTO and parallelization with build2 Message-ID: Hello, Fedora is looking to enable LTO by default in F33[1], and recently they decided to use -flto=auto for the default GCC LTO build flags[2]. This means[3] that GCC will attempt to use the GNU make jobserver, if available, but otherwise fall back to using the number of CPU threads on the system. Since build2 has its own scheduler, though, this means there might be e.g. up to n^2 threads on an n CPU thread system during linking, which could negatively impact performance / waste CPU time. So when building with -flto={auto,n} using GCC, build2 may either need to instead invoke the compiler/linker with -flto=m, where m is the number of available threads from the scheduler (I'm not sure the scheduler currently supports something like this), or just disable parallel linking with -flto[=1]. Alternatively, build2 could attempt to act as a GNU make jobserver itself somehow. This also might become relevant during compilation as GCC looks to parallelize compilation[4,5]. I wonder if you've given any thought on how best to handle this. As far as I can tell, ninja hasn't really addressed this either, but it looks like they're considering making ninja work as a GNU make jobserver client[6,7], and there's a recent PR to make ninja work with multiple simultaneous ninja processes, all sharing a job limit[8]. Best, Matthew [1] https://fedoraproject.org/wiki/LTOByDefault [2] https://src.fedoraproject.org/rpms/redhat-rpm-config/c/4637e1bd5512b869bd07c58b7545d2528a9bc4c8?branch=master [3] https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#Optimize-Options [4] https://gcc.gnu.org/wiki/ParallelGcc [5] https://www.phoronix.com/scan.php?page=news_item&px=GCC-2020-More-Parallel-Large-S [6] https://github.com/ninja-build/ninja/issues/1139#issuecomment-520546470 [7] https://github.com/ninja-build/ninja/pull/1140 [8] https://github.com/ninja-build/ninja/pull/1815 From bsutherland at soundhound.com Sun Aug 2 23:53:20 2020 From: bsutherland at soundhound.com (Bruce Sutherland) Date: Mon, 3 Aug 2020 08:53:20 +0900 Subject: [build2] Generating source code in build2 projects In-Reply-To: References: Message-ID: Thank you Boris, that's very helpful. I'll start out with ad hoc recipes, and will reach out again if we run into trouble. On Fri, Jul 31, 2020 at 1:42 PM Boris Kolpackov wrote: > > Bruce Sutherland writes: > > > So I assume we need to configure blubc as a tool in build2, and define > > dependencies from .blub to .cpp and .h files. > > > > How should we do that with build2? The documentation mentions > > generated source code, but doesn't go into much detail. Can anyone > > point me to an example of a project with a similar configuration? > > Currently there are two way to do this: you can provide a rule for > compiling any .blub to .h/.cpp or you can provide an ad hoc recipe > for compiling a specific foo.blub to foo.h/foo.cpp. > > To provide a rule, currently, you will have to write a build system > module that implements it (see, for example, libbuild2-rust[1]). In > the next release we are also planning to add support for ad hoc rules > that can be written directly in buildfiles (similar to ad hoc recipes > below). > > An ad hoc recipe you can write directly in the buildfile and I would > suggest that you start with that since writing a rule/module is fairly > involved. The release notes for 0.13.0[2] have a number of examples > that should get you started (in particular, see the xxd example). > There is also a NASM example[3] I posted a few days ago that shows > how to define a custom target type (e.g., for .blub). > > Note also that this is still an area of active development so there > might be rough edges, missing pieces, etc. So we are quite interested > in usage feedback and also don't mind doing a bit of hand-holding. So > let me know how it goes or if there are any issues. In particular, if > you have a lot of .blub, then writing ad hoc recipes could get tedious > and that's where support for ad hoc rules could be helpful. > > [1] https://github.com/build2/libbuild2-rust > [2] https://build2.org/release/0.13.0.xhtml#adhoc-recipe > [3] https://lists.build2.org/archives/users/2020-July/000823.html From boris at codesynthesis.com Mon Aug 3 11:51:03 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon, 3 Aug 2020 13:51:03 +0200 Subject: [build2] LTO and parallelization with build2 In-Reply-To: References: Message-ID: Matthew Krupcale writes: > Fedora is looking to enable LTO by default in F33[1], and recently > they decided to use -flto=auto for the default GCC LTO build flags[2]. > This means[3] that GCC will attempt to use the GNU make jobserver, if > available, but otherwise fall back to using the number of CPU threads > on the system. Since build2 has its own scheduler, though, this means > there might be e.g. up to n^2 threads on an n CPU thread system during > linking, which could negatively impact performance / waste CPU time. The bigger issue is potential memory usage (I've seen translation units that take over 1G to compile). > So when building with -flto={auto,n} using GCC, build2 may either need > to instead invoke the compiler/linker with -flto=m, where m is the > number of available threads from the scheduler (I'm not sure the > scheduler currently supports something like this), or just disable > parallel linking with -flto[=1]. Alternatively, build2 could attempt > to act as a GNU make jobserver itself somehow. This also might become > relevant during compilation as GCC looks to parallelize > compilation[4,5]. > > I wonder if you've given any thought on how best to handle this. As > far as I can tell, ninja hasn't really addressed this either, but it > looks like they're considering making ninja work as a GNU make > jobserver client[6,7], and there's a recent PR to make ninja work with > multiple simultaneous ninja processes, all sharing a job limit[8]. For -flto to work via jobserver, I believe build2 (and ninja) would need to implement the server proper, not just the client. And having been subscribed to make-alpha for the past decade, I can tell you that jobserver in GNU make has been a never-ending source of bugs, corner case, and compatibility issues (see this post[1] for a primer). So I would like to avoid touching that can of worms if I can help it (we could probably do the client if really necessary, but the server is a whole different story). Now on to how we could handle -flto in build2. It would actually be quite easy for the link rule to allocate more than one hardware thread (if available) in order to pass it on to the linker. There is no such support in the scheduler now but it should be pretty straightforward to add. With this idea then it's only a matter of rewriting -flto=auto or -flto=jobserver with -flto=N where N is the number of hardware threads allocated. There are two potentially problems with this: 1. If GCC does not use all the allocated threads, then they will be wasted, which would be pretty bad. Do you know if GCC will always utilize all the threads given? It appears to be generating a Makefile that it then passes to make, so probably it depends on what's in that Makefile. 2. Theoretically, via the jobserver, the linker can utilize additional threads as they become available. In our case, the number of allocated threads would be fixed at the linker start time. Thoughts? [1] http://make.mad-scientist.net/papers/jobserver-implementation/ From mkrupcale at matthewkrupcale.com Mon Aug 3 14:56:38 2020 From: mkrupcale at matthewkrupcale.com (Matthew Krupcale) Date: Mon, 3 Aug 2020 10:56:38 -0400 Subject: [build2] LTO and parallelization with build2 In-Reply-To: References: Message-ID: On Mon, Aug 3, 2020 at 7:51 AM Boris Kolpackov wrote: > > The bigger issue is potential memory usage (I've seen translation units > that take over 1G to compile). Yeah, very good point. This will especially be a problem on ARM. > For -flto to work via jobserver, I believe build2 (and ninja) would > need to implement the server proper, not just the client. Yes, I believe that's right. The ninja devs in issue 1139 I think were primarily trying to address a different issue which is what happens if ninja is invoked by an outer make, or somehow make is the primary scheduler. > And having been subscribed to make-alpha for the past decade, I can tell you > that jobserver in GNU make has been a never-ending source of bugs, > corner case, and compatibility issues (see this post[1] for a primer). > So I would like to avoid touching that can of worms if I can help it Yeah, it seems like implementing the make jobserver would be rather complex. > Now on to how we could handle -flto in build2. It would actually > be quite easy for the link rule to allocate more than one hardware > thread (if available) in order to pass it on to the linker. There > is no such support in the scheduler now but it should be pretty > straightforward to add. With this idea then it's only a matter > of rewriting -flto=auto or -flto=jobserver with -flto=N where > N is the number of hardware threads allocated. Yeah, that's kind of what I had in mind. > There are two potentially problems with this: > > 1. If GCC does not use all the allocated threads, then they will be > wasted, which would be pretty bad. > > Do you know if GCC will always utilize all the threads given? It > appears to be generating a Makefile that it then passes to make, > so probably it depends on what's in that Makefile. The LTO WHOPR mode[1] is enabled when -flto is passed[2] and an LTO partitioning algorithm is used[3]. The LGEN phase should be executed in parallel already by build2 since it invokes the compiler in parallel for the TUs. Then lto-wrapper forks and execs the two remaining stages, each executed with the specified parallelism: 1. The WPA stage is partitioned[4] with the output of each partition done by separate forks of the LTO process[5]. Currently, the WPA stage doesn't support the jobserver mode[6]. The default partitioning algorithm is balanced[7], with the number of partitions controlled by lto-partitions parameter[7] (default 32[8]). It looks like the lto-partitions should exceed the number of CPUs used for compilation. 2. The LTRANS stage also operates on each partition independently. Whether or not using the jobserver, the parallel LTO mode generates a temporary Makefile[9], but if not using the jobserver, make will be invoked without --jobserver-fd args[10] and with a statically generated number of make jobs[11]. So provided that there are more partitions than allocated CPU threads (i.e. lto-partitions > n), both WPA and LTRANS stages of GCC LTO should utilize all n threads from the scheduler. > 2. Theoretically, via the jobserver, the linker can utilize additional > threads as they become available. In our case, the number of > allocated threads would be fixed at the linker start time. Yeah, I had thought about this issue with the static thread number approach. Without the build2 jobserver, both WPA and LTRANS stages are limited to n threads. Only the LTRANS stage currently supports the use of the jobserver, though. Thus, even if build2 implements a make jobserver, the WPA stage will currently be limited to at most n threads from the build2 scheduler when spawning the linker. Hypothetically, the WPA stage might support the jobserver at some point, so both stages could support dynamic thread allocation from a build2 jobserver. But I think the above analysis at least indicates that the static thread approach should be capable of fully utilizing the threads assigned for linking. It would then be up to build2 to make sure it's keeping the other threads busy with other tasks while linking. [1] https://gcc.gnu.org/onlinedocs/gccint/LTO-Overview.html [2] https://github.com/gcc-mirror/gcc/blob/d2ae6d5c053315c94143103eeae1d3cba005ad9d/gcc/lto-wrapper.c#L1521 [3] https://github.com/gcc-mirror/gcc/blob/d2ae6d5c053315c94143103eeae1d3cba005ad9d/gcc/lto-wrapper.c#L1564 [4] https://github.com/gcc-mirror/gcc/blob/d2ae6d5c053315c94143103eeae1d3cba005ad9d/gcc/lto-wrapper.c#L1747 [5] https://github.com/gcc-mirror/gcc/blob/d1961e648e0fedebd06e4ad786c1bfc536312ef7/gcc/lto/lto.c#L2398 [6] https://github.com/gcc-mirror/gcc/blob/d1961e648e0fedebd06e4ad786c1bfc536312ef7/gcc/lto/lto.c#L3154 [7] https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html [8] https://github.com/gcc-mirror/gcc/blob/51e85e64e125803502fde94b9e22037c0ccaa8b2/gcc/params.def#L1097 [9] https://github.com/gcc-mirror/gcc/blob/d2ae6d5c053315c94143103eeae1d3cba005ad9d/gcc/lto-wrapper.c#L1877 [10] https://github.com/gcc-mirror/gcc/blob/d2ae6d5c053315c94143103eeae1d3cba005ad9d/gcc/lto-wrapper.c#L1949 [11] https://github.com/gcc-mirror/gcc/blob/d2ae6d5c053315c94143103eeae1d3cba005ad9d/gcc/lto-wrapper.c#L1968 From boris at codesynthesis.com Wed Aug 5 12:21:32 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed, 5 Aug 2020 14:21:32 +0200 Subject: [build2] LTO and parallelization with build2 In-Reply-To: References: Message-ID: Matthew Krupcale writes: > Yeah, it seems like implementing the make jobserver would be rather > complex. One idea came into my head: we could provide hooks in the scheduler to allow a build system module to interface with something that controls hardware thread allocation (we would also need some kind of a command line option for module pre-load since such a module wouldn't be loaded from buildfiles). One could then implement a module that provides the jobserver functionality either just the client or the server and either from scratch or by reusing GNU make. Just putting the idea out there. > The LTO WHOPR mode[1] is enabled when -flto is passed[2] and an LTO > partitioning algorithm is used[3]. The LGEN phase should be executed > in parallel already by build2 since it invokes the compiler in > parallel for the TUs. Then lto-wrapper forks and execs the two > remaining stages, each executed with the specified parallelism: > > [...] Thanks for the overview! > So provided that there are more partitions than allocated CPU threads > (i.e. lto-partitions > n), both WPA and LTRANS stages of GCC LTO > should utilize all n threads from the scheduler. Does the number of partititions somehow correlate with the number of TUs being linked? In a sense we have two problems: 1. We could supply too few hardware threads (e.g., because other threads are still being used by build2 at the start of linking but may become available during linking). 2. We could supply too many hardware threads that the linker cannot utilize but that build2 could have used for other tasks. One interesting heuristics against (2) would be to supply up to half of the available hardware threads. With hyper-threaded CPUs one would only waste at most 10-20% of performance in the worst case scenario. > Hypothetically, the WPA stage might support the jobserver at some > point, so both stages could support dynamic thread allocation from a > build2 jobserver. But I think the above analysis at least indicates > that the static thread approach should be capable of fully utilizing > the threads assigned for linking. It would then be up to build2 to > make sure it's keeping the other threads busy with other tasks while > linking. Yes, that's the problem, build2 may have nothing else to do and no way to communicate to the linker that it can use more threads (e.g., during the second stage). But it's probably a good enough first approximation. From mkrupcale at matthewkrupcale.com Wed Aug 5 16:24:13 2020 From: mkrupcale at matthewkrupcale.com (Matthew Krupcale) Date: Wed, 5 Aug 2020 12:24:13 -0400 Subject: [build2] LTO and parallelization with build2 In-Reply-To: References: Message-ID: On Wed, Aug 5, 2020 at 8:21 AM Boris Kolpackov wrote: > > One idea came into my head: we could provide hooks in the scheduler > to allow a build system module to interface with something that > controls hardware thread allocation (we would also need some kind > of a command line option for module pre-load since such a module > wouldn't be loaded from buildfiles). One could then implement a module > that provides the jobserver functionality either just the client or > the server and either from scratch or by reusing GNU make. Just putting > the idea out there. That's an interesting idea indeed. It would require implementing the jobserver somehow as you said (either custom or reusing GNU make), but at least it would be somewhat isolated from the existing scheduler then. > Does the number of partititions somehow correlate with the number of > TUs being linked? I think for -flto-partition=balanced (the default) the number of partitions is always statically determined by --param lto-partitions=m, regardless of the number of TUs (by default, m=32). For -flto-partition=1to1, m might be equivalent to the number of TUs. > 1. We could supply too few hardware threads (e.g., because other > threads are still being used by build2 at the start of linking > but may become available during linking). Yeah, this is the problem that would require integrating linker stages 2-3 with the build2 jobserver somehow. > 2. We could supply too many hardware threads that the linker cannot > utilize but that build2 could have used for other tasks. Based on my reading of how the partitioning works, as long as lto-partitons=m > -flto=n, I think it should be possible for the linker to fully utilize all given threads. This might require some experimentation and verification, though. > But it's probably a good enough first approximation. Yeah, that's kind of what I was thinking. With the current understanding of the partitioning, I was also wondering if -flto=auto might be as likely to OOM as we previously thought (i.e. with current build2 without changes). build2 will already spawn as many linker threads (each currently spawning 1 process when CFLAGS has -flto or -flto=1) as the scheduler has available (call it n). Let's say those n threads are linking TU's with cumulative size N_i bytes (or some other metric of code complexity, like nodes in the tree representation), 0