[build2] LTO and parallelization with build2

[Boris Kolpackov]

Matthew Krupcale <mkrupcale at matthewkrupcale.com> writes:

> [...] (by default, m=32) [...]
> 
> Based on my reading of how the partitioning works, as long as
> lto-partitons=m > -flto=n, I think it should be possible for the
> linker to fully utilize all given threads. This might require some
> experimentation and verification, though.

CPUs with more than 32 hardware threads are not that uncommon these
days.


> With the current understanding of the partitioning, I was also
> wondering if -flto=auto might be as likely to OOM as we previously
> thought (i.e. with current build2 without changes).
> 
> [...]
> 
> This of course has some assumptions about the cumulative memory of the
> m partitioned threads being the same as 1 thread doing the full link,
> but maybe it's not far off.

Could be. There is at least some fixed memory cost for each process
spawned.

In any case, I've added[1] the scheduler API to allocate extra active
threads. What's left is to translate -flto=auto. Would you like to take
a stab at it? I could give you some pointers if you are interested.

(BTW, there is now etc/bootstrap[2] script for setting up the build2
development environment. It might still have some rough edges but we
find it pretty usable.)


[1] https://github.com/build2/build2/commit/600da2b97e
[2] https://git.build2.org/cgit/etc/tree/bootstrap