[build2] Benchmarking with build2

[Boris Kolpackov]

Luiz Fernando Afra Brito <luizfernandoafrabrito at gmail.com> writes:

> I started a lib project, then I implemented the data strutures in the
> library subdirectory and some tests in the ``tests'' subdirectory. Now I
> want a ``benchmarks'' subdirectory to code some benchmarks.
> 
> I think it would be very convenient if I could type something like "b
> benchmark", or something close, to run all benchmarks at once.

I think inventing a new operation (which is what `b benchmark` would
entail) is probably an overkill since it is essentially a variant of
a test. Another common kind-of-like-test case is examples and we
probably wouldn't want to also support `b example` or some such.

On the other hand, if you put all your benchmarks into the benchmarks/
subdirectory and run them as test, you can do:

$ b test: benchmarks/

Which I think is fairly close.


> How woud you approach this problem using build2? Would you create
> another exe project separated from the library or as an inner project
> using the new --subdirectory flag? 'What is the best practice?

The best practice is to create another subproject (in the build
system terms) similar to tests/ (the same recommendatation would
also apply to examples, BTW).

While there is currently no bdep option to do this, it's pretty
easy to achieve by simply copying the tests/ subdirectory, for
example:

$ cp -r tests/ benchmarks/

Then you can rename benchmarks/basics/ to something more suitable,
add more benchmarks next to it, etc.

With this arrangement, whenever you run `b test` from the root of
your project, you will also run the benchmarks. If you don't want
that (for example, because they take too long), you can exclude
benchmarks/ from the default target. To achieve this, replace
the following line in your project's root buildfile:

./: {*/ -build/} manifest

With this:

./: {*/ -build/ -benchmarks/} manifest
./: benchmarks/: include = false

Now to run benchmarks, you will have to explicitly specify this
directory:

$ b test: benchmarks/