[build2] Bootstrap build failing bpkg and bdep tests

Fri Sep 14 14:48:23 UTC 2018

Hello,

I am attempting to make an RPM for build2[1], and I have built a
bootstrap version of the latest build2 toolchain release (0.8.0). It
builds and installs fine, but some of the tests for bpkg and bdep fail
(the tests for build2, libbutl, and libbpkg all pass). There are
several issues I have found (the first 3 are minor):

1. When running bpkg (and bdep) tests, the "--build path" argument is
passed to bpkg through

build = $recall($build.path)
test.options += --build $build

in {bpkg,bdep}-0.8.0/tests/common.testscript. As far as I can tell,
$recall($build.path) returns argv[0] of the build2 process, which when
that path is relative, and the testscript changes directory, will
cause bpkg or bdep to fail to find the build2 executable. This shows
up in the logs[2] with --verbose 3 (and -j 1 to get sequential output)
like:

test /builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/testscript{pkg-system.testscript}
mkdir /builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/test/pkg-system/
touch /builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/test/pkg-system/.buildignore
cd /builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/test/pkg-system/
: +set cert_fp <<</builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/auth/default-cert-fp
: +echo 'certificate: \'
>=/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/test/pkg-system/cert-manifest
echo "certificate: \"
: +cat <<</builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/auth/default-cert.pem
>+/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/test/pkg-system/cert-manifest
cat
: +echo '\' >+/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/test/pkg-system/cert-manifest
echo \
: +/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/bpkg/bpkg
--build build2-0.8.0/build2/b-boot cfg-create -d cfg 2>- &cfg/***
/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/bpkg/bpkg
--build build2-0.8.0/build2/b-boot cfg-create -d cfg
/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/config.testscript:13:2:
error: /builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/bpkg/bpkg
exit code 1 != 0
  info: while testing
/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/dir{tests/}
  info: while testing
/builddir/build/BUILD/build2-toolchain-0.8.0/dir{bpkg-0.8.0/}

When I go to one of these test directories and execute the bpkg
command, I see the error more clearly:

<mock-chroot> sh-4.4# echo $PWD
/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/test/pkg-update
<mock-chroot> sh-4.4#
/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/bpkg/bpkg
--build build2-0.8.0/build2/b-boot cfg-create -d cfg
error: unable to execute build2-0.8.0/build2/b-boot: no such file or directory

As I alluded to, the simple fix for this is to run the build2 command
using the full path to the executable so that argv[0] is absolute. But
I wonder if maybe $recall($build.path) should get the absolute path?

2. When running bdep on this bootstrap-built system, it cannot find
bpkg since it is not in the default search path and is not passed as
an argument "--bpkg path" to bdep in the testscript. This produces
logs[2] such as the following:

cd /builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/tests/test/sync/single-pkg-cfg/
:  /builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/bdep/bdep
--build build2-0.8.0/build2/b-boot new --no-checks -C @cfg prj cc
'config.cxx=g++' 2>! &prj/*** &prj-cfg/***
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/bdep/bdep
--build build2-0.8.0/build2/b-boot new --no-checks -C @cfg prj cc
config.cxx=g++
created new executable project prj in
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/tests/test/sync/single-pkg-cfg/prj/
error: unable to execute bpkg: no such file or directory
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/tests/sync.testscript:17:3:
error: /builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/bdep/bdep
exit code 1 != 0
  info: while testing
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/dir{tests/}
  info: while testing
/builddir/build/BUILD/build2-toolchain-0.8.0/dir{bdep-0.8.0/}

This time we must put bpkg on the search $PATH since there's no way to
pass the bpkg path as an argument to build2 which runs the testscript
that runs bdep. Again, a relatively simple fix.

3. Similar to the previous error, after adding bpkg to the path, bdep
appears to be trying to execute "b" and failing:

cd /builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/tests/test/new/cfg/dir-and-name/
:  /builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/bdep/bdep
--build /builddir/build/BUILD/build2
-toolchain-0.8.0/build2-0.8.0/build2/b-boot new --no-checks -C
prj-config @cfg prj cc 'config.cxx=g++' 2>
>/~%EOE% &prj/*** &prj-config/***
created new executable project prj in
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/tests/test/
new/cfg/dir-and-name/prj/
created configuration @cfg
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/tests/test/new/cfg/dir
-and-name/prj-config/ 1 default,forwarded,auto-synchronized
synchronizing:
%  new prj.+19700101000000%
EOE
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/bdep/bdep
--build /builddir/build/BUILD/build2-to
olchain-0.8.0/build2-0.8.0/build2/b-boot new --no-checks -C prj-config
@cfg prj cc config.cxx=g++
:  /builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/bdep/bdep
--build /builddir/build/BUILD/build2
-toolchain-0.8.0/build2-0.8.0/build2/b-boot status -d prj >'prj
configured 0.1.0-a.0.19700101000000'
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/bdep/bdep
--build /builddir/build/BUILD/build2-to
olchain-0.8.0/build2-0.8.0/build2/b-boot status -d prj
diff -u /builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/tests/test/new/cfg/dir-and-name/stdout-2.
orig /builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/tests/test/new/cfg/dir-and-name/stdout-2
:  /builddir/build/BUILD/build2-toolchain-0.8.0/build2-0.8.0/build2/b-boot
prj/ 2>>~%EOE%
%(mkdir|c\+\+|ld|ln) .+%{4}
EOE
/builddir/build/BUILD/build2-toolchain-0.8.0/build2-0.8.0/build2/b-boot prj/
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/tests/new.testscript:191:5:
error: /builddir/buil
d/BUILD/build2-toolchain-0.8.0/build2-0.8.0/build2/b-boot exit code 1 != 0
  info: stderr:
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/tests/test/new/cfg/dir-and-name/s
tderr-3
error: unable to execute b: no such file or directory
  info: while testing
/builddir/build/BUILD/build2-toolchain-0.8.0/bdep-0.8.0/dir{tests/}
  info: while testing
/builddir/build/BUILD/build2-toolchain-0.8.0/dir{bdep-0.8.0/}

Is build2 (i.e. b-boot) trying to run bpkg which then runs build2, but
the bpkg process was not given the location of build2 (i.e. argv[0])
as one of its arguments through "--build path", and thus it defaults
to "b"? (before adding bpkg to the $PATH, the error for this test was:
"error: unable to execute bpkg: no such file or directory") If this is
the case, I'm not sure that build2 must always invoke bpkg with
build2's location, but maybe it should check if argv[0] != "b"  or
something to decide if it should (and maybe use its absolute path if
there are any directory changes that might happen).

The simple workaround for this is to also put the "b" executable on
the search $PATH.

4. The remaining errors[3] are for bpkg, and I have not been able to
figure out how to fix them. The errors are to do with parsing a
certificate:

/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/bpkg/bpkg
--build /builddir/build/BUILD/build2-toolchain-0.8.0/build2-0.8.0/build2/b-boot
rep-create --key
/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/auth/key.pem
/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/test/pkg-update/pkg/1/build2.org/pkg-update/hello
added libhello 1.0.0
error: unable to parse certificate for
/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/test/pkg-update/pkg/1/build2.org/pkg-update/hello/:
no such file or directory
/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/tests/pkg-update.testscript:47:3:
error: /builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/bpkg/bpkg
exit code 1 != 0
  info: while testing
/builddir/build/BUILD/build2-toolchain-0.8.0/bpkg-0.8.0/dir{tests/}
  info: while testing
/builddir/build/BUILD/build2-toolchain-0.8.0/dir{bpkg-0.8.0/}

I tried to debug this, and from what I can tell, the error occurs in
bpkg/auth.cxx:parse_cert[4], where it calls

bad_cert ("no common name (CN)");

thus throwing a std::invalid_argument, which is caught as a
std::system_error, and then parse_failed is called with this
exception, printing the message "no such file or directory". What I
don't understand is to what file this is referring (I don't think it's
key.pem because that appears to have been read just fine, and it
certainly exists).

Do you know what the issue in this test is?

Best,
Matthew Krupcale

[1] https://copr.fedorainfracloud.org/coprs/mkrupcale/build2
[2] https://copr-be.cloud.fedoraproject.org/results/mkrupcale/build2/fedora-28-x86_64/00797979-build2/builder-live.log
[3] https://copr-be.cloud.fedoraproject.org/results/mkrupcale/build2/fedora-28-x86_64/00798141-build2/builder-live.log
[4] https://git.build2.org/cgit/bpkg/tree/bpkg/auth.cxx#n213