Compiler search uses a pool of workers

Question

closes #10127

Following the request in https://github.com/spack/spack/pull/10127#pullrequestreview-186647721 I started refactoring the logic underneath:

console
$ spack compiler add

The version in this PR shows much better performance than the version in develop. The main modifications are:

1. spack.compilers.find_compilers employs a multiprocess.pool.ThreadPool to execute system commands for the detection of compiler versions.
2. A few memoized functions have been introduced to avoid poking the filesystem multiple times for the same results.

Timing tests will be posted in the discussion below.

该提问来源于开源项目：spack/spack

Answer 1

Local workstation (Ubuntu 18.04 - SSD local filesystem)

With a1565f142c3df6549382df3c58b07d663a8ad54b:

console
<this branch>$ time spack compiler add
==> Added 3 new compilers to /home/mculpo/.spack/linux/compilers.yaml
    gcc.2.0  gcc.3.0  clang.0.1-svn334776-1~exp1~20181018152737.116
==> Compilers are defined in the following files:
    /home/mculpo/.spack/linux/compilers.yaml

real    0m0,808s
user    0m0,910s
sys     0m0,289s

</this>

while using develop at ec2c5e5b0dfbbf5d70d0ccca91da4e8c149642fc:

console
<develop>$ time spack compiler add
==> Added 3 new compilers to /home/mculpo/.spack/linux/compilers.yaml
    gcc.2.0  gcc.3.0  clang.0.1-svn334776-1~exp1~20181018152737.116
==> Compilers are defined in the following files:
    /home/mculpo/.spack/linux/compilers.yaml

real    0m1,508s
user    0m3,755s
sys     0m0,718s
</develop>

Fidis front-end (rhel 7.3 - GPFS filesystem) With a1565f142c3df6549382df3c58b07d663a8ad54b:

console
<this-branch>$ time spack compiler add
==> Added 1 new compiler to /home/culpo/.spack/linux/compilers.yaml
    gcc.8.5
==> Compilers are defined in the following files:
    /home/culpo/.spack/linux/compilers.yaml

real    0m0.611s
user    0m0.432s
sys 0m0.180s
</this-branch>

while using develop at ec2c5e5b0dfbbf5d70d0ccca91da4e8c149642fc:

console
<develop>$ time spack compiler add
==> Added 1 new compiler to /home/culpo/.spack/linux/compilers.yaml
    gcc.8.5
==> Compilers are defined in the following files:
    /home/culpo/.spack/linux/compilers.yaml

real    0m1.908s
user    0m1.449s
sys 0m0.463s
</develop>