[WIP] [sol-compiler] Support parallel compilation

Question

Description

Enable parallel compilation via Node.js Child Processes. A call to Compiler.compileAsync() will child_process.spawn() a separate process for each source module indicated during Compiler construction, and then it will wait to return until all compilations have completed.

Testing instructions

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• [x] Prefix the title of this PR with [WIP] if it is a work in progress.
• [x] Prefix the title of this PR with bracketed package name(s) corresponding to the changed package(s). For example: [sol-cov] Fixed bug.
• [x] Refactor to move solcjs invocation to behind a command-line interface, so that it can be spawned as a parallel process via Node's Child Process API.
• [x] Change Compiler to actually spawn multiple parallel processes, and to wait for them all to complete before finishing a call to compileAync().
• [x] Fix bug with truncated JSON stdout from command-line utility. Check.
• [x] Consider other IPC methods, besides parsing stdin/stdout to and from JSON. Some may be more performant or appropriate for our use case, and perhaps we could simplify our code. For example, we may be able to eliminate much of our parsing and glue code by using Node's plain-old-js-object-driven message events API, or maybe there's an RPC technique that will allow the Resolver to stay in the parent process. Check.
• [x] Allay race condition when downloading solcjs executable. Two parallel processes may try to download it to the same location at the same time. Perhaps move compilerBinFilename initialization block from cli_resolver_solc.ts to just before its invocation in the Compiler class, so that downloading the executable happens in the single-threaded parent process rather than in the parallel child processes.
• [x] Implement a configuration/option for degree of parallelism. As it stands now, if your project has 50 contracts, then you'll get 50 parallel processes. Perhaps a command line flag, akin to GNU Make's -j. Check.
• [ ] Clean up yargs usage in cli_resolver_solc.ts, marking things required, etc.
• [ ] Add tests to cover my changes, or decided that tests would be too impractical.
• [ ] Updat documentation as needed.
• [ ] Add new entries to the relevant CHANGELOG.jsons.

该提问来源于开源项目：0xProject/0x-monorepo

Answer 1

I've got an initial implementation of parallel compilation in place here, and it seems to be working well. You can see it in action in the latest CircleCI build for this PR

~However,~ [Edit: The following describes a problem that has since been fixed.] ~there is still an outstanding issue in the build of the contracts package: It appears that the stdout buffer coming from the new command-line utility is being maxed out. (The JSON data coming into compiler.ts is truncated, and its length exactly matches the length of the stdout buffer, and running the solc-wrapper script directly, from the command line, yields the full output, roughly double the length of what compiler.ts receives.)~

~I had hoped that the problem would be solved when switching from child_process.execFileAsync to child_process.spawn, since with spawn one must provide a callback to receive batches of data, whereas with execFileAsync the data is given by the return value from the execFileAsync() call itself. Unfortunately that hope didn't come to fruition.~

~Investigation continues...~