Feasibility of a multi-threaded or multi-process architecture?

[varungandhi-src]

Thanks for your work on Pyright, we have a fork here (https://github.com/sourcegraph/scip-python) which we use to generate index data for Python code.

I was wondering if you've considered and/or explored a multi-threaded or multi-process architecture for Pyright. Broadly, I can see there being two ways of doing this:

• Dependency graph aware - Somehow, one would determine the dependency DAG between various modules/sub-directories first, and then process the DAG bottom up. Based on Python's import system, AIUI, determining the dependency graph will require parsing the code + doing import resolution first, which may be quite time-consuming.
• Dependency graph oblivious - Using the directories with certain configuration files as roots (e.g. pyproject.toml), try to process them all in parallel. This has the risk of duplicating the work of repeatedly type-checking common dependencies multiple times, de-duplicating that work would require extra coordination between the sub-tasks.

Do you have thoughts on the feasibility of either of these approaches? Or do you think the best way to introduce parallelism would simply be to run Pyright over various sub-directories using something like GNU Parallel and then combine any results as necessary?

[erictraut]

I recently added experimental support in the pyright CLI for multi-process type checking. You can use the --threads option to enable this. I find that it works well for large projects, and I've seen it improve overall analysis time by 3x or more. For smaller projects, it doesn't help much. The approach I took was very coarse-grained and doesn't rely on any module/sub-module analysis to partition the work. It does take into account directory locality — i.e. it attempts to send nearby files to the same worker queue.