Auto multithreaded build limiting

[henryiii]

If NPY_NUM_BUILD_JOBS is unset, we could try to estimate the total memory and limit the number of jobs to 1 if it might fail multithreaded.

[LovelyBuggies]

If threads are unset, shouldn't it be 1 by default? I think you are worried about the insufficient memory issue. If so, resetting jobs to 1 might not be the best solution.

We can get the available system memory using psutil. But how could we estimate the memory we will use in those threads (and it varies in different systems, doesn't it)? Plus, it's unreasonable to reset h.fill(vals, threads=64) to h.fill(vals, threads=1) once you realize this might cause memory problem. It might be okay for threads=32 or 16, and hence saving time.

I think maybe a good way is to monitor the total memory usage, and exit/kill some threads when it reaches the warning line.

[henryiii]

Sorry for missing this - the issue is building, not histogramming. pip install boost-histogram is very slow if you need to build and you don't multithread by default. You can roughly guess how much memory a system has available or total (via psutil), divide that by roughly the amount you think it takes to build per thread (probably varies a bit per system, but is on order 1-2GB), then limit the automatic selection threads via that. That's the idea, anyway,.