Multiprocessing overloads cores

[nathanmlim]

I have an SDF file which contains multiple molecules for which I want to calculate all the descriptors for. But, running mordred from the command line appears to overload the CPU cores. Even when using the -p 1, it shows more than a single core being used.

--

Information

OS: CentOS7
Installation: conda
Python Version: 3.7

library version

# packages in environment at /data/nlim/anaconda3:
#
# Name                    Version                   Build  Channel
_anaconda_depends         2019.03                  py37_0  
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0  
_libgcc_mutex             0.1                        main  
alabaster                 0.7.12                   py37_0  
anaconda                  custom                   py37_1  
anaconda-client           1.7.2                    py37_0  
anaconda-navigator        1.9.7                    py37_0  
anaconda-project          0.8.3                      py_0  
asn1crypto                0.24.0                   py37_0  
astroid                   2.2.5                    py37_0  
astropy                   3.2.1            py37h7b6447c_0  
atomicwrites              1.3.0                    py37_1  
attrs                     19.1.0                   py37_1  
babel                     2.7.0                      py_0  
backcall                  0.1.0                    py37_0  
backports                 1.0                        py_2  
backports.functools_lru_cache 1.5                        py_2  
backports.os              0.1.1                    py37_0  
backports.shutil_get_terminal_size 1.0.0                    py37_2  
backports.tempfile        1.0                        py_1  
backports.weakref         1.0.post1                  py_1  
bcrypt                    3.1.7            py37h7b6447c_0  
beautifulsoup4            4.7.1                    py37_1  
bitarray                  0.9.3            py37h7b6447c_0  
bkcharts                  0.2                      py37_0  
blas                      1.0                         mkl  
bleach                    3.1.0                    py37_0  
blosc                     1.16.3               hd408876_0  
bokeh                     1.2.0                    py37_0  
boto                      2.49.0                   py37_0  
bottleneck                1.2.1            py37h035aef0_1  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2019.11.27                    0  
cairo                     1.14.12              h8948797_3  
certifi                   2019.11.28               py37_0  
cffi                      1.12.3           py37h2e261b9_0  
chardet                   3.0.4                    py37_1  
click                     7.0                      py37_0  
cloudpickle               1.2.1                      py_0  
clyent                    1.2.2                    py37_1  
colorama                  0.4.1                    py37_0  
colorclass                2.2.0                    pypi_0    pypi
conda                     4.8.1                    py37_0  
conda-build               3.18.8                   py37_0  
conda-env                 2.6.0                         1  
conda-package-handling    1.3.11                   py37_0  
conda-verify              3.4.2                      py_1  
contextlib2               0.5.5                    py37_0  
cryptography              2.7              py37h1ba5d50_0  
curl                      7.65.2               hbc83047_0  
cycler                    0.10.0                   py37_0  
cython                    0.29.12          py37he6710b0_0  
cytoolz                   0.10.0           py37h7b6447c_0  
daemonize                 2.5.0                    pypi_0    pypi
dask                      2.1.0                      py_0  
dask-core                 2.1.0                      py_0  
dbus                      1.13.6               h746ee38_0  
decorator                 4.4.0                    py37_1  
defusedxml                0.6.0                      py_0  
distributed               2.1.0                      py_0  
docutils                  0.14                     py37_0  
entrypoints               0.3                      py37_0  
et_xmlfile                1.0.1                    py37_0  
expat                     2.2.6                he6710b0_0  
fabric                    2.5.0                    py37_0  
fastcache                 1.1.0            py37h7b6447c_0  
filelock                  3.0.12                     py_0  
flask                     1.1.1                      py_0  
fontconfig                2.13.0               h9420a91_0  
freetype                  2.9.1                h8a8886c_1  
fribidi                   1.0.5                h7b6447c_0  
future                    0.17.1                   py37_0  
get_terminal_size         1.0.0                haa9412d_0  
gevent                    1.4.0            py37h7b6447c_0  
glib                      2.56.2               hd408876_0  
glob2                     0.7                        py_0  
gmp                       6.1.2                h6c8ec71_1  
gmpy2                     2.0.8            py37h10f8cd9_2  
graphite2                 1.3.13               h23475e2_0  
greenlet                  0.4.15           py37h7b6447c_0  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
h5py                      2.9.0            py37h7918eee_0  
harfbuzz                  1.8.8                hffaf4a1_0  
hdf5                      1.10.4               hb1b8bf9_0  
heapdict                  1.0.0                    py37_2  
html5lib                  1.0.1                    py37_0  
icu                       58.2                 h9c2bf20_1  
idna                      2.8                      py37_0  
imageio                   2.5.0                    py37_0  
imagesize                 1.1.0                    py37_0  
importlib_metadata        0.17                     py37_1  
intel-openmp              2019.4                      243  
invoke                    1.3.0                    py37_0  
ipykernel                 5.1.1            py37h39e3cac_0  
ipython                   7.6.1            py37h39e3cac_0  
ipython_genutils          0.2.0                    py37_0  
ipywidgets                7.5.0                      py_0  
isort                     4.3.21                   py37_0  
itsdangerous              1.1.0                    py37_0  
jbig                      2.1                  hdba287a_0  
jdcal                     1.4.1                      py_0  
jedi                      0.13.3                   py37_0  
jeepney                   0.4                      py37_0  
jinja2                    2.10.1                   py37_0  
joblib                    0.13.2                   py37_0  
jpeg                      9b                   h024ee3a_2  
json5                     0.8.4                      py_0  
jsonschema                3.0.1                    py37_0  
jupyter                   1.0.0                    py37_7  
jupyter_client            5.3.1                      py_0  
jupyter_console           6.0.0                    py37_0  
jupyter_core              4.5.0                      py_0  
jupyterlab                1.0.2            py37hf63ae98_0  
jupyterlab_server         1.0.0                      py_0  
keyring                   18.0.0                   py37_0  
kiwisolver                1.1.0            py37he6710b0_0  
krb5                      1.16.1               h173b8e3_7  
lazy-object-proxy         1.4.1            py37h7b6447c_0  
libarchive                3.3.3                h5d8350f_5  
libboost                  1.67.0               h46d08c1_4  
libcurl                   7.65.2               h20c2e04_0  
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
liblief                   0.9.0                h7725739_2  
libpng                    1.6.37               hbc83047_0  
libsodium                 1.0.16               h1bed415_0  
libssh2                   1.8.2                h1ba5d50_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.0.10               h2733197_2  
libtool                   2.4.6                h7b6447c_5  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.9                hea5a465_1  
libxslt                   1.1.33               h7d1a2b0_0  
llvmlite                  0.29.0           py37hd408876_0  
locket                    0.2.0                    py37_1  
lxml                      4.3.4            py37hefd8a0e_0  
lz4-c                     1.8.1.2              h14c3975_0  
lzo                       2.10                 h49e0be7_2  
markupsafe                1.1.1            py37h7b6447c_0  
matplotlib                3.1.0            py37h5429711_0  
mccabe                    0.6.1                    py37_1  
mistune                   0.8.4            py37h7b6447c_0  
mkl                       2019.4                      243  
mkl-service               2.0.2            py37h7b6447c_0  
mkl_fft                   1.0.12           py37ha843d7b_0  
mkl_random                1.0.2            py37hd81dba3_0  
mock                      3.0.5                    py37_0  
mordred                   1.2.0              pyhe5148d4_0    mordred-descriptor
more-itertools            7.0.0                    py37_0  
mpc                       1.1.0                h10f8cd9_1  
mpfr                      4.0.1                hdf1c602_3  
mpmath                    1.1.0                    py37_0  
msgpack-python            0.6.1            py37hfd86e86_1  
multipledispatch          0.6.0                    py37_0  
navigator-updater         0.2.1                    py37_0  
nbconvert                 5.5.0                      py_0  
nbformat                  4.4.0                    py37_0  
ncurses                   6.1                  he6710b0_1  
networkx                  2.3                        py_0  
nltk                      3.4.4                    py37_0  
nose                      1.3.7                    py37_2  
notebook                  6.0.0                    py37_0  
numba                     0.44.1           py37h962f231_0  
numexpr                   2.6.9            py37h9e4a6bb_0  
numpy                     1.16.4           py37h7e9f1db_0  
numpy-base                1.16.4           py37hde5b4d6_0  
numpydoc                  0.9.1                      py_0  
olefile                   0.46                     py37_0  
openeye-toolkits          2019.10.2                py37_0    openeye
openpyxl                  2.6.2                      py_0  
openssl                   1.1.1d               h7b6447c_3  
packaging                 19.0                     py37_0  
pandas                    0.24.2           py37he6710b0_0  
pandoc                    2.2.3.2                       0  
pandocfilters             1.4.2                    py37_1  
pango                     1.42.4               h049681c_0  
paramiko                  2.6.0                    py37_0  
parso                     0.5.0                      py_0  
partd                     1.0.0                      py_0  
patchelf                  0.9                  he6710b0_3  
path.py                   12.0.1                     py_0  
pathlib2                  2.3.4                    py37_0  
patsy                     0.5.1                    py37_0  
pcre                      8.43                 he6710b0_0  
pep8                      1.7.1                    py37_0  
pexpect                   4.7.0                    py37_0  
pickleshare               0.7.5                    py37_0  
pillow                    6.1.0            py37h34e0f95_0  
pip                       19.1.1                   py37_0  
pixman                    0.38.0               h7b6447c_0  
pkginfo                   1.5.0.1                  py37_0  
pluggy                    0.12.0                     py_0  
ply                       3.11                     py37_0  
prometheus_client         0.7.1                      py_0  
prompt_toolkit            2.0.9                    py37_0  
psutil                    5.6.3            py37h7b6447c_0  
ptyprocess                0.6.0                    py37_0  
pueue                     1.0.5                    pypi_0    pypi
py                        1.8.0                    py37_0  
py-boost                  1.67.0           py37h04863e7_4  
py-lief                   0.9.0            py37h7725739_2  
pycodestyle               2.5.0                    py37_0  
pycosat                   0.6.3            py37h14c3975_0  
pycparser                 2.19                     py37_0  
pycrypto                  2.6.1            py37h14c3975_9  
pycurl                    7.43.0.3         py37h1ba5d50_0  
pyflakes                  2.1.1                    py37_0  
pygments                  2.4.2                      py_0  
pylint                    2.3.1                    py37_0  
pynacl                    1.3.0            py37h7b6447c_0  
pyodbc                    4.0.26           py37he6710b0_0  
pyopenssl                 19.0.0                   py37_0  
pyparsing                 2.4.0                      py_0  
pyqt                      5.9.2            py37h05f1152_2  
pyrsistent                0.14.11          py37h7b6447c_0  
pysocks                   1.7.0                    py37_0  
pytables                  3.5.2            py37h71ec239_1  
pytest                    5.0.1                    py37_0  
pytest-arraydiff          0.3              py37h39e3cac_0  
pytest-astropy            0.5.0                    py37_0  
pytest-doctestplus        0.3.0                    py37_0  
pytest-openfiles          0.3.2                    py37_0  
pytest-remotedata         0.3.1                    py37_0  
python                    3.7.3                h0371630_0  
python-dateutil           2.8.0                    py37_0  
python-libarchive-c       2.8                     py37_11  
pytz                      2019.1                     py_0  
pywavelets                1.0.3            py37hdd07704_1  
pyyaml                    5.1.1            py37h7b6447c_0  
pyzmq                     18.0.0           py37he6710b0_0  
qt                        5.9.7                h5867ecd_1  
qtawesome                 0.5.7                    py37_1  
qtconsole                 4.5.1                      py_0  
qtpy                      1.8.0                      py_0  
rdkit                     2019.09.3.0      py37hc20afe1_1    rdkit
readline                  7.0                  h7b6447c_5  
requests                  2.22.0                   py37_0  
rope                      0.14.0                     py_0  
ruamel_yaml               0.15.46          py37h14c3975_0  
scikit-image              0.15.0           py37he6710b0_0  
scikit-learn              0.21.2           py37hd81dba3_0  
scipy                     1.3.0            py37h7c811a0_0  
seaborn                   0.9.0                    py37_0  
secretstorage             3.1.1                    py37_0  
send2trash                1.5.0                    py37_0  
setuptools                41.0.1                   py37_0  
simplegeneric             0.8.1                    py37_2  
singledispatch            3.4.0.3                  py37_0  
sip                       4.19.8           py37hf484d3e_0  
six                       1.12.0                   py37_0  
snappy                    1.1.7                hbae5bb6_3  
snowballstemmer           1.9.0                      py_0  
sortedcollections         1.1.2                    py37_0  
sortedcontainers          2.1.0                    py37_0  
soupsieve                 1.8                      py37_0  
sphinx                    2.1.2                      py_0  
sphinxcontrib             1.0                      py37_1  
sphinxcontrib-applehelp   1.0.1                      py_0  
sphinxcontrib-devhelp     1.0.1                      py_0  
sphinxcontrib-htmlhelp    1.0.2                      py_0  
sphinxcontrib-jsmath      1.0.1                      py_0  
sphinxcontrib-qthelp      1.0.2                      py_0  
sphinxcontrib-serializinghtml 1.1.3                      py_0  
sphinxcontrib-websupport  1.1.2                      py_0  
spyder                    3.3.6                    py37_0  
spyder-kernels            0.5.1                    py37_0  
sqlalchemy                1.3.5            py37h7b6447c_0  
sqlite                    3.29.0               h7b6447c_0  
statsmodels               0.10.0           py37hdd07704_0  
sympy                     1.4                      py37_0  
tbb                       2019.8               hfd86e86_0  
tblib                     1.4.0                      py_0  
terminado                 0.8.2                    py37_0  
terminaltables            3.1.0                    pypi_0    pypi
testpath                  0.4.2                    py37_0  
tk                        8.6.8                hbc83047_0  
toolz                     0.10.0                     py_0  
tornado                   6.0.3            py37h7b6447c_0  
tqdm                      4.32.1                     py_0  
traitlets                 4.3.2                    py37_0  
unicodecsv                0.14.1                   py37_0  
unixodbc                  2.3.7                h14c3975_0  
urllib3                   1.24.2                   py37_0  
wcwidth                   0.1.7                    py37_0  
webencodings              0.5.1                    py37_1  
werkzeug                  0.15.4                     py_0  
wheel                     0.33.4                   py37_0  
widgetsnbextension        3.5.0                    py37_0  
wrapt                     1.11.2           py37h7b6447c_0  
wurlitzer                 1.0.2                    py37_0  
xlrd                      1.2.0                    py37_0  
xlsxwriter                1.1.8                      py_0  
xlwt                      1.3.0                    py37_0  
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
zeromq                    4.3.1                he6710b0_3  
zict                      1.0.0                      py_0  
zipp                      0.5.1                      py_0  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.3.7                h0b5b093_0  

[gfrogat]

I encountered the same problem. It's due to Intel MKL spawning a lot of additional threads in the background.

You can limit the number of threads by setting these environment variables before you run your program. CPU utilization should be a lot more reasonable then.

export OMP_NUM_THREADS=2
export MKL_NUM_THREADS=2
export MKL_THREADING_LAYER=SEQUENTIAL
export NUMEXPR_NUM_THREADS=2

# When using other BLAS
export OPENBLAS_NUM_THREADS=2 # OpenBLAS
export VECLIB_MAXIMUM_THREADS=2  # Apple BLAS

[EricLee543]

I encountered the same problem. It's due to Intel MKL spawning a lot of additional threads in the background.

You can limit the number of threads by setting these environment variables before you run your program. CPU utilization should be a lot more reasonable then.

export OMP_NUM_THREADS=2
export MKL_NUM_THREADS=2
export MKL_THREADING_LAYER=SEQUENTIAL
export NUMEXPR_NUM_THREADS=2

# When using other BLAS
export OPENBLAS_NUM_THREADS=2 # OpenBLAS
export VECLIB_MAXIMUM_THREADS=2  # Apple BLAS

Very useful suggestion! solved a huge problem for me