Use black to reformat code

[vondele]

this reformats the code using black --exclude=worker/requests .

fixes #634

as with any tool, not all is perfect, but things are pretty consistent.
I propose to apply this patch (or execute the above black command) as part of the next worker change.

[ppigazzini]

@vondele incredible coincidence, I created locally the same branch yesterday :)
"assert" is a statement and not a function, in master was written with parenthesis in a dangerous way, black fixes this dubious coding style.

https://stackoverflow.com/questions/13390401/design-of-python-why-is-assert-a-statement-and-not-a-function
https://stackoverflow.com/questions/3112171/python-assert-with-and-without-parenthesis

[ppigazzini]

@vondele what about sorting the imports using isort --profile black --skip worker/requests . ?

[vondele]

$ isort --profile black --skip worker/requests .
[...]
isort: error: unrecognized arguments: --profile .

(VERSION 4.3.4)

Edit: fixed with pip3 install isort==5.4.2

[ppigazzini]

Tested OK on DEV/workers.

[tomtor]

Looks good to me

[ppigazzini]

@vondele please format and sort the new master.

[vondele]

@ppigazzini updated to the latest master.

[ppigazzini]

Fast check OK on DEV and workers, PROD updated, thank you @vondele :)

[ppigazzini]

I added this GitHub action on my repo to check format and import order on PR:
https://github.com/ppigazzini/fishtest/blob/master/.github/workflows/python-app.yml

After some tests we could add the action of the official repo.

BTW it seems that different black versions work in different ways, the action failed on master and I was able to reformat some files:

../env_black/bin/black --version
black, version 20.8b1

../env_black/bin/black --exclude=worker/requests .
reformatted /home/usr00/_git/fishtest/fishtest/fishtest/__init__.py
reformatted /home/usr00/_git/fishtest/fishtest/fishtest/stats/brownian.py
reformatted /home/usr00/_git/fishtest/fishtest/fishtest/stats/LLRcalc.py
reformatted /home/usr00/_git/fishtest/fishtest/fishtest/stats/sprt.py
reformatted /home/usr00/_git/fishtest/fishtest/fishtest/stats/stat_util.py
reformatted /home/usr00/_git/fishtest/fishtest/fishtest/util.py
reformatted /home/usr00/_git/fishtest/fishtest/fishtest/rundb.py
reformatted /home/usr00/_git/fishtest/worker/games.py
All done! ✨ 🍰 ✨
8 files reformatted, 29 files left unchanged.

[vondele]

Here:
black, version 19.3b0
(Edit: I can probably upgrade)

[ppigazzini]

It's a problem only if adding a github action to check the format.

BTW it seems not possible to auto format a PR opened from a fork, see:
https://github.com/cclauss/autoblack
https://peterevans.dev/posts/github-actions-how-to-automate-code-formatting-in-pull-requests/

[vondele]

well if not all devs use the same version of black, probably PRs will be a bit cluttered.

[ppigazzini]

These should be the actual smart ways to setup a python dev env:
https://jacobian.org/2019/nov/11/python-environment-2020/
https://cjolowicz.github.io/posts/hypermodern-python-01-setup/

[vondele]

btw, I just checked that the older version of black I use doesn't change back the changes made by the newer black. All changes where in the documentation strings of functions. Presumably that was like a new feature introduced.

[ppigazzini]

@vondele @tomtor @vdbergh IMO this is interesting https://jacobian.org/2019/nov/11/python-environment-2020/

pyenv

I updated my dev env, DEV and my test workers to python3.8.5 to test the stability, here is a benchmark for the python releases (PROD uses 3.6.9)
https://docs.python.org/3.9/whatsnew/3.9.html#optimizations

Notes:

• pyenv-installer slowed down my shell with pyenv-virtualenv (YMMV), so I reinstalled cloning the repo.
• on my Ubuntu servers (and WSL) I put the variables in .profile (sudo su -i -u seems to skip .bashrc)
• pyenv don't view system python on Ubuntu 20.04, not a big trouble, there is a wip PR
  Pyenv can't find system python (3.8) on Ubuntu 20.04 pyenv/pyenv#1613
  "system" python - support cases where python3 is in PATH but not python pyenv/pyenv#1655
• tutorial https://realpython.com/intro-to-pyenv/

pipx

Useful to have black, isort, awscli and other cli python tools in only one standard venv

poetry

Not really used yet :)

[ppigazzini]

@vondele @tomtor @vdbergh python 3.8.6 is saving a lot of memory on PROD.
Here is a script to setup on ubuntu/wsl a python dev env with pyenv, pipx, jupyterlab, nvm (required by jupyterlab):

#!/bin/bash
# 200928
# start the script with:
# bash setup_python_devenv.sh

# setup pyenv and install latest python (3.8.6)
# https://github.com/pyenv/pyenv

if ! command -v pyenv &>/dev/null; then
  python_ver="3.8.6"
  sudo apt update; sudo apt install -y --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

  pyenv_dir="${HOME}/.pyenv"
  git clone https://github.com/pyenv/pyenv.git "$pyenv_dir"

  cat << EOF >> ${HOME}/.profile
  
# pyenv: keep at the end of the file
export PYENV_ROOT="\$HOME/.pyenv"
export PATH="\$PYENV_ROOT/bin:\$PATH"
if command -v pyenv &>/dev/null; then
  eval "\$(pyenv init -)"
fi
EOF

  . ${HOME}/.profile
  pyenv install ${python_ver}
  pyenv global ${python_ver}
fi

# setup pipx and install some cli python packages
# https://github.com/pipxproject/pipx

if ! command -v pipx &>/dev/null; then
  python3 -m pip install --user pipx
  python3 -m pipx ensurepath
  . ${HOME}/.profile
  
  for pkg_name in black isort flake8; do
    if ! command -v ${pkg_name} &>/dev/null; then
      pipx install ${pkg_name}
    fi
  done
fi


# setup nvm and install latest node.js
# https://github.com/nvm-sh/nvm
# required to install jupyterlab extensions (ipywidgtes)

if [ ! -d "${HOME}/.nvm" ]; then
  nvm_dir="${HOME}/.nvm"
  git clone https://github.com/nvm-sh/nvm.git "$nvm_dir"
  cd "$nvm_dir"
  git checkout `git describe --abbrev=0 --tags --match "v[0-9]*" $(git rev-list --tags --max-count=1)`
  . "${nvm_dir}/nvm.sh"
  nvm install node

  cat << EOF >> ${HOME}/.profile

# nvm
export NVM_DIR="\$HOME/.nvm"
# [ -s "\$NVM_DIR/nvm.sh" ] && . "\$NVM_DIR/nvm.sh" --no-use
EOF
fi


# setup jupyterlab and configure the ipywidgets extension
# https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html
# https://ipywidgets.readthedocs.io/en/latest/user_install.html

if [ ! -d "${HOME}/_jupyterlab" ]; then
  jupy_dir="${HOME}/_jupyterlab"  
  mkdir "${jupy_dir}"
  cd "${jupy_dir}"
  
  # load nvm
  . "${NVM_DIR}/nvm.sh"
  
  # create a virtual environment
  python3 -m venv env
  
  # update virtual environment default packages
  "${jupy_dir}"/env/bin/python3 -m pip install --upgrade pip setuptools wheel
  
  # install jupyterlab and configure the ipywidgets extension
  "${jupy_dir}"/env/bin/python3 -m pip install jupyterlab
  "${jupy_dir}"/env/bin/python3 -m pip install ipywidgets
  "${jupy_dir}"/env/bin/jupyter labextension install @jupyter-widgets/jupyterlab-manager
  
  # install some scientific packages
  "${jupy_dir}"/env/bin/python3 -m pip install numpy scipy matplotlib scikit-learn pandas
  
  # start jupyterlab
  "${jupy_dir}"/env/bin/jupyter lab --no-browser
   
  # Instructions:
  #  - check the WSL console output below, find the jupyter address, something like this:
  #    http://localhost:8888/?token=4af54d9d47012e5a9c595111dab0c6ee9e469b8a0a2cd9a6
  #  - select the address with the mouse left button
  #  - press the "Return" key button to copy 
  #  - open a browser and paste the address
fi