Build System and Versioning changes (#239)

* Rework the build system

- Adds a pyproject.toml with build dependencies, so users can also easily
  do a "pip install"

- Update MANIFEST.in file for sdist

- add a test_requirements.txt for easy-install of any dependencies
  necessary for testing

- Update README with build instructions

Changes in setup.py:

- remove Bluegene stuff, it wasn't used anywhere in the code at all
- make some cosmetic changes
- allow doing things like "sdist" and "clean"
  without depending on Cython
- Depend atleast on Cython 0.29.30 as minimum version now.
  This way we can mostly make sure that users have recent Cython
  version to compile

Version naming is changed:

Make up the pyslurm version from the Slurm Major release (e.g. 22.5)
and the current pyslurm patch-level for this major release, so we have
for example:

22.5.0

We must make sure (document it) that users don't confuse this with
Slurms patch version

* Disable the auto_pickle feature which was causing
that pyslurm wasn't able to be compiled on some kernels.

auto pickling may also not be really needed in pyslurm,
because by default classes with pointers/structs as
attributes aren't generated with pickle support by
cython anyway.

For more info, check #236

Fixes #236

* Use libslurm.so instead of libslurmfull.so

libslurm should be used for interfacing with the C-API,
libslurmfull is more internal to the Slurm tools itself and cannot
be guaranteed to be stable when used externally.

No functions from libslurmfull were actually used
in pyslurm.pyx so we can safely make the switch now.
Also removes a few functions in slurm/extra.pxi, which are
in libslurmfull but not used anywhere in the code

Fixes #209

Co-authored-by: tazend <[email protected]>

Author: tazend

MANIFEST.in

@@ -1,11 +1,8 @@
-include pyslurm/alps_cray.h
-include pyslurm/bluegene.pxi
-include pyslurm/slurm_defines.pxi
-include pyslurm/slurm.pxd
-include pyslurm/xmalloc.h
 include README.rst
 include COPYING.txt
-include THANKS.rst
 graft examples
 graft tests
 graft doc
+graft pyslurm/slurm
+graft pyslurm/pydefines
+include pyslurm/alps_cray.h

README.md

@@ -8,56 +8,51 @@ PySlurm is the Python client library for the [Slurm](https://slurm.schedmd.com)
 
 ## Prerequisites
 
-* [Slurm](https://slurm.schedmd.com)
-* [Python](https://www.python.org)
-* [Cython](https://cython.org)
+* [Slurm](https://slurm.schedmd.com) - Slurm shared library and header files
+* [Python](https://www.python.org) - >= 3.6
+* [Cython](https://cython.org) - >= 0.29.30 but < 3.0
 
-This PySlurm branch has been tested with:
-
-* Cython (latest stable)
-* Python 3.6, 3.7, 3.8, and 3.9
-* Slurm 22.05
+This PySlurm branch is for the Slurm Major-Release 22.05
 
 ## Installation
 
-You will need to instruct the setup.py script where either the Slurm install
-root directory or where the Slurm libraries and Slurm header files are.
+By default, it is searched inside `/usr/include` for the Header files and in
+`/usr/lib64` for Slurms shared-library (`libslurm.so`) during Installation.
+For Slurm installations in different locations, you will need to provide
+the corresponding paths to the necessary files.
 
-### Slurm installed using system defaults (/usr)
+You can specify these Paths with environment variables, for example:
 
 ```shell
-python setup.py build
-python setup.py install
+export SLURM_INCLUDE_DIR=/opt/slurm/22.05/include
+export SLURM_LIB_DIR=/opt/slurm/22.05/lib
 ```
 
-### Custom installation location
+Then you can proceed to install PySlurm, for example:
 
 ```shell
-python setup.py build --slurm=PATH_TO_SLURM_DIR
-python setup.py install
+pip install pyslurm==22.05.0
 ```
 
-### Custom Slurm library and include directories
+Or by cloning the repository:
 
 ```shell
-python setup.py build --slurm-lib=PATH_TO_SLURM_LIB --slurm-inc=PATH_TO_SLURM_INC
+git clone https://github.com/PySlurm/pyslurm.git && cd pyslurm
 python setup.py install
-```
 
-### Indicate Blue Gene type Q on build line
-
-```shell
-python setup.py build --bgq
+# Or simply with pip
+pip install .
 ```
 
-### Cleanup build artifacts
+Also see `python setup.py --help`
 
-The build will automatically call a cleanup procedure to remove temporary build
-files but this can be called directly if needed as well with :
+## Release Versioning
 
-```shell
-python setup.py clean
-```
+PySlurm's versioning scheme follows the official Slurm versioning. The first
+two numbers (MAJOR.MINOR) always correspond to Slurms Major-Release, for example
+`22.05`. The last number (MICRO) is however not tied in any way to Slurms
+MICRO version. For example, any PySlurm 22.05.X version should work with any
+Slurm 22.05.X release.
 
 ## Documentation
 
@@ -138,8 +133,7 @@ already compiled and installed:
 ```shell
 git clone https://github.com/PySlurm/pyslurm.git
 cd pyslurm
-python3.9 setup.py build
-python3.9 setup.py install
+pip install .
 ./scripts/configure.sh
 pipenv sync --dev
 pipenv run pytest -sv

pyproject.toml

@@ -0,0 +1,8 @@
+[build-system]
+# Minimum requirements
+requires = [
+    "setuptools==59.2.0",
+    "wheel==0.37.0",
+    "Cython>=0.29.30,<3.0",
+]
+

pyslurm/__version__.py

@@ -1 +1 @@
-__version__ = "22.05.2.0"
+__version__ = "22.5.0"

pyslurm/pyslurm.pyx

@@ -1,6 +1,7 @@
 # cython: embedsignature=True
 # cython: profile=False
 # cython: language_level=3
+# cython: auto_pickle=False
 import os
 import re
 import sys

pyslurm/slurm/extra.pxi

@@ -26,12 +26,10 @@ cdef inline void xfree_ptr(void *__p):
 
 cdef extern char *slurm_xstrdup(const char *str)
 
-
 #
 # Slurm time functions
 #
 
-
 cdef extern void slurm_secs2time_str(time_t time, char *string, int size)
 cdef extern void slurm_mins2time_str(time_t time, char *string, int size)
 cdef extern int slurm_time_str2mins(const char *string)
@@ -43,7 +41,6 @@ cdef extern time_t slurm_parse_time(char *time_str, int past)
 # Slurm Job functions
 #
 
-
 cdef extern void slurm_free_job_desc_msg(job_desc_msg_t *msg)
 cdef extern void slurm_free_job_info(job_info_t *job)
 cdef extern void slurm_free_job_info_members(job_info_t *job)
@@ -55,16 +52,11 @@ cdef extern char *slurm_job_share_string(uint16_t shared)
 
 #
 # Slurm environment functions
-#
 
 cdef extern void slurm_env_array_merge(char ***dest_array, const char **src_array)
 cdef extern char **slurm_env_array_create()
 cdef extern int slurm_env_array_overwrite(char ***array_ptr, const char *name, const char *value)
 cdef extern void slurm_env_array_free(char **env_array)
-# cdef extern void slurm_env_array_merge_slurm(char ***dest_array, const char **src_array)
-
-
-cdef extern int slurm_select_fini()
 
 #
 # Misc
@@ -78,9 +70,4 @@ cdef extern void slurm_free_stats_response_msg (stats_info_response_msg_t *msg)
 cdef extern int slurm_addto_char_list_with_case(List char_list, char *names, bool lower_case_noralization)
 cdef extern int slurm_addto_step_list(List step_list, char *names)
 cdef extern int slurmdb_report_set_start_end_time(time_t *start, time_t *end)
-cdef extern int debug_str2flags(char *debug_flags, uint64_t *flags_out)
-cdef extern char *debug_flags2str(uint64_t debug_flags)
-cdef extern void slurm_sprint_cpu_bind_type(char *str, cpu_bind_type_t cpu_bind_type)
 cdef extern uint16_t slurm_get_track_wckey()
-cdef extern char *select_type_param_string(uint16_t select_type_param)
-

setup.cfg

@@ -16,7 +16,7 @@ build_requires = python-devel >= 2.7
                  slurm-devel >= 17.11.5
                  python-nose
 requires = slurm-slurmd slurm-slurmdbd
-use-bzip2 = 1
+use_bzip2 = 1
 
 [build_sphinx]
 builder = man