V2.0.4 (#258)

* added a new method to blobcorrector.correctorclass to do distorsion correction using a pixel LUT * Attempt to manage openmp with multiprocessing * Attempt to manage openmp with multiprocessing * Attempt to manage openmp with multiprocessing * drop 3.7 for 3.8 (but keep 2.7 for now) * Update blobcorrector.py Added new property self.dim with detector dimension to blobcorrector.correctorclass, inferred from the splinefile (xmin, xmax, ymin, ymax) * Update blobcorrector.py * TODO: fix fscan2d to be easier to use * Thanks to Andrew Reader's youtube video * added 3.11 (3.x is busted due to numba) * auto reshape for fscan2d * setting threads only for multiprocessing * fixup fscan2d * fixup fscan2d * Update build_flake_pytest_ubuntu2004.yml * Update build_flake_pytest_ubuntu2004.yml * Update setup.py * Update sigma_3_matrices.py (py2, print) * Update properties.py (py2: print) * Update test_datasets.py (py2: fstring/print) * Update cImageD11.py : use OMP_NUM_THREADS and py27 * Enable manually specifying detector and motor names * matrix vs dcm with old scipy * fix py27 windows * binary wheels for win32/py27 * 'fixup' * 'fixup' * 'fixup' * Update appveyor.yml * Update appveyor.yml * Update appveyor.yml * Update appveyor.yml * Update appveyor.yml * Update appveyor.yml * Update appveyor.yml * Add build wheels and tarball * Update README.md * Update __init__.py * Create eps_sig_solver_NEW.py Create a new solver to compute strain and stress from ImageD11 ubis * Bugfix assemble_label.py set rather than list randomly permuted the order of scans. Oups. * compatible for python2.7 * fix file path * corrected mistakes in eps_sig_solve_NEW.py + updated invariant properties function - corrected a few mistakes and typos in function description - corrected a mistake in convert_tensors_to_vect - changed invariant_prop function to return von Mises stress / strain instead of octahedral shear stress / strain * Update eps_sig_solver_NEW.py * corrected typo in form_stiffness_tensor * updated Cij_symmetry * Update eps_sig_solver_NEW.py * Rename eps_sig_solver_NEW.py to stress.py * Rename ImageD11/eps_sig_solver.py to ImageD11/depreciated/eps_sig_solver.py * Update stress.py * Update stress.py * Update stress.py * Update stress.py * Update stress.py * Add scanning 3DXRD example notebooks * Remove f-string for 2.7 compat. * Exclude Python 3.12 in setup.py Until Numba updates to Python 3.12 This should fix the CI for Python 3.x * Exclude 3.12 from GitHub Actions CI * Fix GitHub actions yml typo * Third time lucky * Use GitHub Actions badges in README.md * Correct typo * Update README.md * fix the log strain - needs testing better * helper function from several notebooks * mlem and centers code * function from notebooks * function from notebooks * Change int32 and float32 to 64 * remove broken test * put stress on hold while updating it * Update __init__.py v2.0.2 * Update README.md v2.0.2 * Update release.yml : no 3.12 version * Allow a fuller mask, to mask powder rings for example * fix mask for powder removal and reformat (black) * Resurrect tests for columnfile.py, correct np.int In the Github Actions CI, the command "python -m pytest" is run to test ImageD11. By default, this only runs test files with the naming convention `test_*.py` Renamed `testcolumnfile.py` to `test_columnfile.py` to fix this. Also, replaced `np.int` references `int` to fix numpy depreciation errors. * Fix np.int64 * Add colfile tests to run_tests.py * Significant notebook layout changes * Parallelised iradon * Ensure 0 fill value for np.interp * Add mask to MLEM * First attempt at enforcing file paths * A few bugfixes * update cibuildwheel and upload|download-artifacts * eps_sig_solver was deprecated * eps_sig_solver was deprecated + black formatted * Python 3.12 support * Add col3d file to dataset paths * Try to get rid of the fork problem * point by point - needs some more docs to expain pars * fixup 2.7 * Add outsize support to MLEM * Clean up comment * convert nnz to ip function * error on typo in group names * fix forking problem/error * get the warnings to print, but only when needed * for CI only having 1 thread? * CI fails. Not sure why. * fixup * set forkserver here, todo: windows? * and for windows that does not fork anyway * and for windows that does not fork anyway * and for windows that does not fork anyway * percent, not fraction * tested on SS316 * was not tested yet. pythonpath * tested once: should not be forking * seen to run once * with a threshold image option * Notebook for par file building from poni * A few small sinogram bugfixes * self.shape is undefined * Significant scanning notebook changes General changes: - Moved quite a few common functions into utils.py. These need properly incorporating into ImageD11 in the future - All notebooks updated to obey DataSet filestructures for peaks and grains files 0_S3DXRD_segment: - Added widget written by Pierre-Olivier for adjusting segmenter parameters - Added slurm bash "listener" that waits for slurm job to complete before next cell completes - Moves individual sparse files to their own subfolder after assembling 1_S3DXRD_index: - More sensible indexer that prioritises low-multiplicity rings - Grains now saved in HDF5 format - Added another example notebook for indexing a minor phase 2_S3DXRD_sinograms: - Added whole-sample sinogram reconstruction to use as a mask for individual grain reconstructions. - Added MLEM refinement stage (uses SLURM) to grain reconstructions using run_mlem_recon.py - Added another example notebook for mapping a minor phase (uses grain recons for centre-of-mass) 3_S3DXRD_strain_maps_pbp: - Performs a point-by-point refinement of the grain-averaged UBI using 2D peak selection - Uses grain labels array from sinograms for grain shapes - Should give "relative" Type-III strains 4_S3DXRD_plot_both_phases: - Small example notebook for plotting both phases in the same map - Some initial work on orientation relationship identification using Orix (not finished yet) * Add box-beam example notebooks - Similar to scanning notebooks but for box-beam 3DXRD - Currently only working for Frelon data - Uses Frelon segmenter from ma4750 (in sandbox) for peaksearching - Global parameter refinement notebook - Uses makemap.py to refine grain positions - Some extra notebooks for importing multiple slices in a z series and looking at peak shapes (still under development) * Build parameter file from CIF and pyFAI PONI * Clear notebook outputs * Correct pyplot typo * Allow flake 2.7 linting to pass * Python 2 compat * Remove duplicate segmentation notebook * Common utils between scanning and box-beam notebooks - Moved a few more functions out of notebooks into utils - This significantly simplifies the notebooks - MLEM recon: y0 now gets written to/read from the grain HDF5 file rather than command-line option * Replace FileNotFoundError with IOError for py2.7 * Minor phase scanning notebooks now use nb_utils, needs testing * Correct ImageD11_gui bugs - broken log buttons and tth_histogram * Rename and add batch scripting * Add batch cells to scanning notebooks * All notebooks now use nb_utils.py * Remove WIP notebook from commit * Add RGB colour output to sinogram notebooks * Add widget with sliders for Frelon peaksearch data * Add RGB IPF X,Y,Z outputs to HDF, allows viewing with H5Web * py2 compat * Small bugfixes and tidying up * Small changes and bugfixes * Clear output * secret spacegroup number hidden option * secret spacegroup number hidden option * secret spacegroup number hidden option * secret spacegroup number hidden option * Correct very dodgy default cosine tolerance * Correct grain translation sign problems * Correct dodgy do_index scan * Handle missing samples in skips_dict, dset lists are now sorted too * More accurately find user home directory path * Just use default arguments for assemble_label and properties * Force only 2 threads for indexer (confirmed faster) * Improve comments about peak filtration * Correct origin for recon plots * Much faster spatial correction using lut * Lima segmenter now used in interactive notebook * Add howmany to Eiger interactive segmenter * Segment everything in parallel on SLURM * Cleaned-up local PBP indexer * Correct PBP results filename * Fix dodgy pbp name * Fix PBP output name for real this time * fixes #236 * fixes #236 * Correct home directory searching * added options to score_all_pairs for ring selection * added options to score_all_pairs for ring selection * Optionally scale sinogram rows by ring current * Forgot to also add to batch cells * Correct dodgy minor phase grain positioning * nb_utils function for ring current correction per scan * H5py in nb_utils * Make ring current correction optional and separate * More sensible default dstol * tag maintenance release --------- Co-authored-by: Jean-Baptiste Jacob <[email protected]> Co-authored-by: jadball <[email protected]> Co-authored-by: Thomas VINCENT <[email protected]> Co-authored-by: James Ball <[email protected]> Co-authored-by: Jerome Kieffer <[email protected]> Co-authored-by: Jonathan Wright <[email protected]>
FABLE-3DXRD · Mar 13, 2024 · 3cd02c7 · 3cd02c7
1 parent 1b83acc
commit 3cd02c7
Show file tree

Hide file tree

Showing 399 changed files with 79,461 additions and 9,359 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -0,0 +1,56 @@
+# Python CircleCI 2.0 configuration file
+#
+# Check https://circleci.com/docs/2.0/language-python/ for more details
+#
+version: 2
+jobs:
+  build:
+    docker:
+      # specify the version you desire here
+      # use `-browsers` prefix for selenium tests, e.g. `3.6.1-browsers`
+      - image: circleci/python:3.8.0
+
+      # Specify service dependencies here if necessary
+      # CircleCI maintains a library of pre-built images
+      # documented at https://circleci.com/docs/2.0/circleci-images/
+      # - image: circleci/postgres:9.4
+
+    working_directory: ~/repo
+
+    steps:
+      - checkout
+
+      # Download and cache dependencies
+      - restore_cache:
+          keys:
+          # fallback to using the latest cache if no exact match is found
+          - v1-dependencies-
+
+      - run:
+          name: install dependencies
+          command: |
+            python3 -m venv venv
+            . venv/bin/activate
+            python3 -m pip install pip setuptools --upgrade
+            python3 -m pip install pytest numpy
+            python3 -m pip install --editable .
+
+      - save_cache:
+          key:  ImageD11-{{arch}}-{{ checksum "setup.py" }}
+          paths:
+            - ./venv
+
+      # run tests
+      # this example uses Django's built-in test-runner
+      # other common Python testing frameworks include pytest and nose
+      # https://pytest.org
+      # https://nose.readthedocs.io
+      - run:
+          name: run tests
+          command: |
+            . venv/bin/activate
+            python3 -m pytest
+
+      - store_artifacts:
+          path: test-reports
+          destination: test-reports
diff --git a/.gitattributes b/.gitattributes
@@ -1,4 +1,12 @@
+*.out_merge_t200 linguist-generated
+*.out  linguist-generated
 *.flt  linguist-generated
+*.gve  linguist-generated
+*.ubi  linguist-generated
+*.par  linguist-generated
+*.prm  linguist-generated
+*.pars  linguist-generated
 *.html linguist-documentation
 *.pdf  linguist-generated
 *.odt  linguist-documentation
+*.pks  linguist-generated
diff --git a/.github/workflows/build_flake_pytest_ubuntu2004.yml b/.github/workflows/build_flake_pytest_ubuntu2004.yml
@@ -0,0 +1,51 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: ImageD11_pytest
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  build:
+    # because ESRF forces 20.04 for now.
+    runs-on: ubuntu-20.04
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["2.7", "3.8", "3.9", "3.10", "3.11", "3.12"]
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      if: matrix.python-version != '2.7'
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install Ubuntu python 2.7
+      if: matrix.python-version == '2.7'
+      run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends python2 python2-dev python3-virtualenv
+          virtualenv -p python2 ${HOME}/cp27
+          source ${HOME}/cp27/bin/activate
+          ${HOME}/cp27/bin/python -m pip install -U pip
+          ${HOME}/cp27/bin/python -m pip install -U setuptools wheel
+          echo "${HOME}/cp27/bin" >> $GITHUB_PATH
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install flake8 pytest
+        if [ -f requirements.txt ]; then python -m pip install -r requirements.txt; fi
+        python -m pip install .
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        python -m flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics --exclude="*depreciated*"
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        # flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: Test with pytest
+      run: |
+        cd test
+        python -m pytest
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -0,0 +1,115 @@
+name: Build and deploy
+
+on:
+  workflow_dispatch:
+  release:
+    types:
+      - published
+
+jobs:
+  build_wheels:
+    name: Build wheels on ${{ matrix.os }}-${{ matrix.cibw_archs }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      # Ensure that a wheel builder finishes even if another fails
+      fail-fast: false
+      matrix:
+        include:
+          - os: ubuntu-20.04
+            cibw_archs: "auto64"
+          #- os: ubuntu-20.04
+          #  cibw_archs: "auto32"
+          - os: ubuntu-20.04
+            cibw_archs: "aarch64"
+          - os: ubuntu-20.04
+            cibw_archs: "ppc64le"
+          - os: windows-2019
+            cibw_archs: "auto64"
+          #- os: windows-2019
+          #  cibw_archs: "auto32"
+          - os: macos-11
+            cibw_archs: "universal2"
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up QEMU
+        if: runner.os == 'Linux'
+        uses: docker/setup-qemu-action@v3
+        with:
+          platforms: all
+
+      - name: Build wheels
+        uses: pypa/[email protected]
+        env:
+          CIBW_PROJECT_REQUIRES_PYTHON: ">=3.7"
+          CIBW_BUILD: cp37-* cp38-* cp39-* cp310-* cp311-* cp312-*
+          # Do not build for pypy, muslinux and python3.12 on ppc64le
+          CIBW_SKIP: pp* *-musllinux_* cp312-*linux_ppc64le
+          CIBW_ARCHS: ${{ matrix.cibw_archs }}
+
+          # Use silx wheelhouse: needed for ppc64le
+          CIBW_ENVIRONMENT_LINUX: "PIP_FIND_LINKS=https://www.silx.org/pub/wheelhouse/ PIP_TRUSTED_HOST=www.silx.org"
+
+          CIBW_TEST_REQUIRES: pytest
+          CIBW_TEST_COMMAND: pytest {project}/test
+          # Skip tests for 32bits and emulated architectures, arm64 macos and on Windows
+          CIBW_TEST_SKIP: "*-*linux_i686 *-*linux_{aarch64,ppc64le,s390x} *-macosx_arm64 *-macosx_universal2:arm64 *-win32 *-win_amd64"
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
+          path: ./wheelhouse/*.whl
+
+  build_sdist:
+    name: Build source distribution
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build twine
+
+      - name: Build sdist
+        run: python -m build --sdist
+
+      - name: Check the package
+        run: |
+          python -m twine check dist/*
+
+      - name: Install and test sdist
+        run: |
+          pip install --pre dist/ImageD11*.tar.gz
+          pytest test
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: cibw-sdist
+          path: dist/*.tar.gz
+
+  pypi-publish:
+    needs: [build_wheels, build_sdist]
+    name: Upload release to PyPI
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+    permissions:
+      id-token: write
+    if: github.event_name == 'release' && github.event.action == 'published'
+    # or, alternatively, upload to PyPI on every tag starting with 'v' (remove on: release above to use this)
+    # if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          pattern: cibw-*
+          path: dist
+          merge-multiple: true
+
+      - uses: pypa/gh-action-pypi-publish@release/v1
diff --git a/.gitignore b/.gitignore
@@ -18,6 +18,7 @@ __pycache__/
 \#*#
 
 # Distribution / packaging
+.vscode
 .Python
 env/
 build/

diff --git a/.travis.yml b/.travis.yml
@@ -0,0 +1,18 @@
+language: python            # this works for Linux but is an error on macOS or Windows
+jobs:
+  include:
+    - name: "Python 3.8.0 on Xenial Linux"
+      python: 3.8           # this works for Linux but is ignored on macOS or Windows
+    - name: "Python 3.7.4 on macOS"
+      os: osx
+      osx_image: xcode11.2  # Python 3.7.4 running on macOS 10.14.4
+      language: shell       # 'language: python' is an error on Travis CI macOS, could be generic
+install: 
+  - python3 -m pip install --upgrade pip setuptools  
+# fabio could not be installed as a dependency on mac automatically
+  - python3 -m pip install cython
+  - python3 -m pip install fabio
+  - python3 -m pip install --no-use-pep517 --editable .  
+# 'python3' is a 'command not found' error on Windows but 'py' works on Windows only
+script: 
+  - OMP_NUM_THREADS=2 python3 -m pytest
-Original file line number
+Diff line change
@@ Expand Up / @@ -18,6 +18,7 @@ __pycache__/ @@
     \#*#
     # Distribution / packaging
+    .vscode
     .Python
     env/
     build/
@@ Expand Down @@