v0.3 (#22)

* Ir (#1) * update script, needs refactoring * update * fix benchmark * run at least for 10 ts * rename GKOBase to GKOlduBase * create ldu and Ldu matrices * add ldu GKOCG and GKOBiCGStab * add Ldu matrix * add asymetric solver * clean up * start implementing of stopping criteria * add common file * update common * inject c++14 * update * use modified boxTurb16 case * update benchmark * keep CF constant * start reuse sorting idx * add IR solver, make sorting optional * add line buffering, further cli args * Add basic implementation of storing sorting idxs * add IR implementation * make sorting default * store gkomatrix * add common.C * make executor an IOPtr * prepare for forHLR test * fix block clean for different block start syntax * transfer to desktop * hardcoded stored omp executor * load different executor * fix min/maxIter bug * debug reference executor issue * add clang format to keep formating constant * fix cuda executor * clean up * clean up * remove unneeded bash benchmark script * format + prepare preconditioner * run omp benchmarks for different number of threads * make pressure solver settings compatible with OF8 * changed size of cases * fix benchmarking folders * set dns case boundaries to walls * Fix several compiler warnings, reordering issues, * create additional executor to force eager copy * log succes state of run, keep retrying on failure at least for now * move OpenFoam stopping criterion to separate file * Use Foam switches to read dictionary entries * Make export system a free function, clean logger and imports * Move IterationLogger to common.H * Fix execution if sysMatrix is not stored if sysMatrix is not stored and thus the gkomatrix_ptr==NULL the system matrix is recreated * fix loop ctr * Fix IR solver - initialize IR with CG - remove BJ for now * Add Solver classes to distinguish between OF and GKO solvers, set time out for runs, fix log files * fix formating * Fix typos in smoothSolver class * Fix int to unsingned long conversion * Add initial CMakeLists file * Add basic cmake files * Fix return type warning * move setting device_exec to separate function * add third_party folder * Update README.md * Update README.md * Use FOAM_ enviromental variables * install libginkgo.so files along with libOGL.so if no external ginkgo is used * Create a initial cmake.yml file * clone and source OpenFOAM * Fix non ascii symbol * display OpenFOAM folder content * show content of OpenFOAM-8 folder * Update cmake.yml * set OpenFOAM env variables manually * add build badge * Update Readme file * remove Test folder since it is no longer needed * re-enable exporting the system matrix, rhs and initial guess * Read maxBlockSize from dictionary * Rename controlDict_ member variable to dictionary_ * Make BiCGStabFactory consistent with CGFactory * Clean unneeded comment * Fix sorting bounds of sorting_idxs * Update cmake.yml * add basic travis file * remove github actions file * update .travis.yml * fix OpenFOAM path * update travis file * compile full OpenFOAM for now * install openmpi-dev * install flexc++ * add OpenFOAM-dev to build matrix * add fl dev * install openfoam from the repository * for now just test if OGL builds * add build badge * fix sourcing depending on of version * switch to ubuntu 20.04 * make OGL_USE_EXTERNAL_GINKGO an option * use --no-check-certificate since importing the key fails regularily * pass --no-check-certificate to wget * fix type of tmp_col_idxs and tmp_row_idxs * sort the system matrix to avoid cuda convergence issue * add preconditioner to path * remove benchmark script to separate repository * fix find_package * Fix find_package execution after external ginkgo was included #5 Co-authored-by: tcojean <[email protected]> * Add OpenFOAM version shields * Travis external ginkgo (#9) * add external ginkgo flag to ci/cd matrix * fix ci/cd matrix + test conditional job * add a test script for external ginkgo build * fix external ginkgo build flags to speedup build * Add missing sudo to install ginkgo to system. * fix cmake build arguments * Clean ldu ldu base (#13) * create separate IOExecutorHandler class * add IOExecutorHandler files * fix compilation issue * create separate IOSortingIdxHandler class * Implement separate IOGKOMatrixHandler class * move host matrix handling to separate class * fix matrix sorting/updating host matrix * Fix #10 remove deprecated wmake folder * This PR cleans the common lduLduBase class (#16) * create separate IOExecutorHandler class * add IOExecutorHandler files * fix compilation issue * create separate IOSortingIdxHandler class * Implement separate IOGKOMatrixHandler class * move host matrix handling to separate class * fix matrix sorting/updating host matrix * Reuse init guess (#19) * create separate IOExecutorHandler class * add IOExecutorHandler files * fix compilation issue * create separate IOSortingIdxHandler class * Implement separate IOGKOMatrixHandler class * move host matrix handling to separate class * fix matrix sorting/updating host matrix * implement reusable initial guess * fix host executor call * fix device_x size, IR improvements * Split init and update of host matrix from sparsity pattern (#21) Co-authored-by: tcojean <[email protected]>
hpsim · May 2, 2021 · 266d079 · 266d079
1 parent 781cfd1
commit 266d079
Show file tree

Hide file tree

Showing 19 changed files with 1,122 additions and 661 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -8,8 +8,10 @@ os:
   - linux
 
 env:
-    - VERSION=8
-    - VERSION=7
+    - VERSION=8 EXTERNAL_GINKGO=OFF
+    - VERSION=8 EXTERNAL_GINKGO=ON
+    - VERSION=7 EXTERNAL_GINKGO=OFF
+    - VERSION=7 EXTERNAL_GINKGO=ON
 
 cache:
   directories:
@@ -33,12 +35,13 @@ before_install:
 install:
   - sudo apt-get -y install openfoam$VERSION
   - source /opt/openfoam$VERSION/etc/bashrc
+  - ./scripts/travis_install_external_ginkgo.sh
 
 before_script:
-  - cmake --version
+  - cmake --version 
   - mkdir build
   - cd build
-  - cmake ..
+  - cmake -DOGL_USE_EXTERNAL_GINKGO=$EXTERNAL_GINKGO ..
 
 script:
   - make -j4
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -92,13 +92,23 @@ add_compile_definitions(
 
 target_sources(OGL
   PRIVATE
+common/common.C
+IOHandler/IOPtr/IOPtr.C
+IOHandler/IOSortingIdxHandler/IOSortingIdxHandler.C
+IOHandler/IOExecutorHandler/IOExecutorHandler.C
+IOHandler/IOGKOMatrixHandler/IOGKOMatrixHandler.C
 lduMatrix/GKOlduBase/GKOlduBase.C
 lduMatrix/GKOCG/GKOCG.C
 lduMatrix/GKOIR/GKOIR.C
 lduMatrix/GKOBiCGStab/GKOBiCGStab.C
 LduMatrix/GKOLduBase/GKOLduBase.C
 LduMatrix/GKOACG/GKOACG.C
   PUBLIC
+common/common.H
+IOHandler/IOPtr/IOPtr.H
+IOHandler/IOSortingIdxHandler/IOSortingIdxHandler.H
+IOHandler/IOExecutorHandler/IOExecutorHandler.H
+IOHandler/IOGKOMatrixHandler/IOGKOMatrixHandler.H
 lduMatrix/GKOlduBase/GKOlduBase.H
 lduMatrix/GKOCG/GKOCG.H
 lduMatrix/GKOIR/GKOIR.H
@@ -110,13 +120,17 @@ LduMatrix/GKOACG/GKOACG.H
 
 target_include_directories(OGL
   PUBLIC
-  $ENV{FOAM_SRC}/OpenFOAM/lnInclude
-  $ENV{FOAM_SRC}/meshTools/lnInclude
   $ENV{FOAM_SRC}/finiteVolume/lnInclude
+  $ENV{FOAM_SRC}/meshTools/lnInclude
+  $ENV{FOAM_SRC}/OpenFOAM/lnInclude
   $ENV{FOAM_SRC}/OSspecific/POSIX/lnInclude
-    common/
-    lduMatrix/GKOlduBase
-    LduMatrix/GKOLduBase
+  common/
+  IOHandler/IOPtr
+  IOHandler/IOExecutorHandler/
+  IOHandler/IOSortingIdxHandler/
+  IOHandler/IOGKOMatrixHandler/
+  lduMatrix/GKOlduBase
+  LduMatrix/GKOLduBase
   )
 
 

diff --git a/IOHandler/IOExecutorHandler/IOExecutorHandler.C b/IOHandler/IOExecutorHandler/IOExecutorHandler.C
@@ -0,0 +1,82 @@
+
+#include <ginkgo/ginkgo.hpp>
+#include <type_traits>
+
+#include "IOExecutorHandler.H"
+
+namespace Foam {
+
+IOExecutorHandler::IOExecutorHandler(const objectRegistry &db,
+                                     const dictionary &controlDict)
+    : device_executor_name_(
+          controlDict.lookupOrDefault("executor", word("reference"))),
+      app_executor_name_(
+          controlDict.lookupOrDefault("app_executor", word("reference")))
+{
+    // create executors
+    bool app_exec_stored = db.foundObject<regIOobject>(app_executor_name_);
+
+    if (app_exec_stored) {
+        ref_exec_ptr_ =
+            &db.lookupObjectRef<GKOReferenceExecPtr>(app_executor_name_);
+        if (device_executor_name_ == app_executor_name_) {
+            return;
+        }
+    }
+
+
+    bool device_exec_stored =
+        db.foundObject<regIOobject>(device_executor_name_);
+    if (device_exec_stored) {
+        if (device_executor_name_ == "omp") {
+            omp_exec_ptr_ =
+                &db.lookupObjectRef<GKOOmpExecPtr>(device_executor_name_);
+            return;
+        }
+        if (device_executor_name_ == "cuda") {
+            cuda_exec_ptr_ =
+                &db.lookupObjectRef<GKOCudaExecPtr>(device_executor_name_);
+            return;
+        }
+        if (device_executor_name_ == "hip") {
+            hip_exec_ptr_ =
+                &db.lookupObjectRef<GKOHipExecPtr>(device_executor_name_);
+            return;
+        }
+    }
+
+    const fileName app_exec_store = app_executor_name_;
+    ref_exec_ptr_ =
+        new GKOReferenceExecPtr(IOobject(app_exec_store, db),
+                                gko::give(gko::ReferenceExecutor::create()));
+
+    const fileName device_exec_store = device_executor_name_;
+    const fileName omp_exec_store = "omp";
+    omp_exec_ptr_ = new GKOOmpExecPtr(IOobject(omp_exec_store, db),
+                                      gko::OmpExecutor::create());
+
+    if (device_executor_name_ == "cuda") {
+        cuda_exec_ptr_ =
+            new GKOCudaExecPtr(IOobject(device_exec_store, db),
+                               gko::give(gko::CudaExecutor::create(
+                                   0, omp_exec_ptr_->get_ptr(), true)));
+    }
+    if (device_executor_name_ == "hip") {
+        hip_exec_ptr_ =
+            new GKOHipExecPtr(IOobject(device_exec_store, db),
+                              gko::give(gko::HipExecutor::create(
+                                  0, omp_exec_ptr_->get_ptr(), true)));
+    }
+    if (device_executor_name_ == "omp") {
+        omp_exec_ptr_ = new GKOOmpExecPtr(IOobject(device_exec_store, db),
+                                          omp_exec_ptr_->get_ptr());
+    }
+}
+
+defineTemplateTypeNameWithName(GKOExecPtr, "ExecIOPtr");
+defineTemplateTypeNameWithName(GKOCudaExecPtr, "CudaExecIOPtr");
+defineTemplateTypeNameWithName(GKOOmpExecPtr, "OmpExecIOPtr");
+defineTemplateTypeNameWithName(GKOHipExecPtr, "HipExecIOPtr");
+defineTemplateTypeNameWithName(GKOReferenceExecPtr, "ReferenceExecIOPtr");
+
+}  // namespace Foam
diff --git a/IOHandler/IOExecutorHandler/IOExecutorHandler.H b/IOHandler/IOExecutorHandler/IOExecutorHandler.H
@@ -0,0 +1,110 @@
+/*---------------------------------------------------------------------------*\
+License
+    This file is part of OGL.
+
+    OGL is free software: you can redistribute it and/or modify it
+    under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
+    ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+    FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+    for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with OGL.  If not, see <http://www.gnu.org/licenses/>.
+
+Class
+    Foam::IOExecutorHandler
+
+Author: Gregor Olenik <[email protected]>
+
+SourceFiles
+    IOExecutorHandler.C
+
+\*---------------------------------------------------------------------------*/
+
+#ifndef OGL_IOExecutorHandler_INCLUDED_H
+#define OGL_IOExecutorHandler_INCLUDED_H
+
+#include "../IOPtr/IOPtr.H"
+#include "fvCFD.H"
+#include "regIOobject.H"
+
+#include <ginkgo/ginkgo.hpp>
+
+namespace Foam {
+
+using vec = gko::matrix::Dense<scalar>;
+using mtx = gko::matrix::Coo<scalar>;
+
+
+class IOExecutorHandler {
+private:
+    const word device_executor_name_;
+
+    const word app_executor_name_;
+
+    // executor where Ginkgo will perform the computation
+    GKOExecPtr *device_exec_ptr_;
+
+    // executor of the application
+    GKOExecPtr *app_exec_ptr_;
+
+    GKOReferenceExecPtr *ref_exec_ptr_;
+
+    GKOCudaExecPtr *cuda_exec_ptr_;
+
+    GKOOmpExecPtr *omp_exec_ptr_;
+
+    GKOHipExecPtr *hip_exec_ptr_;
+
+public:
+    IOExecutorHandler(const objectRegistry &db, const dictionary &controlDict);
+
+    word get_device_executor_name() const { return device_executor_name_; }
+
+    std::shared_ptr<gko::Executor> get_device_executor() const
+    {
+        const word device_executor_name{get_device_executor_name()};
+        if (device_executor_name == "omp") {
+            return omp_exec();
+        }
+        if (device_executor_name == "cuda") {
+            return cuda_exec();
+        }
+        if (device_executor_name == "hip") {
+            return hip_exec();
+        }
+        return ref_exec();
+    };
+
+    std::shared_ptr<gko::Executor> app_exec() const
+    {
+        return app_exec_ptr_->get_ptr();
+    };
+
+    std::shared_ptr<gko::OmpExecutor> omp_exec() const
+    {
+        return omp_exec_ptr_->get_ptr();
+    };
+
+    std::shared_ptr<gko::CudaExecutor> cuda_exec() const
+    {
+        return cuda_exec_ptr_->get_ptr();
+    };
+
+    std::shared_ptr<gko::HipExecutor> hip_exec() const
+    {
+        return hip_exec_ptr_->get_ptr();
+    };
+
+    std::shared_ptr<gko::ReferenceExecutor> ref_exec() const
+    {
+        return ref_exec_ptr_->get_ptr();
+    };
+};
+
+}  // namespace Foam
+#endif
diff --git a/IOHandler/IOGKOMatrixHandler/IOGKOMatrixHandler.C b/IOHandler/IOGKOMatrixHandler/IOGKOMatrixHandler.C
@@ -0,0 +1,97 @@
+/*---------------------------------------------------------------------------*\
+License
+    This file is part of OGL.
+
+    OGL is free software: you can redistribute it and/or modify it
+    under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
+    ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+    FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+    for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with OGL.  If not, see <http://www.gnu.org/licenses/>.
+
+Class
+    Foam::GKOCG
+
+Author: Gregor Olenik <[email protected]>
+
+SourceFiles
+    GKOCG.C
+
+\*---------------------------------------------------------------------------*/
+#include <ginkgo/ginkgo.hpp>
+#include "IOGKOMatrixHandler.H"
+
+namespace Foam {
+
+void IOGKOMatrixHandler::init_device_matrix(
+    const objectRegistry &db, std::vector<scalar> &values_host,
+    std::vector<label> &col_idxs_host, std::vector<label> &row_idxs_host,
+    const label nElems, const label nCells, const bool store) const
+{
+    std::shared_ptr<gko::Executor> device_exec = get_device_executor();
+
+    if (sys_matrix_stored_) {
+        gkomatrix_ptr_ = &db.lookupObjectRef<GKOCOOIOPtr>(sys_matrix_name_);
+        return;
+    }
+
+    std::shared_ptr<idx_array> col_idx;
+    std::shared_ptr<idx_array> row_idx;
+    if (sparsity_pattern_stored_) {
+        io_col_idxs_ptr_ =
+            &db.lookupObjectRef<GKOIDXIOPtr>(sparsity_pattern_name_cols_);
+        io_row_idxs_ptr_ =
+            &db.lookupObjectRef<GKOIDXIOPtr>(sparsity_pattern_name_rows_);
+        col_idx = io_col_idxs_ptr_->get_ptr();
+        row_idx = io_row_idxs_ptr_->get_ptr();
+    } else {
+        // if not stored yet create sparsity pattern from correspondent
+        // views
+        auto col_idx_view = idx_array::view(gko::ReferenceExecutor::create(),
+                                            nElems, &col_idxs_host[0]);
+        auto row_idx_view = idx_array::view(gko::ReferenceExecutor::create(),
+                                            nElems, &row_idxs_host[0]);
+
+        // copy sparsity pattern to device and leave it there
+        col_idx = std::make_shared<idx_array>(col_idx_view);
+        col_idx->set_executor(device_exec);
+        row_idx = std::make_shared<idx_array>(row_idx_view);
+        row_idx->set_executor(device_exec);
+    }
+
+    // if system matrix is not stored create it and set shared pointer
+    auto gkomatrix =
+        gko::share(mtx::create(device_exec, gko::dim<2>(nCells, nCells),
+                               val_array::view(gko::ReferenceExecutor::create(),
+                                               nElems, &values_host[0]),
+                               *col_idx.get(), *row_idx.get()));
+
+    // if updating system matrix is not needed store ptr in obj registry
+    if (store) {
+        const fileName path = sys_matrix_name_;
+        gkomatrix_ptr_ = new GKOCOOIOPtr(IOobject(path, db), gkomatrix);
+    } else {
+        gkomatrix_ptr_ = NULL;
+        gkomatrix_ = gkomatrix;
+    }
+
+
+    // in any case store sparsity pattern
+    const fileName path_col = sparsity_pattern_name_cols_;
+    const fileName path_row = sparsity_pattern_name_rows_;
+    io_col_idxs_ptr_ = new GKOIDXIOPtr(IOobject(path_col, db), col_idx);
+    io_row_idxs_ptr_ = new GKOIDXIOPtr(IOobject(path_row, db), row_idx);
+};
+
+
+defineTemplateTypeNameWithName(GKOIDXIOPtr, "IDXIOPtr");
+defineTemplateTypeNameWithName(GKOCOOIOPtr, "COOIOPtr");
+defineTemplateTypeNameWithName(GKOVECIOPtr, "VECIOPtr");
+
+}  // namespace Foam