diff --git a/config/namelist.dyn b/config/namelist.dyn index fb7919588..91816a5e4 100644 --- a/config/namelist.dyn +++ b/config/namelist.dyn @@ -4,7 +4,8 @@ visc_gamma1 = 0.1 ! [nodim], for computation of the flow aware viscosity visc_gamma2 = 0.285 ! [s/m], is only used in easy backscatter option visc_easybsreturn= 1.5 -opt_visc = 5 +opt_visc = 5 +check_opt_visc=.true. ! check if optvisc=5 is valid based on ratio resol/rossbyR ! 5=Kinematic (easy) Backscatter ! 6=Biharmonic flow aware (viscosity depends on velocity Laplacian) ! 7=Biharmonic flow aware (viscosity depends on velocity differences) diff --git a/docs/getting_started/getting_started.rst b/docs/getting_started/getting_started.rst index af6b00035..715dc5f8c 100644 --- a/docs/getting_started/getting_started.rst +++ b/docs/getting_started/getting_started.rst @@ -8,7 +8,7 @@ This chapter describes several ways of getting started with FESOM2. First we sho TL;DR version for supported HPC systems ======================================= -Supported systems are: generic ``ubuntu``, ``ollie`` at AWI, ``mistral`` at DKRZ, ``JURECA`` at JSC, ``HLRN``, ``Hazel Hen``, ``Marinostrum 4`` at BSC. During configuration the system will be recognised and apropriate environment variables and compiler options should be used. +Supported systems are: generic ``ubuntu``, ``ollie`` at AWI, ``mistral`` at DKRZ, ``JURECA`` at JSC, ``HLRN``, ``Hazel Hen``, ``MareNostrum 4`` at BSC. During configuration the system will be recognised and apropriate environment variables and compiler options should be used. :: git clone https://github.com/FESOM/fesom2.git @@ -47,7 +47,7 @@ Clone the GitHub repository with a git command: git clone https://github.com/FESOM/fesom2.git -The repository contains model code and two additional libraries: `Metis` (domain partitioner) and `Parms` (solver), necessary to run FESOM2. To build FESOM2 executable one have to compile Parms library and the code of the model (`src` folder). In order to build executable that is used for model domain partitioning (distribution of the model mesh between CPUs) one have to compile `Metis` library and also some code located in the src directory (see :ref:`partitioning`). Building of the model executable and the partitioner is usually done automatically with the use of CMake. If you going to build the code not on one of the supported platforms (ollie, DKRZ, HLRN, and HAZELHEN, general Ubuntu), you might need to do some (usually small) modifications described in `Adding new platform for compilation`_ section. +The repository contains model code and two additional libraries: `Metis` (domain partitioner) and `Parms` (solver), necessary to run FESOM2. To build FESOM2 executable one have to compile Parms library and the code of the model (`src` folder). In order to build executable that is used for model domain partitioning (distribution of the model mesh between CPUs) one have to compile `Metis` library and also some code located in the src directory (see :ref:`partitioning`). Building of the model executable and the partitioner is usually done automatically with the use of CMake. If you going to build the code not on one of the supported platforms (ollie, DKRZ, HLRN, HAZELHEN, and BSC, general Ubuntu), you might need to do some (usually small) modifications described in `Adding new platform for compilation`_ section. Change to the `fesom2` folder and execute: @@ -57,7 +57,7 @@ Change to the `fesom2` folder and execute: In the best case scenario, your platform will be recognized and the Parms library and model executable will be built and copied to the bin directory. If something went wrong have a look at Troubleshooting_ section. -If you would like to select platform manually (which is nessesary in the case of Ubuntu, for eample), type: +If you would like to select platform manually (which is necessary in the case of Ubuntu, for example), type: :: @@ -67,7 +67,7 @@ If you would like to select platform manually (which is nessesary in the case of Data and mesh files ------------------- -The FESOM2 repository contains only very small example meshes and data (in the ``test`` directory, see the note below). However, if you want to run realistic simulations, you ether have to have them on your system, or download an archive with sample data. THere is a chance that your system already have some of the necesseary files, you can check it in the ``setups/paths.yml`` file. If not, the easiest way to start is to download example set from `DKRZ cloud`_ (12 Gb) by executing: +The FESOM2 repository contains only very small example meshes and data (in the ``test`` directory, see the note below). However, if you want to run realistic simulations, you ether have to have them on your system, or download an archive with sample data. There is a chance that your system already has some of the necesseary files, you can check it in the ``setups/paths.yml`` file. If not, the easiest way to start is to download example set from `DKRZ cloud`_ (12 Gb) by executing: :: @@ -100,7 +100,7 @@ You have to do several basic things in order to prepare the run. First, create a mkdir results -you might make a link to some other directory located on the part of the system where you have a lot of storage. In the results directory, you have to create ``fesom.clock`` file (NOTE, if you change ``runid`` in ``namelist.config`` to something like ``runid=mygreatrun``, the file will be named ``mygreatrun.clock``). Inside the file you have to put two identical lines: +You might make a link to some other directory located on the part of the system where you have a lot of storage. In the results directory, you have to create ``fesom.clock`` file (NOTE, if you change ``runid`` in ``namelist.config`` to something like ``runid=mygreatrun``, the file will be named ``mygreatrun.clock``). Inside the file you have to put two identical lines: :: diff --git a/docs/index.rst b/docs/index.rst index 8dfe72d41..28af8b1fb 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -6,6 +6,15 @@ FESOM2 documentation ==================== +The Finite volumE Sea Ice-Ocean Model (FESOM2). + +Multi-resolution ocean general circulation model that solves +the equations of motion describing the ocean and sea ice using +finite-volume methods on unstructured computational grids. The +model is developed and supported by researchers at the Alfred +Wegener Institute, Helmholtz Centre for Polar and Marine +Research (AWI), in Bremerhaven, Germany. + Authors ------- diff --git a/src/MOD_DYN.F90 b/src/MOD_DYN.F90 index bcd8aa49f..1345527c3 100644 --- a/src/MOD_DYN.F90 +++ b/src/MOD_DYN.F90 @@ -89,6 +89,7 @@ MODULE MOD_DYN ! 6=Biharmonic flow aware (viscosity depends on velocity Laplacian) ! 7=Biharmonic flow aware (viscosity depends on velocity differences) ! 8=Dynamic Backscatter + logical :: check_opt_visc= .true. integer :: opt_visc = 5 ! gamma0 [m/s], backgroung viscosity= gamma0*len, it should be as small diff --git a/src/MOD_PARTIT.F90 b/src/MOD_PARTIT.F90 index 5c8a598af..fb4a88542 100644 --- a/src/MOD_PARTIT.F90 +++ b/src/MOD_PARTIT.F90 @@ -47,10 +47,10 @@ module MOD_PARTIT integer :: eDim_nod2D integer, allocatable, dimension(:) :: myList_nod2D - integer :: myDim_elem2D + integer :: myDim_elem2D, myDim_elem2D_shrinked integer :: eDim_elem2D integer :: eXDim_elem2D - integer, allocatable, dimension(:) :: myList_elem2D + integer, allocatable, dimension(:) :: myList_elem2D, myInd_elem2D_shrinked integer :: myDim_edge2D integer :: eDim_edge2D @@ -75,6 +75,7 @@ module MOD_PARTIT integer, allocatable :: s_mpitype_nod3D(:,:,:), r_mpitype_nod3D(:,:,:) integer :: MPIERR + !!! remPtr_* are constructed during the runtime and shall not be dumped!!! integer, allocatable :: remPtr_nod2D(:), remList_nod2D(:) integer, allocatable :: remPtr_elem2D(:), remList_elem2D(:) diff --git a/src/cpl_driver.F90 b/src/cpl_driver.F90 index 1a6cf2642..9e2829ae2 100755 --- a/src/cpl_driver.F90 +++ b/src/cpl_driver.F90 @@ -2,13 +2,16 @@ module cpl_driver !====================================================================== ! - ! for coupling between the FESOM ocean ECHAM6 atmosphere using OASIS3-MCT + ! for coupling between the FESOM and an AOGCM using OASIS3-MCT ! !===================================================================== ! History : ! 09-09 (R. Redler, Germany) Original code ! 09-09 (K.Fieg, AWI Germany) Adjustment for FESOM ! 07-12 (D.Barbi, AWI Germany) Switch to ECHAM6.1 and OASIS3-MCT + ! 01-19 (J.Streffing, AWI Germany) Added OpenIFS coupling + ! 03-23 (J.Streffing, AWI Germany) Added corner point computation + ! for 1st order conserv remapping !---------------------------------------------------------------------- ! Modules used ! @@ -205,8 +208,8 @@ subroutine cpl_oasis3mct_define_unstr(partit, mesh) integer :: il_flag logical :: new_points - integer :: i, j, k ! local loop indicees - integer :: l,m,n ! local loop indicees + integer :: i, j, k ! local loop indicees + integer :: l,m,n, done ! local loop indicees character(len=32) :: point_name ! name of the grid points @@ -217,10 +220,12 @@ subroutine cpl_oasis3mct_define_unstr(partit, mesh) integer :: my_displacement integer :: my_max_elem(partit%npes) integer :: my_max_edge(partit%npes) - integer :: all_max_elem, all_max_edge + integer :: all_max_elem, all_max_edge, n_neg, n_pos integer :: el(2), enodes(2), edge - integer,allocatable :: unstr_mask(:,:) + integer,allocatable :: unstr_mask(:,:), coastal_edge_list(:,:) + real(kind=WP) :: max_x ! max longitude on corners of control volume + real(kind=WP) :: min_x ! min longitude on corners of control volume real(kind=WP) :: temp ! temp storage for corner sorting real(kind=WP) :: this_x_coord ! longitude coordinates real(kind=WP) :: this_y_coord ! latitude coordinates @@ -229,6 +234,12 @@ subroutine cpl_oasis3mct_define_unstr(partit, mesh) ! ! Corner data structure for a OASIS3-MCT Reglonlatvrt grid ! + real(kind=WP), allocatable :: pos_x(:) ! longitude to the right of dateline + real(kind=WP), allocatable :: pos_y(:) ! latitude to the right of dateline + real(kind=WP), allocatable :: neg_x(:) ! longitude to the left of dateline + real(kind=WP), allocatable :: neg_y(:) ! latitude to the left of dateline + real(kind=WP), allocatable :: temp_x_coord(:) ! longitude coordinates + real(kind=WP), allocatable :: temp_y_coord(:) ! longitude coordinates real(kind=WP), allocatable :: my_x_coords(:) ! longitude coordinates real(kind=WP), allocatable :: my_y_coords(:) ! latitude coordinates real(kind=WP), allocatable :: angle(:,:) ! array for holding corner angle for sorting @@ -299,27 +310,36 @@ subroutine cpl_oasis3mct_define_unstr(partit, mesh) CALL MPI_BARRIER(MPI_COMM_FESOM, ierror) my_max_elem=0 - all_max_elem=0 - my_max_elem(mype+1)=maxval(nod_in_elem2D_num(1:myDim_nod2D)) - call MPI_AllREDUCE( my_max_elem, all_max_elem, & - npes, MPI_INTEGER,MPI_SUM, & + my_max_elem = maxval(nod_in_elem2D_num(1:myDim_nod2D)) + all_max_elem = 0 + call MPI_Allreduce(my_max_elem, all_max_elem, & + 1, MPI_INTEGER, MPI_MAX, & MPI_COMM_FESOM, MPIerr) my_max_edge=0 + my_max_edge=maxval(nn_num) all_max_edge=0 - my_max_edge(mype+1)=maxval(nn_num) call MPI_AllREDUCE( my_max_edge, all_max_edge, & - npes, MPI_INTEGER,MPI_SUM, & + 1, MPI_INTEGER,MPI_MAX, & MPI_COMM_FESOM, MPIerr) - ig_paral(1) = 1 ! Apple Partition - ig_paral(2) = my_displacement ! Global Offset - ig_paral(3) = my_number_of_points ! Local Extent + CALL MPI_BARRIER(MPI_COMM_FESOM, ierror) if (mype .eq. 0) then print *, 'Max elements per node:', all_max_elem, 'Max edges per node:', all_max_edge print *, 'FESOM before def partition' endif + + ig_paral(1) = 1 ! Apple Partition + ig_paral(2) = my_displacement ! Global Offset + ig_paral(3) = my_number_of_points ! Local Extent + + ! For MPI_GATHERV we need the location of the local segment in the global vector + displs_from_all_pes(1) = 0 + do i = 2, npes + displs_from_all_pes(i) = SUM(counts_from_all_pes(1:(i-1))) + enddo + CALL oasis_def_partition( part_id(1), ig_paral, ierror ) if (mype .eq. 0) then print *, 'FESOM after def partition' @@ -329,19 +349,6 @@ subroutine cpl_oasis3mct_define_unstr(partit, mesh) call oasis_abort(comp_id, 'cpl_oasis3mct_define_unstr', 'def_partition failed') endif - - - ALLOCATE(my_x_coords(my_number_of_points)) - ALLOCATE(my_y_coords(my_number_of_points)) - - ! Node center coordinates - do i = 1, my_number_of_points - this_x_coord = coord_nod2D(1, i) - this_y_coord = coord_nod2D(2, i) - call r2g(my_x_coords(i), my_y_coords(i), this_x_coord, this_y_coord) - end do - - ALLOCATE(coastal_nodes(number_of_all_points)) ALLOCATE(angle(my_number_of_points,all_max_elem+all_max_edge)) ALLOCATE(my_x_corners(my_number_of_points,all_max_elem+all_max_edge)) @@ -351,11 +358,12 @@ subroutine cpl_oasis3mct_define_unstr(partit, mesh) ! We need to know for every node if any of it's edges are coastal, because ! in case they are the center point will be a corner of the nodal area coastal_nodes=.False. + allocate (coastal_edge_list(my_number_of_points*2,my_number_of_points*2)) do edge=1, myDim_edge2D ! local indice of nodes that span up edge enodes=edges(:,edge) ! local index of element that contribute to edge - el=edge_tri(:,edge) + el=edge_tri(:,edge) if(el(2)>0) then ! Inner edge continue @@ -363,10 +371,13 @@ subroutine cpl_oasis3mct_define_unstr(partit, mesh) ! Boundary/coastal edge coastal_nodes(enodes(1))=.True. coastal_nodes(enodes(2))=.True. + coastal_edge_list(enodes(1),enodes(2))=edge + coastal_edge_list(enodes(2),enodes(1))=edge end if end do - - ! For every node, loop over neighbours, find mean of node center and neighbour node center. + + + ! For every node, loop over neighbours, calculate edge center as mean of node center and neighbour node center. coord_e_edge_center=0 do i = 1, my_number_of_points ! if we are on coastal node, include node center n=1 as corner @@ -384,69 +395,169 @@ subroutine cpl_oasis3mct_define_unstr(partit, mesh) end if end do - ! We can have a variable number of corner points. - ! Luckly oasis can deal with that by just repeating the last one. - ! Note, we are only allowed to repeat one coordinate and - ! the last one is not an element center, but an edge center + ALLOCATE(my_x_coords(my_number_of_points)) + ALLOCATE(my_y_coords(my_number_of_points)) + + ! Obtain center coordinates as node center on open ocean and as mean of corners at coastline do i = 1, my_number_of_points - do j = 1, all_max_elem - if (nod_in_elem2D_num(i) < j) then - my_x_corners(i,j) = coord_e_edge_center(1,i,2) - my_y_corners(i,j) = coord_e_edge_center(2,i,2) - else - my_x_corners(i,j) = x_corners(i,j)*rad ! atan2 takes radian and elem corners come in grad - my_y_corners(i,j) = y_corners(i,j)*rad - end if - ! To allow for sorting the corners counterclockwise later we calculate the - ! arctangent to the center - ! Default case is that center and corner coords have same sign - if (my_x_coords(i) <=0 .and. my_x_corners(i,j) <=0 .or. my_x_coords(i) >0 .and. my_x_corners(i,j) >0) then - angle(i,j) = atan2(my_x_corners(i,j) - my_x_coords(i), my_y_corners(i,j) - my_y_coords(i)) - ! If they have different sign we are near the dateline and need to bring the corner onto - ! the same hemisphere as the center (only for angle calc, the coord for oasis remains as before) - else - if (my_x_coords(i) <=0) then - angle(i,j) = atan2(my_x_corners(i,j) - 2*3.141592653589793_WP - my_x_coords(i), my_y_corners(i,j) - my_y_coords(i)) + ! Center coord as mean of corner coordiantes along coastline + if (coastal_nodes(i)==.True.) then + ! So we define temp_corner coordiantes + allocate(temp_x_coord(nod_in_elem2D_num(i)+nn_num(i))) + allocate(temp_y_coord(nod_in_elem2D_num(i)+nn_num(i))) + temp_x_coord=0 + temp_y_coord=0 + do j = 1, nod_in_elem2D_num(i) + temp_x_coord(j) = x_corners(i,j)*rad + temp_y_coord(j) = y_corners(i,j)*rad + end do + ! Loop over edges + do j = 1, nn_num(i) + ! We skip coastal edge center points for the new center point calculation + ! such that 1 element islands have the node center at the right angle + ! We only do so if n elements is > 2, to avoid having only 3 corners + if ((j>1) .and. (nod_in_elem2D_num(i) > 2)) then + edge = coastal_edge_list(i,nn_pos(j,i)) + ! if edge is coastal, we leave it out of the mean equation, replaced by the node center + if (edge>0) then + this_x_coord = coord_nod2D(1, i) + this_y_coord = coord_nod2D(2, i) + ! unrotate grid + call r2g(my_x_coords(i), my_y_coords(i), this_x_coord, this_y_coord) + temp_x_coord(j+nod_in_elem2D_num(i))=my_x_coords(i) + temp_y_coord(j+nod_in_elem2D_num(i))=my_y_coords(i) + ! case for only two elements, we need the real edge centers to ensure center coord + ! is inside polygon + else + temp_x_coord(j+nod_in_elem2D_num(i)) = coord_e_edge_center(1,i,j) + temp_y_coord(j+nod_in_elem2D_num(i)) = coord_e_edge_center(2,i,j) + end if + ! Open ocean case, we just use the corner coords else - angle(i,j) = atan2(my_x_corners(i,j) + 2*3.141592653589793_WP - my_x_coords(i), my_y_corners(i,j) - my_y_coords(i)) + temp_x_coord(j+nod_in_elem2D_num(i)) = coord_e_edge_center(1,i,j) + temp_y_coord(j+nod_in_elem2D_num(i)) = coord_e_edge_center(2,i,j) end if - end if - end do - ! We repeat the procedure with edge center coordinates, and calculate their angles - ! No need to worry about the order here, as we will sort the angles in the next step - do j = 1, all_max_edge - if (nn_num(i) < j) then - my_x_corners(i,j+all_max_elem) = coord_e_edge_center(1,i,2) - my_y_corners(i,j+all_max_elem) = coord_e_edge_center(2,i,2) - else - ! If we are on open ocean, we don't have the last point, as we wrote into n-1 above - if ((coastal_nodes(i)==.False.) .and. (j==nn_num(i))) then - my_x_corners(i,j+all_max_elem) = coord_e_edge_center(1,i,2) - my_y_corners(i,j+all_max_elem) = coord_e_edge_center(2,i,2) - ! Default case + end do + min_x = minval(temp_x_coord) + max_x = maxval(temp_x_coord) + ! if we are at dateline (fesom cell larger than pi) + if (max_x-min_x > pi) then + + ! set up separate data structures for the two hemispheres + n_pos=count(temp_x_coord>=0) + n_neg=count(temp_x_coord<0) + allocate(pos_x(n_pos)) + allocate(pos_y(n_pos)) + allocate(neg_x(n_neg)) + allocate(neg_y(n_neg)) + pos_x = 0 + pos_y = 0 + neg_x = 0 + neg_x = 0 + n=1 + do j = 1, size(temp_x_coord) + ! build separate corner vectors for the hemispheres + if (temp_x_coord(j) >= 0) then + pos_x(n) = temp_x_coord(j) + pos_y(n) = temp_y_coord(j) + n=n+1 + end if + end do + n=1 + do j = 1, size(temp_x_coord) + if (temp_x_coord(j) < 0) then + neg_x(n) = temp_x_coord(j) + neg_y(n) = temp_y_coord(j) + n=n+1 + end if + end do + ! if sum on right side of dateline are further from the dateline we shift the negative sum over to the right + if (-sum(pos_x)+pi*n_pos >= sum(neg_x)+pi*n_neg) then + this_x_coord = (sum(pos_x) + sum(neg_x) + 2*pi*n_neg) / (n_pos + n_neg) + this_y_coord = (sum(pos_y) + sum(neg_y)) / (n_pos + n_neg) + ! else we shift the positive sum over to the left side else - my_x_corners(i,j+all_max_elem) = coord_e_edge_center(1,i,j) - my_y_corners(i,j+all_max_elem) = coord_e_edge_center(2,i,j) + this_x_coord = (sum(pos_x) - 2*pi*n_pos + sum(neg_x)) / (n_pos + n_neg) + this_y_coord = (sum(pos_y) + sum(neg_y)) / (n_pos + n_neg) end if + deallocate(pos_x,pos_y,neg_x,neg_y) + ! max_x-min_x > pi -> we are not at dateline, just a normal mean is enough + else + this_x_coord = sum(temp_x_coord)/(size(temp_x_coord)) + this_y_coord = sum(temp_y_coord)/(size(temp_y_coord)) end if - ! Default case is that center and corner coords have same sign - if (my_x_coords(i) <=0 .and. my_x_corners(i,j+all_max_elem) <=0 .or. my_x_coords(i) >0 .and. my_x_corners(i,j+all_max_elem) >0) then - angle(i,j+all_max_elem) = atan2(my_x_corners(i,j+all_max_elem) - my_x_coords(i), my_y_corners(i,j+all_max_elem) - my_y_coords(i)) + my_x_coords(i)=this_x_coord + my_y_coords(i)=this_y_coord + deallocate(temp_x_coord, temp_y_coord) + ! coastal_nodes(i)==.True. -> Node center on open ocean, we can use node center + else + this_x_coord = coord_nod2D(1, i) + this_y_coord = coord_nod2D(2, i) + ! unrotate grid + call r2g(my_x_coords(i), my_y_coords(i), this_x_coord, this_y_coord) + end if + end do + + ! Add the different corner types to single array in preparation for angle calculation + do i = 1, my_number_of_points + ! First for element center based corners + do j = 1, nod_in_elem2D_num(i) + my_x_corners(i,j) = x_corners(i,j)*rad ! atan2 takes radian and elem corners come in grad + my_y_corners(i,j) = y_corners(i,j)*rad + end do + ! Then we repeat for edge center coordinate + ! The the coast j=1 is the node center + if (coastal_nodes(i)==.True.) then + do j = 1, nn_num(i) + my_x_corners(i,j+nod_in_elem2D_num(i)) = coord_e_edge_center(1,i,j) + my_y_corners(i,j+nod_in_elem2D_num(i)) = coord_e_edge_center(2,i,j) + end do + ! On open ocean we dont use the node center as corner, and thus have one less corner + else + do j = 1, nn_num(i)-1 + my_x_corners(i,j+nod_in_elem2D_num(i)) = coord_e_edge_center(1,i,j) + my_y_corners(i,j+nod_in_elem2D_num(i)) = coord_e_edge_center(2,i,j) + end do + end if + end do + + ! calculate angle between corners and center + do i = 1, my_number_of_points + if (coastal_nodes(i)==.True.) then + n=0 + else + n=1 + end if + do j = 1, nod_in_elem2D_num(i)+nn_num(i)-n ! If they have different sign we are near the dateline and need to bring the corner onto ! the same hemisphere as the center (only for angle calc, the coord for oasis remains as before) + ! Default: same sign -> normal atan2 + if (my_x_coords(i) <=0 .and. my_x_corners(i,j) <=0 .or. my_x_coords(i) >0 .and. my_x_corners(i,j) >0) then + angle(i,j) = atan2(my_x_corners(i,j) - my_x_coords(i), my_y_corners(i,j) - my_y_coords(i)) else - if (my_x_coords(i) <=0) then - angle(i,j+all_max_elem) = atan2(my_x_corners(i,j+all_max_elem) - 2*3.141592653589793_WP - my_x_coords(i), my_y_corners(i,j+all_max_elem) - my_y_coords(i)) + ! at dateline center is on the right side + if (my_x_coords(i) >=pi/2) then + angle(i,j) = atan2(my_x_corners(i,j) + 2*pi - my_x_coords(i), my_y_corners(i,j) - my_y_coords(i)) + ! at dateline center is on the left side + else if (my_x_coords(i) <=-pi/2) then + angle(i,j) = atan2(my_x_corners(i,j) - 2*pi - my_x_coords(i), my_y_corners(i,j) - my_y_coords(i)) + ! at prime meridan -> also default else - angle(i,j+all_max_elem) = atan2(my_x_corners(i,j+all_max_elem) + 2*3.141592653589793_WP - my_x_coords(i), my_y_corners(i,j+all_max_elem) - my_y_coords(i)) + angle(i,j) = atan2(my_x_corners(i,j) - my_x_coords(i), my_y_corners(i,j) - my_y_coords(i)) end if end if end do + end do - - ! Oasis requires corners sorted counterclockwise, so we sort by angle - do l = 1, all_max_elem+all_max_edge-1 - do m = l+1, all_max_elem+all_max_edge + ! Oasis requires corners sorted counterclockwise, so we sort by angle + do i = 1, my_number_of_points + if (coastal_nodes(i)==.True.) then + n=0 + else + n=1 + end if + do l = 1, nod_in_elem2D_num(i)+nn_num(i)-1-n + do m = l+1, nod_in_elem2D_num(i)+nn_num(i)-n if (angle(i,l) < angle(i,m)) then ! Swap angle temp = angle(i,m) @@ -463,7 +574,33 @@ subroutine cpl_oasis3mct_define_unstr(partit, mesh) end if end do end do - end do + end do + + ! We can have a variable number of corner points. + ! Luckly oasis can deal with that by just repeating the last one. + ! Note, we are only allowed to repeat one coordinate and + ! the last one is not an element center, but an edge center + do i = 1, my_number_of_points + do j = 1, all_max_elem+all_max_edge + if (coastal_nodes(i)==.True.) then + if (j < nod_in_elem2D_num(i)+nn_num(i)) then + my_y_corners(i,j)=my_y_corners(i,j) + my_x_corners(i,j)=my_x_corners(i,j) + else + my_y_corners(i,j)=my_y_corners(i,nod_in_elem2D_num(i)+nn_num(i)) + my_x_corners(i,j)=my_x_corners(i,nod_in_elem2D_num(i)+nn_num(i)) + end if + else + if (j < nod_in_elem2D_num(i)+nn_num(i)-1) then + my_y_corners(i,j)=my_y_corners(i,j) + my_x_corners(i,j)=my_x_corners(i,j) + else + my_y_corners(i,j)=my_y_corners(i,nod_in_elem2D_num(i)+nn_num(i)-1) + my_x_corners(i,j)=my_x_corners(i,nod_in_elem2D_num(i)+nn_num(i)-1) + end if + end if + end do + end do ! Oasis takes grad angles my_x_coords=my_x_coords/rad @@ -486,11 +623,6 @@ subroutine cpl_oasis3mct_define_unstr(partit, mesh) ALLOCATE(all_area(1, 1)) endif - ! For MPI_GATHERV we need the location of the local segment in the global vector - displs_from_all_pes(1) = 0 - do i = 2, npes - displs_from_all_pes(i) = SUM(counts_from_all_pes(1:(i-1))) - enddo if (mype .eq. 0) then print *, 'FESOM before 1st GatherV', displs_from_all_pes(npes), counts_from_all_pes(npes), number_of_all_points diff --git a/src/fesom_module.F90 b/src/fesom_module.F90 index 849c6d792..fea998944 100755 --- a/src/fesom_module.F90 +++ b/src/fesom_module.F90 @@ -44,6 +44,7 @@ module fesom_main_storage_module integer :: n, from_nstep, offset, row, i, provided integer :: which_readr ! read which restart files (0=netcdf, 1=core dump,2=dtype) + integer :: total_nsteps integer, pointer :: mype, npes, MPIerr, MPI_COMM_FESOM real(kind=WP) :: t0, t1, t2, t3, t4, t5, t6, t7, t8, t0_ice, t1_ice, t0_frc, t1_frc real(kind=WP) :: rtime_fullice, rtime_write_restart, rtime_write_means, rtime_compute_diag, rtime_read_forcing @@ -88,6 +89,9 @@ module fesom_module subroutine fesom_init(fesom_total_nsteps) use fesom_main_storage_module +#if defined(__MULTIO) + use iom +#endif integer, intent(out) :: fesom_total_nsteps ! EO parameters logical mpi_is_initialized @@ -141,6 +145,7 @@ subroutine fesom_init(fesom_total_nsteps) call setup_model(f%partit) ! Read Namelists, always before clock_init call clock_init(f%partit) ! read the clock file call get_run_steps(fesom_total_nsteps, f%partit) + f%total_nsteps=fesom_total_nsteps if (flag_debug .and. f%mype==0) print *, achar(27)//'[34m'//' --> call mesh_setup'//achar(27)//'[0m' call mesh_setup(f%partit, f%mesh) @@ -241,6 +246,10 @@ subroutine fesom_init(fesom_total_nsteps) write(*,*) '============================================' endif +#if defined(__MULTIO) + call iom_send_fesom_domains(f%partit, f%mesh) +#endif + ! f%dump_dir='DUMP/' ! INQUIRE(file=trim(f%dump_dir), EXIST=f%L_EXISTS) ! if (.not. f%L_EXISTS) call system('mkdir '//trim(f%dump_dir)) @@ -381,7 +390,7 @@ subroutine fesom_runloop(current_nsteps) call output (n, f%ice, f%dynamics, f%tracers, f%partit, f%mesh) f%t5 = MPI_Wtime() - call restart(n, nstart, ntotal, .false., f%which_readr, f%ice, f%dynamics, f%tracers, f%partit, f%mesh) + call restart(n, nstart, f%total_nsteps, .false., f%which_readr, f%ice, f%dynamics, f%tracers, f%partit, f%mesh) f%t6 = MPI_Wtime() f%rtime_fullice = f%rtime_fullice + f%t2 - f%t1 diff --git a/src/fvom_init.F90 b/src/fvom_init.F90 index 77d481f18..eac201990 100755 --- a/src/fvom_init.F90 +++ b/src/fvom_init.F90 @@ -974,12 +974,14 @@ subroutine find_levels_cavity(mesh) use g_config implicit none integer :: nodes(3), elems(3) - integer :: elem, node, nz, j, idx + integer :: elem, node, nz, j + integer :: min_nlvl, max_ulvl, idx, idx2, val, val2 integer :: count_neighb_open, nneighb, cavity_maxlev, count_isoelem - integer :: exit_flag1, count_iter, max_iter=1000, exit_flag2, count_iter2, max_iter2=10 + integer :: exit_flag1, count_iter, max_iter=1000, exit_flag2, count_iter2, max_iter2=25 real(kind=WP) :: dmean character(MAX_PATH) :: file_name - integer, allocatable, dimension(:) :: numelemtonode, idxelemtonode + integer, allocatable, dimension(:) :: aux_arr, aux_idx + integer, allocatable, dimension(:) :: numelemtonode logical, allocatable, dimension(:) :: elemreducelvl, elemfixlvl type(t_mesh), intent(inout), target :: mesh #include "associate_mesh_ini.h" @@ -1034,16 +1036,17 @@ subroutine find_levels_cavity(mesh) ! write out elemental cavity-ocean boundary level file_name=trim(meshpath)//'cavity_elvls_raw.out' open(20, file=file_name) - do elem=1,elem2D - write(20,*) ulevels(elem) - enddo - close(20) + do elem=1,elem2D + write(20,*) ulevels(elem) + enddo + close(20) + !___________________________________________________________________________ ! Eliminate cells that have two cavity boundary faces --> should not be ! possible in FESOM2.0 ! loop over all cavity levels allocate(elemreducelvl(elem2d),elemfixlvl(elem2d)) - allocate(numelemtonode(nl),idxelemtonode(nl)) + allocate(numelemtonode(nl)) !___________________________________________________________________________ ! outer iteration loop @@ -1074,14 +1077,21 @@ subroutine find_levels_cavity(mesh) ! .___________________________.~~~~~~~~~~~~~~~~~~~~~~~~~~ ! |###|###|###|###|###|###|###| ! |# CAVITY |###| . |###|###| OCEAN - ! |###|###|###| /|\|###| + ! |###|###|###|###|/|\|###| ! |###|###| | ! |###| +-- Not good can lead to isolated cells ! + + !___________________________________________________________ + ! (1st) Ask the Question: is nz at element elem an here an + ! valid layer in the ocean if ( nz >= ulevels(elem) .and. nz0) then ! if its a valid boundary triangle, 0=missing value @@ -1095,13 +1105,14 @@ subroutine find_levels_cavity(mesh) end do ! --> do i = 1, nneighb !_______________________________________________________ - ! check how many open faces to neighboring triangles the cell + ! (3rd) check how many open faces to neighboring triangles the cell ! has, if there are less than 2 its isolated (a cell should ! have at least 2 valid neighbours) ! --> in this case shift cavity-ocean interface one level down - if (count_neighb_open<2) then + if (count_neighb_open<=1) then count_isoelem = count_isoelem+1 - ! if cell is isolated convert it to a deeper ocean levels + + ! (4th.1 ) if cell elem is isolated convert it to a deeper ocean level ! except when this levels would remain less than 3 valid ! bottom levels --> in case make the levels of all sorounding ! triangles shallower @@ -1110,31 +1121,45 @@ subroutine find_levels_cavity(mesh) (elemfixlvl( elem) .eqv. .False.) & ) then ulevels(elem)=nz+1 - else - ! --> can not increase depth anymore to eleminate isolated - ! cell, otherwise lessthan 3 valid layers - ! --> therefor reduce depth of ONE!!! of the neighbouring - ! triangles. Choose trinagle whos depth is already closest - ! to nz - idx = minloc(ulevels(elems)-nz, 1, MASK=( (elems>0) .and. ((ulevels(elems)-nz)>0) ) ) - ulevels(elems(idx)) = nz-1 + + ! (4th.2) can not increase depth anymore to eleminate + ! isolated cell, otherwise lessthan 3 valid layers + ! therefor reduce depth of ONE!!! of the neighbouring + ! triangles. Choose triangle whos depth is already closest + ! to nz + else + !PS replace this with do j=1,3... because of + !PS indice -999 conflict in elems, ulevels(-999) + !PS not possible + !PS idx = minloc(ulevels(elems)-nz, 1, MASK=( (elems>0) .and. ((ulevels(elems)-nz)>0) ) ) + val=nl + do j = 1, 3 + if (elems(j)>0) then ! if its a valid boundary triangle, 0=missing value + if (ulevels(elems(j))-nz>0 .and. ulevels(elems(j))-nz do i = 1, nneighb + + ulevels( elems(idx)) = nz elemreducelvl(elems(idx)) = .True. end if - !force recheck for all current ocean cells + ! force recheck for all current ocean cells exit_flag1=0 end if ! --> if (count_neighb_open<2) then end if ! --> if ( nz >= ulevels(elem) .and. nz do elem=1,elem2D + end do ! --> do elem=1,elem2D end do ! --> do while((exit_flag==0).and.(count_iter<1000)) write(*,"(A, I5, A, i5, A, I3)") ' -[iter ]->: ulevel, iter/maxiter=',count_iter,'/',max_iter,', nz=',nz end do ! --> do nz=1,cavity_maxlev !_______________________________________________________________________ - ! vertical vertice level index of cavity_ocean boundary + ! compute vertical vertice level index of cavity_ocean boundary write(*,"(A)" ) ' -[compu]->: ulevels_nod2D ' ulevels_nod2D = nl do elem=1,elem2D @@ -1148,33 +1173,26 @@ subroutine find_levels_cavity(mesh) end do ! --> do elem=1,elem2D !_______________________________________________________________________ - ! check ulevels if ulevels=nlevels(elem)) then write(*,*) ' -[check]->: elem cavity depth deeper or equal bottom depth, elem=',elem exit_flag2 = 0 - end if if (nlevels(elem)-ulevels(elem)<3) then write(*,*) ' -[check]->: less than three valid elem ocean layers, elem=',elem exit_flag2 = 0 - end if end do ! --> do elem=1,elem2D - !_______________________________________________________________________ - ! check ulevels_nod2d if ulevels_nod2d=nlevels_nod2D(node)) then write(*,*) ' -[check]->: vertice cavity depth deeper or equal bottom depth, node=', node exit_flag2 = 0 end if - !___________________________________________________________________ if (nlevels_nod2D(node)-ulevels_nod2D(node)<3) then write(*,*) ' -[check]->: less than three valid vertice ocean layers, node=', node exit_flag2 = 0 @@ -1189,62 +1207,140 @@ subroutine find_levels_cavity(mesh) end do ! --> do elem=1,elem2D !_______________________________________________________________________ - ! compute how many triangle elements contribute to every vertice in every layer + ! compute how many triangle elements contribute to every vertice in every + ! layer + ! + ! --> What can happen is that a node point in the middle of the vertical + ! domain can become isolated due to the cavity constrains. The model + ! would not be able to deal with this kind of situation. So we must + ! prevent it by adapting ulevels! + ! O node + ! _._ + ! _/ | \_ + ! _/ | \_ + ! _/ | \_ + ! elem(1) elem(2) elem(3)... <--elem=nod_in_elem2D(j,node) + ! ._______. ulevel(elem2)=30 + ! |_______| + ! |_______| + ! |_______| + ! |_______| + ! |_______| nlevel(elem2)=38 + ! + ! In this possible gap node points + ! would have no neighboring elements + ! + ! ulevel(elem1)=42 ._______. ._______. ulevel(elem3)=42 + ! |_______| |_______| + ! |_______| |_______| + ! |_______| |_______| + ! |_______| |_______| + ! nlevel(elem1)=46 |_______| |_______| + ! |_______| nlevel(elem3)=48 + ! + ! --> Problem here is we want to keep nlevels fixed so what we can do is + ! to set ulevels(elem1) and ulevels(elem3) towards nlevel(elem2) count_iter=0 do node=1, nod2D !___________________________________________________________________ - numelemtonode=0 - idxelemtonode=0 + ! check if there is a possible gap as described above that would + ! allow for node points without neighbors + min_nlvl = nl + max_ulvl = 1 + do j=1, nod_in_elem2D_num(node) + elem=nod_in_elem2D(j, node) + min_nlvl = min(min_nlvl, nlevels(elem)) + max_ulvl = max(max_ulvl, ulevels(elem)) + end do - !___________________________________________________________________ - ! compute how many triangle elements contribute to vertice in every layer - do j=1,nod_in_elem2D_num(node) - elem=nod_in_elem2D(j,node) - do nz=ulevels(elem),nlevels(elem)-1 - numelemtonode(nz) = numelemtonode(nz) + 1 - idxelemtonode(nz) = elem - end do - end do - - !___________________________________________________________________ - ! check if every vertice in every layer should be connected to at least - ! two triangle elements ! - do nz = ulevels_nod2D(node), nlevels_nod2D(node)-1 + ! found a potential gap + if (min_nlvl < max_ulvl) then !_______________________________________________________________ - ! nodes has zero neighbouring triangles and is completely isolated - ! need to adapt ulevels by hand --> inflicts another outher - ! iteration loop (exit_flag2=0) - if (numelemtonode(nz)==0) then - exit_flag2 = 0 - count_iter = count_iter+1 - write(*,"( A, I1, A, I7, A, I3)") ' -[check]->: node has only ', numelemtonode(nz) ,' triangle: n=', node, ', nz=',nz - !___________________________________________________________ - ! if node has no neighboring triangle somewhere in the middle - ! of the water column at nz (can happen but seldom) than set - ! all ulevels(elem) of sorounding trinagles whos ulevel is - ! depper than nz, equal to nz and fix that value to forbit it - ! to be changed (elemfixlvl > 0) - do j=1,nod_in_elem2D_num(node) - elem=nod_in_elem2D(j,node) - if (ulevels(elem)>nz) then - ulevels(elem) = nz - elemfixlvl(elem) = .True. - end if + ! compute how many triangle elements contribute to vertice in + ! every layer check if there are layers where a node point has + ! only one or even zero neighboring triangles. + numelemtonode=0 + do j=1, nod_in_elem2D_num(node) + elem=nod_in_elem2D(j, node) + do nz=ulevels(elem), nlevels(elem)-1 + numelemtonode(nz) = numelemtonode(nz) + 1 end do - end if + end do !_______________________________________________________________ - ! nodes has just one neighbouring triangle --> but needs two --> - ! inflicts another outher iteration loop (exit_flag2=0) - if (numelemtonode(nz)==1) then - exit_flag2 = 0 - count_iter = count_iter+1 - write(*,"( A, I1, A, I7, A, I3)") ' -[check]->: node has only ', numelemtonode(nz) ,' triangle: n=', node, ', nz=',nz - end if - - end do ! --> do nz = ulevels_nod2D(node), nlevels_nod2D(node)-1 - + ! check in which depth level is an isolated node + nzloop: do nz = ulevels_nod2D(node), nlevels_nod2D(node)-1 + !___________________________________________________________ + ! nodes has zero neighbouring triangles and is completely + ! isolated need to adapt ulevels --> inflicts another + ! outher iteration loop (exit_flag2=0). It needs at least + ! two neighboring elements so the node is considered as + ! connected. Search the index of the two elements where ulevels>nz + ! but that are closest to nz + if (numelemtonode(nz)==0) then + count_iter = count_iter+1 + write(*,"( A, I1, A, I7, A, I3)") ' -[check]->: node has only ', numelemtonode(nz) ,' neighb. triangle: n=', node, ', nz=',nz + !_______________________________________________________ + allocate(aux_arr(nod_in_elem2D_num(node)), aux_idx(nod_in_elem2D_num(node))) + aux_arr(:) = ulevels(nod_in_elem2D(1:nod_in_elem2D_num(node),node)) + aux_arr(:) = aux_arr(:) - nz + ! fill array with index of element + do j=1, nod_in_elem2D_num(node) + aux_idx(j) = j + end do + ! index of closest elem to nz where ulevel>nz + idx = minloc(aux_arr, 1, MASK=((aux_arr>0)) ) + ! index of second closest elem to nz where ulevel>nz + idx2 = minloc(aux_arr, 1, MASK=((aux_arr>0) .and. (aux_idx/=idx)) ) + deallocate(aux_arr, aux_idx) + ulevels( nod_in_elem2D(idx ,node)) = nz + ulevels( nod_in_elem2D(idx2,node)) = nz + elemfixlvl(nod_in_elem2D(idx ,node)) = .True. + elemfixlvl(nod_in_elem2D(idx2,node)) = .True. + + !_______________________________________________________ + ! inflict another outer iteration loop + exit_flag2 = 0 + + !_______________________________________________________ + ! if the upper most isolated layer is fixed all layers below should be fixed as well + ! --> exit loop do nz = ulevels_nod2D(node), nlevels_nod2D(node)-1 + exit nzloop + + !___________________________________________________________ + ! nodes has one neighbouring triangles it needs at least + ! another neighboring elements so the node is considered as + ! connected + elseif (numelemtonode(nz)==1) then + count_iter = count_iter+1 + write(*,"( A, I1, A, I7, A, I3)") ' -[check]->: node has only ', numelemtonode(nz) ,'neighb. triangle: n=', node, ', nz=',nz + !_______________________________________________________ + allocate(aux_arr(nod_in_elem2D_num(node)), aux_idx(nod_in_elem2D_num(node))) + aux_arr(:) = ulevels(nod_in_elem2D(1:nod_in_elem2D_num(node),node)) + aux_arr(:) = aux_arr(:) - nz + ! fill array with index of element + do j=1, nod_in_elem2D_num(node) + aux_idx(j) = j + end do + ! index of closest elem to nz where ulevel>nz + idx = minloc(aux_arr, 1, MASK=((aux_arr>0)) ) + deallocate(aux_arr, aux_idx) + ulevels( nod_in_elem2D(idx,node)) = nz + elemfixlvl(nod_in_elem2D(idx,node)) = .True. + + !_______________________________________________________ + ! inflict another outer iteration loop + exit_flag2 = 0 + + !_______________________________________________________ + ! if the upper most isolated layer is fixed all layers below should be fixed as well + ! --> exit loop do nz = ulevels_nod2D(node), nlevels_nod2D(node)-1 + exit nzloop + + end if + end do nzloop ! --> do nz = ulevels_nod2D(node), nlevels_nod2D(node)-1 + end if ! --> if (min_nlvl < max_ulvl) then end do ! --> do node=1, nod2D !_______________________________________________________________________ @@ -1257,9 +1353,9 @@ subroutine find_levels_cavity(mesh) end if !_______________________________________________________________________ - end do + end do ! --> do while((exit_flag2==0) .and. (count_iter2: Cavity geometry constrains did converge !!! *\(^o^)/*' end if - !___________________________________________________________________________ ! write out cavity mesh files for vertice and elemental position of ! vertical cavity-ocean boundary ! write out elemental cavity-ocean boundary level file_name=trim(meshpath)//'cavity_elvls.out' open(20, file=file_name) - do elem=1,elem2D - write(20,*) ulevels(elem) - enddo + do elem=1,elem2D + write(20,*) ulevels(elem) + enddo close(20) ! write out vertice cavity-ocean boundary level + yes/no cavity flag file_name=trim(meshpath)//'cavity_nlvls.out' open(20, file=file_name) - do node=1,nod2D - write(20,*) ulevels_nod2D(node) - enddo + do node=1,nod2D + write(20,*) ulevels_nod2D(node) + enddo close(20) - end subroutine find_levels_cavity diff --git a/src/ifs_interface/ifs_interface.F90 b/src/ifs_interface/ifs_interface.F90 index 03aba0b48..847fa60ae 100644 --- a/src/ifs_interface/ifs_interface.F90 +++ b/src/ifs_interface/ifs_interface.F90 @@ -9,6 +9,46 @@ MODULE nemogcmcoup_steps INTEGER :: substeps !per IFS timestep END MODULE nemogcmcoup_steps +#if defined(__MULTIO) +SUBROUTINE nemogcmcoup_init_ioserver( icomm, lnemoioserver, irequired, iprovided, lmpi1) + + ! Initialize the NEMO mppio server + USE mpp_io + + IMPLICIT NONE + INTEGER :: icomm + LOGICAL :: lnemoioserver + INTEGER :: irequired, iprovided + LOGICAL :: lmpi1 + + CALL mpp_io_init(icomm, lnemoioserver, irequired, iprovided, lmpi1) +END SUBROUTINE nemogcmcoup_init_ioserver + + +SUBROUTINE nemogcmcoup_init_ioserver_2( icomm ) + ! Initialize the NEMO mppio server + USE mpp_io + + IMPLICIT NONE + INTEGER :: icomm + + CALL mpp_io_init_2( icomm ) + IF (lioserver) THEN + ! IO server finished, clean-up multio objects + CALL mpp_stop() + ENDIF +END SUBROUTINE nemogcmcoup_init_ioserver_2 + +SUBROUTINE nemogcmcoup_end_ioserver + ! Function is only called for the IO client. + USE mpp_io + + IMPLICIT NONE + + CALL mpp_stop() + END SUBROUTINE nemogcmcoup_end_ioserver +#endif + SUBROUTINE nemogcmcoup_init( mype, icomm, inidate, initime, itini, itend, zstp, & & lwaveonly, iatmunit, lwrite ) @@ -535,50 +575,6 @@ SUBROUTINE nemogcmcoup_lim2_get( mype, npes, icomm, & nfield = nfield + 1 pgvcur(:) = zrecvnf(:,nfield) - ! Pack u(v) surface currents on elements - !zsendnfUV(:,1)=fesom%dynamics%UV(1,1,1:myDim_elem2D) - !zsendnfUV(:,2)=fesom%dynamics%UV(2,1,1:myDim_elem2D) !UV includes eDim, leave those away here - !nfielduv = 2 - ! - !do elem=1, myDim_elem2D - ! - ! ! compute element midpoints - ! elnodes=elem2D_nodes(:,elem) - ! rlon=sum(coord_nod2D(1,elnodes))/3.0_wpIFS - ! rlat=sum(coord_nod2D(2,elnodes))/3.0_wpIFS - ! - ! ! Rotate vectors to geographical coordinates (r2g) - ! CALL vector_r2g(zsendnfUV(elem,1), zsendnfUV(elem,2), rlon, rlat, 0) ! 0-flag for rot. coord - ! - !end do - -#ifdef FESOM_TODO - - ! We need to sort out the non-unique global index before we - ! can couple currents - - ! Interpolate: 'pgucur' and 'pgvcur' on Gaussian grid. - IF (lparintmultatm) THEN - CALL parinter_fld_mult( nfielduv, mype, npes, icomm, UVtogauss, & - & myDim_nod2D, zsendnfUV, & - & nopoints, zrecvnfUV ) - ELSE - DO jf = 1, nfielduv - CALL parinter_fld( mype, npes, icomm, UVtogauss, & - & myDim_nod2D, zsendnfUV(:,jf), & - & nopoints, zrecvnfUV(:,jf) ) - ENDDO - ENDIF - pgucur(:) = zrecvnfUV(:,1) - pgvcur(:) = zrecvnfUV(:,2) - -#else - - !pgucur(:) = 0.0 - !pgvcur(:) = 0.0 - -#endif - END SUBROUTINE nemogcmcoup_lim2_get @@ -822,24 +818,20 @@ SUBROUTINE nemogcmcoup_lim2_update( mype, npes, icomm, & ! Sort out incoming arrays from the IFS and put them on the ocean grid ! TODO - shortwave(:)=0. ! Done, updated below. What to do with shortwave over ice?? - !longwave(:)=0. ! Done. Only used in stand-alone mode. - prec_rain(:)=0. ! Done, updated below. - prec_snow(:)=0. ! Done, updated below. - evap_no_ifrac=0. ! Done, updated below. This is evap over ocean, does this correspond to evap_tot? - sublimation=0. ! Done, updated below. + shortwave(:)=0. + !longwave(:)=0. + prec_rain(:)=0. + prec_snow(:)=0. + evap_no_ifrac=0. + sublimation=0. ! - ice_heat_flux=0. ! Done. This is qns__ice currently. Is this the non-solar heat flux? ! non solar heat fluxes below ! (qns) - oce_heat_flux=0. ! Done. This is qns__oce currently. Is this the non-solar heat flux? + ice_heat_flux=0. + oce_heat_flux=0. ! - !runoff(:)=0. ! not used apparently. What is runoffIN, ocerunoff? - !evaporation(:)=0. - !ice_thermo_cpl.F90: !---- total evaporation (needed in oce_salt_balance.F90) - !ice_thermo_cpl.F90: evaporation = evap_no_ifrac*(1.-a_ice) + sublimation*a_ice - stress_atmice_x=0. ! Done, taux_ice - stress_atmice_y=0. ! Done, tauy_ice - stress_atmoce_x=0. ! Done, taux_oce - stress_atmoce_y=0. ! Done, tauy_oce + stress_atmice_x=0. + stress_atmice_y=0. + stress_atmoce_x=0. + stress_atmoce_y=0. ! =================================================================== ! ! Pack all arrays @@ -1147,4 +1139,4 @@ SUBROUTINE nemogcmcoup_final endif CALL fesom_finalize -END SUBROUTINE nemogcmcoup_final +END SUBROUTINE nemogcmcoup_final \ No newline at end of file diff --git a/src/ifs_interface/ifs_notused.F90 b/src/ifs_interface/ifs_notused.F90 index bc711a8c6..1cedbc82c 100644 --- a/src/ifs_interface/ifs_notused.F90 +++ b/src/ifs_interface/ifs_notused.F90 @@ -3,33 +3,6 @@ ! ! -Original code by Kristian Mogensen, ECMWF. -SUBROUTINE nemogcmcoup_init_ioserver( icomm, lnemoioserver ) - - ! Initialize the NEMO mppio server - - IMPLICIT NONE - INTEGER :: icomm - LOGICAL :: lnemoioserver - - WRITE(*,*)'No mpp_ioserver' - CALL abort - -END SUBROUTINE nemogcmcoup_init_ioserver - - -SUBROUTINE nemogcmcoup_init_ioserver_2( icomm ) - - ! Initialize the NEMO mppio server - - IMPLICIT NONE - INTEGER :: icomm - - WRITE(*,*)'No mpp_ioserver' - CALL abort - -END SUBROUTINE nemogcmcoup_init_ioserver_2 - - SUBROUTINE nemogcmcoup_mlflds_get( mype, npes, icomm, & & nlev, nopoints, pgt3d, pgs3d, pgu3d, pgv3d ) @@ -202,7 +175,7 @@ SUBROUTINE nemogcmcoup_update_add( mype, npes, icomm, & ! Local variables if(fesom%mype==0) then - WRITE(0,*)'nemogcmcoup_update_add should not be called when coupling to fesom. Commented ABORT. Proceeding...' + WRITE(0,*)'In nemogcmcoup_update_add FESOM dummy routine. Proceeding...' !CALL abort endif @@ -331,17 +304,3 @@ SUBROUTINE nemogcmcoup_wam_update_stress( mype, npes, icomm, npoints, & CALL abort END SUBROUTINE nemogcmcoup_wam_update_stress - -SUBROUTINE nemogcmcoup_end_ioserver - - ! Close io servers - - IMPLICIT NONE - INTEGER :: icomm - LOGICAL :: lnemoioserver - - WRITE(*,*)'No mpp_ioserver' - CALL abort - -END SUBROUTINE nemogcmcoup_end_ioserver - diff --git a/src/ifs_interface/iom.F90 b/src/ifs_interface/iom.F90 new file mode 100644 index 000000000..27ec77942 --- /dev/null +++ b/src/ifs_interface/iom.F90 @@ -0,0 +1,412 @@ +!===================================================== +! Input/Output manager : Library to write output files +! +! -Original code for NEMOv40 by ECMWF. +! -Adapted to FESOM2 by Razvan Aguridan, ECMWF, 2023. +!----------------------------------------------------- + +MODULE iom +#if defined(__MULTIO) + USE multio_api + USE, INTRINSIC :: iso_fortran_env, only: real64 + + IMPLICIT NONE + PRIVATE + + TYPE(multio_handle) :: mio_handle + INTEGER(8), PRIVATE :: mio_parent_comm + + PUBLIC iom_initialize, iom_init_server, iom_finalize + PUBLIC iom_send_fesom_domains + PUBLIC iom_field_request, iom_send_fesom_data + + PRIVATE ctl_stop + !!---------------------------------------------------------------------- + !! NEMO/OCE 4.0 , NEMO Consortium (2018) + !! $Id: iom.F90 13297 2020-07-13 08:01:58Z andmirek $ + !! Software governed by the CeCILL license (see ./LICENSE) + !!---------------------------------------------------------------------- + + TYPE iom_field_request + CHARACTER(100) :: name = REPEAT(" ", 100) + CHARACTER(5) :: gridType = REPEAT(" ", 5) + REAL(real64), DIMENSION(:), POINTER :: values => NULL() + INTEGER :: globalSize = 0 + INTEGER :: level = 0 + INTEGER :: step = 0 + END TYPE + +CONTAINS + + SUBROUTINE multio_custom_error_handler(context, err) + USE mpi + + IMPLICIT NONE + INTEGER(8), INTENT(INOUT) :: context ! Use mpi communicator as context + INTEGER, INTENT(IN) :: err + INTEGER :: mpierr + + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop( 'MULTIO ERROR: ', multio_error_string(err)) + IF (context /= MPI_UNDEFINED) THEN + CALL mpi_abort(int(context), MPI_ERR_OTHER, mpierr) + context = MPI_UNDEFINED + ENDIF + ENDIF + END SUBROUTINE + + + SUBROUTINE iom_initialize(client_id, local_comm, return_comm, global_comm ) + USE mpi + + IMPLICIT NONE + CHARACTER(LEN=*), INTENT(IN) :: client_id + INTEGER,INTENT(IN), OPTIONAL :: local_comm + INTEGER,INTENT(OUT), OPTIONAL :: return_comm + INTEGER,INTENT(IN), OPTIONAL :: global_comm + TYPE(multio_configuration) :: conf_ctx + INTEGER :: err + CHARACTER(len=16) :: err_str + + mio_parent_comm = mpi_comm_world + + err = multio_initialise() + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('Initializing multio failed: ', multio_error_string(err)) + END IF + + IF (PRESENT(global_comm)) THEN + mio_parent_comm = global_comm + ENDIF + + ! Prepare context and check errors explicitly until everything is set up - then failure handler is used + BLOCK + CHARACTER(:), allocatable :: config_file + INTEGER :: config_file_length + + CALL get_environment_variable('MULTIO_FESOM_CONFIG_FILE', length=config_file_length) + IF (config_file_length == 0) THEN + call ctl_stop('The fesom plan file is not correctly set!') + err = conf_ctx%new() + ELSE + ALLOCATE(character(len=config_file_length + 1) :: config_file) + + CALL get_environment_variable('MULTIO_FESOM_CONFIG_FILE', config_file) + err = conf_ctx%new(config_file) + + DEALLOCATE(config_file) + ENDIF + END BLOCK + + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('Creating multio configuration context failed: ', multio_error_string(err)) + END IF + + err = conf_ctx%mpi_allow_world_default_comm(.FALSE._1) + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('conf_ctx%mpi_allow_world_default_comm(.FALSE._1) failed: ', multio_error_string(err)) + END IF + + err = conf_ctx%mpi_client_id(client_id) + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('conf_ctx%mpi_client_id(', TRIM(client_id),') failed: ', multio_error_string(err)) + END IF + + err = conf_ctx%mpi_return_client_comm(return_comm) + IF (err /= MULTIO_SUCCESS) THEN + WRITE (err_str, "(I10)") return_comm + CALL ctl_stop('conf_ctx%mpi_return_client_comm(', err_str,') failed: ', multio_error_string(err)) + END IF + + err = conf_ctx%mpi_parent_comm(int(mio_parent_comm)) + IF (err /= MULTIO_SUCCESS) THEN + WRITE (err_str, "(I10)") mio_parent_comm + CALL ctl_stop('conf_ctx%mpi_parent_comm(', err_str,') failed: ', multio_error_string(err)) + END IF + + err = mio_handle%new(conf_ctx) + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('mio_handle%new(conf_ctx) failed: ', multio_error_string(err)) + END IF + + err = conf_ctx%delete() + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('conf_ctx%delete() failed: ', multio_error_string(err)) + END IF + + ! Setting a failure handler that reacts on interface problems or exceptions that are not handled within the interface + err = multio_set_failure_handler(multio_custom_error_handler, mio_parent_comm) + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('setting multio failure handler failed: ', multio_error_string(err)) + END IF + + err = mio_handle%open_connections(); + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('mio_handle%open_connections failed: ', multio_error_string(err)) + END IF + END SUBROUTINE iom_initialize + + SUBROUTINE iom_finalize() + IMPLICIT NONE + INTEGER :: err + + err = mio_handle%close_connections(); + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('mio_handle%close_connections failed: ', multio_error_string(err)) + END IF + + err = mio_handle%delete(); + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('mio_handle%delete failed: ', multio_error_string(err)) + END IF + END SUBROUTINE iom_finalize + + SUBROUTINE iom_init_server(server_comm) + IMPLICIT NONE + INTEGER, INTENT(IN) :: server_comm + type(multio_configuration) :: conf_ctx + INTEGER :: err + CHARACTER(len=16) :: err_str + + mio_parent_comm = server_comm + + err = multio_initialise() + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('Initializing multio failed: ', multio_error_string(err)) + END IF + + ! Prepare context and check errors explicitly until everything is set up - then failure handler is used + + BLOCK + CHARACTER(:), allocatable :: config_file + INTEGER :: config_file_length + + CALL get_environment_variable('MULTIO_FESOM_CONFIG_FILE', length=config_file_length) + IF (config_file_length == 0) THEN + err = conf_ctx%new() + ELSE + ALLOCATE(character(len=config_file_length + 1) :: config_file) + + CALL get_environment_variable('MULTIO_FESOM_CONFIG_FILE', config_file) + err = conf_ctx%new(config_file) + + DEALLOCATE(config_file) + ENDIF + END BLOCK + + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('Creating multio server configuration context failed: ', multio_error_string(err)) + END IF + + err = conf_ctx%mpi_allow_world_default_comm(.FALSE._1) + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('conf_ctx%mpi_allow_world_default_comm(.FALSE._1) failed: ', multio_error_string(err)) + END IF + + err = conf_ctx%mpi_parent_comm(int(mio_parent_comm)) + IF (err /= MULTIO_SUCCESS) THEN + WRITE (err_str, "(I10)") mio_parent_comm + CALL ctl_stop('conf_ctx%mpi_parent_comm(', err_str,') failed: ', multio_error_string(err)) + END IF + + ! Setting a failure handler that reacts on interface problems or exceptions that are not handled within the interface + ! Set handler before invoking blocking start server call + err = multio_set_failure_handler(multio_custom_error_handler, mio_parent_comm) + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('setting multio failure handler failed: ', multio_error_string(err)) + END IF + + ! Blocking call + err = multio_start_server(conf_ctx) + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('multio_start_server(conf_ctx) failed: ', multio_error_string(err)) + END IF + + err = conf_ctx%delete() + IF (err /= MULTIO_SUCCESS) THEN + CALL ctl_stop('conf_ctx%delete() failed: ', multio_error_string(err)) + END IF + END SUBROUTINE iom_init_server + + SUBROUTINE iom_send_fesom_domains(partit, mesh) + USE MOD_MESH + USE MOD_PARTIT + + IMPLICIT NONE + + TYPE(multio_metadata) :: md + INTEGER :: cerr + INTEGER :: elem, elnodes(3), aux + TYPE(t_partit), INTENT(IN), TARGET :: partit + TYPE(t_mesh), intent(in), TARGET :: mesh + INTEGER, DIMENSION(:), POINTER :: temp + +#include "../associate_part_def.h" +#include "../associate_mesh_def.h" +#include "../associate_part_ass.h" +#include "../associate_mesh_ass.h" + + cerr = md%new() + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: ngrid, md%new() failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_string("name", "ngrid") + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: ngrid, md%set_string(name) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_string("category", "fesom-domain-nodemap") + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: ngrid, md%set_string(category) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_string("representation", "unstructured") + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: ngrid, md%set_string(representation) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_int("globalSize", mesh%nod2D) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: ngrid, md%set_int(globalSize) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_bool("toAllServers", .TRUE._1) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: ngrid, md%set_bool(toAllServers) failed: ', multio_error_string(cerr)) + END IF + + temp => partit%myList_nod2D(1:partit%myDim_nod2D) + cerr = mio_handle%write_domain(md, temp - 1) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: ngrid, mio_handle%write_domain() failed: ', multio_error_string(cerr)) + END IF + + cerr = md%delete() + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: ngrid, md%delete() failed: ', multio_error_string(cerr)) + END IF + + !declare grid at elements + cerr = md%new() + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: egrid, md%new() failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_string("name", "egrid") + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: egrid, md%set_string(name) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_string("category", "fesom-domain-elemmap") + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: egrid, md%set_string(category) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_string("representation", "unstructured") + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: egrid, md%set_string(representation) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_int("globalSize", mesh%elem2D) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: egrid, md%set_int(globalSize) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_bool("toAllServers", .TRUE._1) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: egrid, md%set_bool(toAllServers) failed: ', multio_error_string(cerr)) + END IF + + cerr = mio_handle%write_domain(md, partit%myList_elem2D(partit%myInd_elem2D_shrinked) - 1) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: egrid, mio_handle%write_domain() failed: ', multio_error_string(cerr)) + END IF + + cerr = md%delete() + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_domains: egrid, md%delete() failed: ', multio_error_string(cerr)) + END IF + END SUBROUTINE iom_send_fesom_domains + + SUBROUTINE iom_send_fesom_data(data) + IMPLICIT NONE + + TYPE(iom_field_request), INTENT(INOUT) :: data + INTEGER :: cerr + TYPE(multio_metadata) :: md + + cerr = md%new() + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: md%new() failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_string("category", "fesom-grid-output") + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: md%set_string(category) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_int("globalSize", data%globalSize) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: md%set_int(globalSize) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_int("level", data%level) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: md%set_int(level) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_bool("toAllServers", .FALSE._1) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: md%set_bool(toAllServers) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_string("name", trim(data%name)) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: md%set_string(name) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_string("gridSubType", data%gridType) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: md%set_string(gridSubType) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_string("domain", data%gridType) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: md%set_string(domain) failed: ', multio_error_string(cerr)) + END IF + + cerr = md%set_int("step", data%step) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: md%set_int(step) failed: ', multio_error_string(cerr)) + END IF + + cerr = mio_handle%write_field(md, data%values) + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: mio_handle%write_field failed: ', multio_error_string(cerr)) + END IF + + cerr = md%delete() + IF (cerr /= MULTIO_SUCCESS) THEN + CALL ctl_stop('send_fesom_data: md%delete failed: ', multio_error_string(cerr)) + END IF + END SUBROUTINE + + SUBROUTINE ctl_stop(m1, m2, m3, m4) + USE mpi + + IMPLICIT NONE + CHARACTER(len=*), INTENT(in), OPTIONAL :: m1, m2, m3, m4 + INTEGER :: dummy + + IF ( PRESENT(m1) ) WRITE(*,*) m1 + IF ( PRESENT(m2) ) WRITE(*,*) m2 + IF ( PRESENT(m3) ) WRITE(*,*) m3 + IF ( PRESENT(m4) ) WRITE(*,*) m4 + + CALL mpi_abort(mpi_comm_world, 1, dummy) + END SUBROUTINE ctl_stop + + !!====================================================================== +#endif +END MODULE iom diff --git a/src/ifs_interface/mpp_io.F90 b/src/ifs_interface/mpp_io.F90 new file mode 100644 index 000000000..eda8feae7 --- /dev/null +++ b/src/ifs_interface/mpp_io.F90 @@ -0,0 +1,182 @@ +!===================================================== +! Ocean output intialisation. +! +! -Original code for NEMOv40 by Kristian Mogensen, ECMWF. +! -Adapted to FESOM2 by Razvan Aguridan, ECMWF, 2023. +!----------------------------------------------------- + +MODULE mpp_io +#if defined(__MULTIO) + USE iom + IMPLICIT NONE + PRIVATE + + PUBLIC & + & mpp_io_init, & + & mpp_io_init_2, & + & mpp_stop + + INTEGER :: ntask_multio = 0 + INTEGER :: ntask_xios = 0 + LOGICAL, PUBLIC :: lioserver, lmultioserver, lmultiproc + INTEGER :: ntask_notio + INTEGER, SAVE :: mppallrank, mppallsize, mppiorank, mppiosize + INTEGER, SAVE :: mppmultiorank, mppmultiosize + INTEGER, SAVE :: mppcomprank, mppcompsize + INTEGER, SAVE :: pcommworld, pcommworldmultio + + CONTAINS + + SUBROUTINE mpp_io_init( iicomm, lio, irequired, iprovided, lmpi1 ) + + INCLUDE "mpif.h" + INTEGER, INTENT(INOUT) :: iicomm + LOGICAL, INTENT(INOUT) :: lio + INTEGER, INTENT(INOUT) :: irequired, iprovided + LOGICAL, INTENT(IN) :: lmpi1 + + INTEGER :: icode, ierr, icolor + LOGICAL :: mpi_called + CHARACTER(len=128) :: cdlogfile + INTEGER :: ji + NAMELIST/namio/ntask_multio,ntask_xios + + CALL mpi_initialized( mpi_called, icode ) + IF ( icode /= MPI_SUCCESS ) THEN + WRITE(*,*)' mpp_io_init: Error in routine mpi_initialized' + CALL mpi_abort( mpi_comm_world, icode, ierr ) + ENDIF + + IF( mpi_called ) THEN + WRITE(*,*)' mpi_io_init assumes that it is initialising MPI' + CALL mpi_abort( mpi_comm_world, 1, ierr ) + ENDIF + + IF (lmpi1) THEN + CALL mpi_init( icode ) + ELSE +#ifdef MPI1 + WRITE(0,*)'mpp_io_init:' + WRITE(0,*)'MPI1 defined but lmpi1 is false' + CALL abort +#else + CALL mpi_init_thread(irequired,iprovided,icode) +#endif + ENDIF + + IF ( icode /= MPI_SUCCESS ) THEN + WRITE(*,*)' mpp_io_init: Error in routine mpi_init' + CALL mpi_abort( mpi_comm_world, icode, ierr ) + ENDIF + + CALL mpi_comm_rank( mpi_comm_world, mppallrank, ierr ) + CALL mpi_comm_size( mpi_comm_world, mppallsize, ierr ) + + OPEN(10,file='namio.in') + READ(10,namio) + WRITE(*,namio) + CLOSE(10) + + IF ( ntask_xios + ntask_multio == 0 ) THEN + iicomm = mpi_comm_world + lio=.FALSE. + RETURN + ENDIF + + ntask_notio = mppallsize - ntask_xios - ntask_multio + IF ((mppallrank+1)<=ntask_notio) THEN + icolor=1 + lioserver=.FALSE. + lmultioserver=.FALSE. + ELSE + icolor=3 + lioserver=.TRUE. + lmultioserver=.TRUE. + ENDIF + lio=lioserver + + CALL mpi_comm_split( mpi_comm_world, icolor, 0, iicomm, icode ) + IF ( icode /= MPI_SUCCESS ) THEN + WRITE(*,*)' mpp_io_init: Error in routine mpi_comm_split' + CALL mpi_abort( mpi_comm_world, icode, ierr ) + ENDIF + IF (lioserver) THEN + CALL mpi_comm_rank( iicomm, mppiorank, ierr ) + CALL mpi_comm_size( iicomm, mppiosize, ierr ) + WRITE(cdlogfile,'(A,I4.4,A)')'nemo_io_server.',mppiorank,'.log' + ELSE + mppiorank=0 + mppiosize=0 + ENDIF + lio=lioserver + + END SUBROUTINE mpp_io_init + + SUBROUTINE mpp_io_init_2( iicomm ) + + INTEGER, INTENT(INOUT) :: iicomm + + INTEGER :: icode, ierr, icolor, iicommx, iicommm, iicommo + INTEGER :: ji,inum + LOGICAL :: lcompp + INCLUDE "mpif.h" + + ! Construct multio server communicator + + IF (lmultioserver.OR..NOT.lioserver) THEN + icolor=12 + ELSE + icolor=13 + ENDIF + + CALL mpi_comm_split( iicomm, icolor, 0, pcommworldmultio, icode ) + IF ( icode /= MPI_SUCCESS ) THEN + WRITE(*,*)' mpp_io_init2: Error in routine mpi_comm_split' + CALL mpi_abort( mpi_comm_world, icode, ierr ) + ENDIF + + CALL mpi_comm_rank( pcommworldmultio, mppmultiorank, ierr ) + CALL mpi_comm_size( pcommworldmultio, mppmultiosize, ierr ) + + ! Construct compute communicator + + IF (.NOT.lioserver) THEN + icolor=14 + lcompp=.TRUE. + ELSE + icolor=15 + lcompp=.FALSE. + ENDIF + + CALL mpi_comm_split( iicomm, icolor, 0, iicommo, icode ) + IF ( icode /= MPI_SUCCESS ) THEN + WRITE(*,*)' mpp_io_init2: Error in routine mpi_comm_split' + CALL mpi_abort( mpi_comm_world, icode, ierr ) + ENDIF + + CALL mpi_comm_rank( iicommo, mppcomprank, ierr ) + CALL mpi_comm_size( iicommo, mppcompsize, ierr ) + + IF (.NOT.lioserver) THEN + CALL iom_initialize( "for_xios_mpi_id", return_comm=iicommm, global_comm = pcommworldmultio ) ! nemo local communicator given by xios + ELSE + ! For io-server tasks start an run the right server + CALL iom_init_server( server_comm = pcommworldmultio ) + ENDIF + + ! Return to the model with iicomm being compute only tasks + iicomm = iicommo + + END SUBROUTINE mpp_io_init_2 + + SUBROUTINE mpp_stop + INTEGER :: ierr + + IF (.NOT.lioserver) THEN + call iom_finalize() + ENDIF + + CALL mpi_finalize( ierr ) + END SUBROUTINE mpp_stop +#endif +END MODULE mpp_io diff --git a/src/io_meandata.F90 b/src/io_meandata.F90 index 314ee73a2..03d6906c1 100644 --- a/src/io_meandata.F90 +++ b/src/io_meandata.F90 @@ -20,6 +20,8 @@ module io_MEANDATA type(t_partit), pointer :: p_partit integer :: ndim integer :: glsize(2) + integer :: shrinked_size + integer, allocatable, dimension(:) :: shrinked_indx integer :: accuracy real(real64), allocatable, dimension(:,:) :: local_values_r8 real(real32), allocatable, dimension(:,:) :: local_values_r4 @@ -1151,6 +1153,9 @@ subroutine output(istep, ice, dynamics, tracers, partit, mesh) use MOD_ICE use mod_tracer use io_gather_module +#if defined(__MULTIO) + use iom +#endif #if defined (__icepack) use icedrv_main, only: init_io_icepack #endif @@ -1166,11 +1171,10 @@ subroutine output(istep, ice, dynamics, tracers, partit, mesh) type(t_tracer), intent(in) , target :: tracers type(t_dyn) , intent(in) , target :: dynamics type(t_ice) , intent(inout), target :: ice - - character(:), allocatable :: filepath - real(real64) :: rtime !timestamp of the record + character(:), allocatable :: filepath + real(real64) :: rtime !timestamp of the record - ctime=timeold+(dayold-1.)*86400 +ctime=timeold+(dayold-1.)*86400 !___________________________________________________________________________ if (lfirst) then @@ -1188,14 +1192,18 @@ subroutine output(istep, ice, dynamics, tracers, partit, mesh) !___________________________________________________________________________ !PS if (partit%flag_debug .and. partit%mype==0) print *, achar(27)//'[33m'//' -I/O-> call update_means'//achar(27)//'[0m' call update_means - !___________________________________________________________________________ ! loop over defined streams do n=1, io_NSTREAMS !_______________________________________________________________________ ! make pointer for entry onto io_stream object entry=>io_stream(n) - +!#if defined(__MULTIO) +! call mio_write_nod(mio, entry) +! lfirst=.false. +! return +!#endif + !_______________________________________________________________________ !check whether output will be written based on event frequency do_output=.false. @@ -1220,6 +1228,7 @@ subroutine output(istep, ice, dynamics, tracers, partit, mesh) ! if its time for output --> do_output==.true. if (do_output) then if (vec_autorotate) call io_r2g(n, partit, mesh) ! automatically detect if a vector field and rotate if makes sense! +#if !defined(__MULTIO) if(entry%thread_running) call entry%thread%join() entry%thread_running = .false. @@ -1272,7 +1281,7 @@ subroutine output(istep, ice, dynamics, tracers, partit, mesh) entry%rec_count=max(entry%rec_count, 1) write(*,*) trim(entry%name)//': current mean I/O counter = ', entry%rec_count end if ! --> if(partit%mype == entry%root_rank) then - +#endif !___________________________________________________________________ ! write double precision output if (entry%accuracy == i_real8) then @@ -1297,17 +1306,22 @@ subroutine output(istep, ice, dynamics, tracers, partit, mesh) END DO ! --> DO J=1, size(entry%local_values_r4,dim=2) !$OMP END PARALLEL DO end if ! --> if (entry%accuracy == i_real8) then - !___________________________________________________________________ entry%addcounter = 0 ! clean_meanarrays entry%ctime_copy = ctime - + +#if defined(__MULTIO) +! if (n==1) then + entry%rec_count = istep + call send_data_to_multio(entry) +! end if +#else !___________________________________________________________________ ! this is where the magic happens --> here do_output_callback is ! triggered as a method of the io_stream object --> call write_mean(...) call entry%thread%run() entry%thread_running = .true. - +#endif endif ! --> if (do_output) then end do ! --> do n=1, io_NSTREAMS lfirst=.false. @@ -1585,8 +1599,16 @@ subroutine def_stream_after_dimension_specific(entry, name, description, units, !___________________________________________________________________________ if(entry%glsize(1)==mesh%nod2D .or. entry%glsize(2)==mesh%nod2D) then entry%is_elem_based = .false. + entry%shrinked_size=partit%myDim_nod2D else if(entry%glsize(1)==mesh%elem2D .or. entry%glsize(2)==mesh%elem2D) then entry%is_elem_based = .true. + entry%shrinked_size=partit%myDim_elem2D_shrinked + allocate(entry%shrinked_indx(entry%shrinked_size)) + entry%shrinked_indx=partit%myInd_elem2D_shrinked +! write(*,*) partit%mype, partit%myDim_elem2D, partit%myDim_elem2D_shrinked, partit%myDim_elem2D-partit%myDim_elem2D_shrinked +! entry_index=0 +! call MPI_AllREDUCE(partit%myDim_elem2D_shrinked, entry_index, 1, MPI_INTEGER, MPI_SUM, partit%MPI_COMM_FESOM, err) +! write(*,*) 'total elem=', mesh%elem2D, entry_index else if(partit%mype == 0) print *,"can not determine if ",trim(name)," is node or elem based" stop @@ -1723,4 +1745,47 @@ subroutine io_r2g(n, partit, mesh) !$OMP END PARALLEL DO END IF end subroutine + +#if defined(__MULTIO) +SUBROUTINE send_data_to_multio(entry) + USE iom + USE multio_api + + IMPLICIT NONE + + TYPE(Meandata), TARGET, INTENT(INOUT) :: entry + TYPE(iom_field_request) :: request + REAL(real64), DIMENSION(SIZE(entry%shrinked_indx)), TARGET :: temp + INTEGER :: numLevels, globalSize, lev, i + + numLevels = entry%glsize(1) + globalSize = entry%glsize(2) + + request%name = trim(entry%name) + IF (.NOT. entry%is_elem_based) THEN + request%gridType = "ngrid" + ELSE + request%gridType = "egrid" + END IF + request%globalSize = globalSize + request%step = entry%rec_count + + ! loop over vertical layers --> do gather 3d variables layerwise in 2d slices + DO lev=1, numLevels + request%level = lev + + IF (.NOT. entry%is_elem_based) THEN + request%values => entry%local_values_r8_copy(lev, 1:entry%shrinked_size) + ELSE + DO i = 1, SIZE(entry%shrinked_indx) + temp(i) = entry%local_values_r8_copy(lev, entry%shrinked_indx(i)) + END DO + + request%values => temp + END IF + + CALL iom_send_fesom_data(request) + END DO +END SUBROUTINE +#endif end module diff --git a/src/io_restart.F90 b/src/io_restart.F90 index 86a04c5e6..6475655f2 100644 --- a/src/io_restart.F90 +++ b/src/io_restart.F90 @@ -82,15 +82,15 @@ subroutine ini_ocean_io(year, dynamics, tracers, partit, mesh) !___Save restart variables for TKE and IDEMIX_________________________________ ! if (trim(mix_scheme)=='cvmix_TKE' .or. trim(mix_scheme)=='cvmix_TKE+IDEMIX') then if (mix_scheme_nmb==5 .or. mix_scheme_nmb==56) then - call oce_files%def_node_var('tke', 'Turbulent Kinetic Energy', 'm2/s2', tke(:,:), mesh, partit) + call oce_files%def_node_var_optional('tke', 'Turbulent Kinetic Energy', 'm2/s2', tke(:,:), mesh, partit) endif ! if (trim(mix_scheme)=='cvmix_IDEMIX' .or. trim(mix_scheme)=='cvmix_TKE+IDEMIX') then if (mix_scheme_nmb==6 .or. mix_scheme_nmb==56) then - call oce_files%def_elem_var('iwe', 'Internal Wave Energy' , 'm2/s2', iwe(:,:), mesh, partit) + call oce_files%def_elem_var_optional('iwe', 'Internal Wave Energy' , 'm2/s2', iwe(:,:), mesh, partit) endif if (dynamics%opt_visc==8) then - call oce_files%def_elem_var('uke', 'unresolved kinetic energy', 'm2/s2', uke(:,:), mesh, partit) - call oce_files%def_elem_var('uke_rhs', 'unresolved kinetic energy rhs', 'm2/s2', uke_rhs(:,:), mesh, partit) + call oce_files%def_elem_var_optional('uke', 'unresolved kinetic energy', 'm2/s2', uke(:,:), mesh, partit) + call oce_files%def_elem_var_optional('uke_rhs', 'unresolved kinetic energy rhs', 'm2/s2', uke_rhs(:,:), mesh, partit) endif do j=1,tracers%num_tracers diff --git a/src/oce_ale.F90 b/src/oce_ale.F90 index 5d1d64778..dc72d7c16 100644 --- a/src/oce_ale.F90 +++ b/src/oce_ale.F90 @@ -3355,7 +3355,13 @@ subroutine oce_timestep_ale(n, ice, dynamics, tracers, partit, mesh) else call pressure_force_4_zxxxx(tracers, partit, mesh) end if - + + !___________________________________________________________________________ + ! check validity of visc_opt=5 selection + ! --> need to know buoyancy frequency to do so. + ! --> only check on the first timestep + if (n==1) call check_viscopt(dynamics, partit, mesh) + !___________________________________________________________________________ ! calculate alpha and beta ! it will be used for KPP, Redi, GM etc. Shall we keep it on in general case? diff --git a/src/oce_dyn.F90 b/src/oce_dyn.F90 index 57ac24fe7..db6bf24d8 100755 --- a/src/oce_dyn.F90 +++ b/src/oce_dyn.F90 @@ -54,6 +54,18 @@ subroutine visc_filt_bidiff(dynamics, partit, mesh) end interface end module +module check_validviscopt_interface + interface + subroutine check_validviscopt_5(partit, mesh) + USE MOD_MESH + USE MOD_PARTIT + USE MOD_PARSUP + type(t_partit), intent(inout), target :: partit + type(t_mesh) , intent(in) , target :: mesh + end subroutine + end interface +end module + ! ! Contains routines needed for computations of dynamics. ! includes: update_vel, compute_vel_nodes @@ -611,7 +623,7 @@ SUBROUTINE visc_filt_bidiff(dynamics, partit, mesh) !$OMP END PARALLEL DO !___________________________________________________________________________ -!$OMP PARALLEL DEFAULT(SHARED) PRIVATE(u1, v1, len, vi, ed, el, nz, nzmin, nzmax) +!$OMP PARALLEL DEFAULT(SHARED) PRIVATE(u1, v1, len, vi, ed, el, nz, nzmin, nzmax, update_u, update_v) !$OMP DO DO ed=1, myDim_edge2D+eDim_edge2D if(myList_edge2D(ed)>edge2D_in) cycle @@ -826,3 +838,151 @@ SUBROUTINE compute_apegen(dynamics, tracers, partit, mesh) call exchange_nod(dynamics%ke_swA, partit) call exchange_nod(dynamics%ke_swB, partit) END SUBROUTINE compute_apegen + + +! +! +!_______________________________________________________________________________ +! check validity of visc_opt=5 selection on basis of ratio betweem resolution and +! 1st baroclinic rossby radius. visc_opt=5 ("easy backscatter") in eddy +! resolving/partly eddy permitting setups can lead to problems in form of +! exedingly strong near boundary currents that can lead e.g to very weak +! Drake Passage throughflow (<80Sv). In this case better use visc_opt=7, +! which is the flow aware viscosity option +subroutine check_viscopt(dynamics, partit, mesh) + USE MOD_DYN + USE MOD_PARTIT + USE MOD_PARSUP + USE MOD_MESH + USE check_validviscopt_interface + IMPLICIT NONE + type(t_dyn) , intent(inout), target :: dynamics + type(t_partit), intent(inout), target :: partit + type(t_mesh) , intent(in) , target :: mesh + + if (dynamics%check_opt_visc) then + select case (dynamics%opt_visc) + case(5) + ! check validity of visc_opt=5 especially in higher resolved setups + ! --> there it can lead to problems + call check_validviscopt_5(partit, mesh) + end select + end if +end subroutine check_viscopt + +! +! +!_______________________________________________________________________________ +! check if viscopt=5 is a valid and recommended option for the used configuration +subroutine check_validviscopt_5(partit, mesh) + USE MOD_MESH + USE MOD_PARTIT + USE MOD_PARSUP + USE o_PARAM , ONLY: rad + USE o_ARRAYS, ONLY: bvfreq + USE g_CONFIG + USE g_comm_auto + IMPLICIT NONE + type(t_mesh), intent(in), target :: mesh + type(t_partit), intent(inout), target :: partit + integer :: node, nz, nzmax, nzmin + real(kind=WP) :: f_min=1.e-6_WP, z_min=100.0_WP, r_max=200000._WP, r_min=2000.0_WP + real(kind=WP) :: c_min=0.5_WP, c1 + real(kind=WP) :: excl_EQ=30.0, thresh_ratio=1.5 + real(kind=WP) :: loc_R, loc_A + real(kind=WP) :: glb_R, glb_A + real(kind=WP) :: fac_ResR1barocl, rossbyr_1barocl +#include "associate_part_def.h" +#include "associate_mesh_def.h" +#include "associate_part_ass.h" +#include "associate_mesh_ass.h" + + !___________________________________________________________________________ + loc_R = 0.0_WP + loc_A = 0.0_WP + +!$OMP PARALLEL DEFAULT(SHARED) PRIVATE(node, nz, nzmax, nzmin, c1, rossbyr_1barocl, & +!$OMP fac_ResR1barocl) REDUCTION(+:loc_R, loc_A) +!$OMP DO + do node=1, myDim_nod2D + !_______________________________________________________________________ + ! Exlude the equator |lat|<30° from checking the ration between resolution + ! and first baroclinic rossby radius + if (abs(mesh%geo_coord_nod2D(2,node)/rad)=1 setup is not eddy + ! resolving in best case eddy permitting --> GM/Redi will still be neccessary + fac_ResR1barocl =min(mesh_resolution(node)/rossbyr_1barocl, 5._WP) + + !_______________________________________________________________________ + ! compute local mean ratio but exclude equator |lat|<30°, since Rossby radius + ! at equator becomes very large + loc_R = loc_R + fac_ResR1barocl + loc_A = loc_A + 1.0_WP + end do +!$OMP END DO +!$OMP END PARALLEL +!$OMP BARRIER + + !___________________________________________________________________________ + ! compute global mean ratio --> core2 Ratio=4.26 (eddy parameterizted), + ! dart Ratio=0.97 (eddy resolving/permitting) + call MPI_AllREDUCE(loc_R, glb_R, 1, MPI_DOUBLE_PRECISION, MPI_SUM, MPI_COMM_FESOM, MPIerr) + call MPI_AllREDUCE(loc_A, glb_A, 1, MPI_DOUBLE_PRECISION, MPI_SUM, MPI_COMM_FESOM, MPIerr) + glb_R = glb_R/glb_A + + !___________________________________________________________________________ + ! create warning message when ratio Mean Ratio Resol/Rrossby = ', glb_R + write(*,*) '____________________________________________________________________' + write(*,*) ' WARNING: You want to use opt_visc=5 (easy backscatter viscosity) in' + write(*,*) ' an eddy resolving to eddy permitting (Resol/Rrossby<1.5) ' + write(*,*) ' configuration. It revealed to us that this can lead to ' + write(*,*) ' problems in form of unrealistically strong near boundary ' + write(*,*) ' currents that can ultimatively lead to an e.g very weak ' + write(*,*) ' Drake Passage throughflow (<80Sv in spinup up dart ' + write(*,*) ' configuration). For these kind of setups we recommend to ' + write(*,*) ' use opt_visc=7, which is the flow aware viscosity ' + write(*,*) ' parameterisation. ' + write(*,*) ' The easy backscatter viscosity was designed to energetise ' + write(*,*) ' coarse resolved configurations!!! ' + write(*,*) ' --> So please change the viscosity option in namelist.dyn ' + write(*,*) ' to opt_visc=7 . ' + write(*,*) ' --> If you insist in using opt_visc=5 in your simulation, ' + write(*,*) ' you can switch off this check in namelist.dyn by ' + write(*,*) ' setting check_opt_visc=.false. ' + write(*,*) ' ' + write(*,*) '____________________________________________________________________' + print *, achar(27)//'[0m' + write(*,*) + call par_ex(partit%MPI_COMM_FESOM, partit%mype, 0) + end if + else + if (mype==0) then + write(*,*) ' --check opt_visc--> Mean Ratio Resol/Rrossby = ', glb_R + end if + end if + +end subroutine check_validviscopt_5 diff --git a/src/oce_mesh.F90 b/src/oce_mesh.F90 index 653785313..1843e345b 100755 --- a/src/oce_mesh.F90 +++ b/src/oce_mesh.F90 @@ -842,28 +842,28 @@ SUBROUTINE read_mesh(partit, mesh) n=com_elem2D_full%sptr(com_elem2D_full%sPEnum+1)-1 ALLOCATE(com_elem2D_full%slist(n)) read(fileID,*) com_elem2D_full%slist - -!!$ read(fileID,*) com_edge2D%rPEnum -!!$ ALLOCATE(com_edge2D%rPE(com_edge2D%rPEnum)) -!!$ read(fileID,*) com_edge2D%rPE -!!$ ALLOCATE(com_edge2D%rptr(com_edge2D%rPEnum+1)) -!!$ read(fileID,*) com_edge2D%rptr -!!$ ALLOCATE(com_edge2D%rlist(eDim_edge2D)) -!!$ read(fileID,*) com_edge2D%rlist -!!$ -!!$ read(fileID,*) com_edge2D%sPEnum -!!$ ALLOCATE(com_edge2D%sPE(com_edge2D%sPEnum)) -!!$ read(fileID,*) com_edge2D%sPE -!!$ ALLOCATE(com_edge2D%sptr(com_edge2D%sPEnum+1)) -!!$ read(fileID,*) com_edge2D%sptr -!!$ n=com_edge2D%sptr(com_edge2D%sPEnum+1)-1 -!!$ ALLOCATE(com_edge2D%slist(n)) -!!$ read(fileID,*) com_edge2D%slist close(fileID) + if (mype==0) write(*,*) 'communication arrays are read' deallocate(rbuff, ibuff) deallocate(mapping) - + +! necessary for MULTIO auxuary data: +! one element might belong to several processes hence we unify the element partition +! such that sum(myDim_elem2D_shrinked) over all processors will give elem2D + partit%myDim_elem2D_shrinked=0 + DO n=1, myDim_elem2D + if (mesh%elem2D_nodes(1, n) > myDim_nod2D) cycle + partit%myDim_elem2D_shrinked=partit%myDim_elem2D_shrinked+1 + END DO + allocate(partit%myInd_elem2D_shrinked(partit%myDim_elem2D_shrinked)) +! fill the respective indicies + nn=1 + DO n=1, myDim_elem2D + if (mesh%elem2D_nodes(1, n) > myDim_nod2D) cycle + partit%myInd_elem2D_shrinked(nn)=n + nn=nn+1 + END DO ! no checksum for now, execute_command_line is failing too often. if you think it is important, please drop me a line and I will try to revive it: jan.hegewald@awi.de mesh%representative_checksum = '' diff --git a/src/oce_setup_step.F90 b/src/oce_setup_step.F90 index 9984328ab..d0fb21572 100755 --- a/src/oce_setup_step.F90 +++ b/src/oce_setup_step.F90 @@ -261,6 +261,7 @@ subroutine ocean_setup(dynamics, tracers, partit, mesh) write(*,*) 'maximum allowed CDF on explicit W is set to: ', dynamics%wsplit_maxcfl write(*,*) '******************************************************************************' end if + end subroutine ocean_setup ! ! @@ -390,6 +391,7 @@ SUBROUTINE dynamics_init(dynamics, partit, mesh) logical :: use_freeslip =.false. logical :: use_wsplit =.false. logical :: ldiag_KE =.false. + logical :: check_opt_visc=.true. real(kind=WP) :: wsplit_maxcfl logical :: use_ssh_splitexpl_subcycl=.false. integer :: splitexpl_BTsteps @@ -397,7 +399,7 @@ SUBROUTINE dynamics_init(dynamics, partit, mesh) logical :: splitexpl_visc, splitexpl_bottdrag, splitexpl_bdrag_si real(kind=WP) :: splitexpl_visc_gamma0, splitexpl_visc_gamma1, splitexpl_visc_gamma2 - namelist /dynamics_visc / opt_visc, visc_gamma0, visc_gamma1, visc_gamma2, & + namelist /dynamics_visc / opt_visc, check_opt_visc, visc_gamma0, visc_gamma1, visc_gamma2, & use_ivertvisc, visc_easybsreturn namelist /dynamics_general/ momadv_opt, use_freeslip, use_wsplit, wsplit_maxcfl, & ldiag_KE, use_ssh_splitexpl_subcycl, splitexpl_BTsteps, & @@ -427,6 +429,7 @@ SUBROUTINE dynamics_init(dynamics, partit, mesh) !___________________________________________________________________________ ! set parameters in derived type dynamics%opt_visc = opt_visc + dynamics%check_opt_visc = check_opt_visc dynamics%visc_gamma0 = visc_gamma0 dynamics%visc_gamma1 = visc_gamma1 dynamics%visc_gamma2 = visc_gamma2 @@ -448,7 +451,7 @@ SUBROUTINE dynamics_init(dynamics, partit, mesh) dynamics%splitexpl_visc_gamma1 = splitexpl_visc_gamma1 dynamics%splitexpl_visc_gamma2 = splitexpl_visc_gamma2 end if - + !___________________________________________________________________________ ! define local vertice & elem array size elem_size=myDim_elem2D+eDim_elem2D @@ -791,6 +794,7 @@ SUBROUTINE arrays_init(num_tracers, partit, mesh) !!PS dum_3d_n = 0.0_WP !!PS dum_2d_e = 0.0_WP !!PS dum_3d_e = 0.0_WP + END SUBROUTINE arrays_init ! ! diff --git a/src/write_step_info.F90 b/src/write_step_info.F90 index 33517654e..94421ac0f 100644 --- a/src/write_step_info.F90 +++ b/src/write_step_info.F90 @@ -448,7 +448,7 @@ subroutine check_blowup(istep, ice, dynamics, tracers, partit, mesh) end if ! --> if ( .not. trim(which_ALE)=='linfs' .and. ... - do nz=1,nlevels_nod2D(n)-1 + do nz=ulevels_nod2D(n),nlevels_nod2D(n)-1 !_______________________________________________________________ ! check temp if ( (tracers%data(1)%values(nz, n) /= tracers%data(1)%values(nz, n)) .or. & @@ -502,11 +502,11 @@ subroutine check_blowup(istep, ice, dynamics, tracers, partit, mesh) !_______________________________________________________________ ! check salt if ( (tracers%data(2)%values(nz, n) /= tracers%data(2)%values(nz, n)) .or. & - tracers%data(2)%values(nz, n) < 0 .or. tracers%data(2)%values(nz, n)>50 ) then + tracers%data(2)%values(nz, n) <=3.0_WP .or. tracers%data(2)%values(nz, n)>=45.0_WP ) then !$OMP CRITICAL found_blowup_loc=1 write(*,*) '___CHECK FOR BLOW UP___________ --> mstep=',istep - write(*,*) ' --STOP--> found salinity becomes NaN or <0, >50' + write(*,*) ' --STOP--> found salinity becomes NaN or <=3.0, >=45.0' write(*,*) 'mype = ',mype write(*,*) 'mstep = ',istep write(*,*) 'node = ',n diff --git a/work/job_ini_ollie b/work/job_ini_ollie deleted file mode 100755 index 693296bad..000000000 --- a/work/job_ini_ollie +++ /dev/null @@ -1,31 +0,0 @@ -#!/bin/bash -#SBATCH --job-name=fesom2.0_INI -#SBATCH -p mpp -#SBATCH --ntasks=32 -#SBATCH --time=12:00:00 -#SBATCH -o slurm-out.out -#SBATCH -e slurm-err.out - -set -x - -ulimit -s unlimited - -module purge -module load intel.compiler -module load intel.mpi -module load netcdf/4.4.0_intel - -ln -s ../bin/fesom_ini.x . # cp -n ../bin/fvom_ini.x -cp -n ../config/namelist.config . -cp -n ../config/namelist.forcing . -cp -n ../config/namelist.oce . -cp -n ../config/namelist.ice . - - -# determine JOBID -JOBID=`echo $SLURM_JOB_ID |cut -d"." -f1` - -date -srun --mpi=pmi2 --ntasks=1 ./fesom_ini.x > "fvom_ini.out" -date - diff --git a/work/job_ollie b/work/job_ollie deleted file mode 100755 index ac2516115..000000000 --- a/work/job_ollie +++ /dev/null @@ -1,34 +0,0 @@ -#!/bin/bash -#SBATCH --job-name=fesom2.0 -#SBATCH -p mpp -#SBATCH --ntasks=288 -#SBATCH --time=00:05:00 -#SBATCH -o slurm-out.out -#SBATCH -e slurm-err.out -module load intel.compiler intel.mpi netcdf/4.4.0_intel -module load centoslibs - -set -x - -ulimit -s unlimited - -# determine JOBID -JOBID=`echo $SLURM_JOB_ID |cut -d"." -f1` - -ln -s ../bin/fesom.x . # cp -n ../bin/fesom.x -cp -n ../config/namelist.config . -cp -n ../config/namelist.forcing . -cp -n ../config/namelist.oce . -cp -n ../config/namelist.ice . -cp -n ../config/namelist.io . -cp -n ../config/namelist.icepack . - -date -srun --mpi=pmi2 ./fesom.x > "fesom2.0.out" -date - -#qstat -f $PBS_JOBID -#export EXITSTATUS=$? -#if [ ${EXITSTATUS} -eq 0 ] || [ ${EXITSTATUS} -eq 127 ] ; then -#sbatch job_ollie -#fi diff --git a/work/job_ollie_chain b/work/job_ollie_chain deleted file mode 100755 index 8f98f1515..000000000 --- a/work/job_ollie_chain +++ /dev/null @@ -1,317 +0,0 @@ -#!/bin/bash -#___SET SLURM OPTIONS___________________________________________________________ -#SBATCH -J chain -#SBATCH -p mpp -#SBATCH --ntasks=432 -#SBATCH --time=10:00:00 -#SBATCH --mail-type=END -#SBATCH --mail-user=Patrick.Scholz@awi.de -#SBATCH -o fesom2.0_%x_%j.out -#SBATCH -e fesom2.0_%x_%j.out - -## module load intel.compiler intel.mpi netcdf centoslibs - -#___DEFAULT INPUT_______________________________________________________________ -# how many job chains should be applied -chain_n=3 # number chain cycles -chain_s=1 # starting chain id - -# time frame of model simulation -# ___COREv2___ -year_s=1948 -year_e=2009 -# ___JRA55____ -#year_s=1958 -#year_e=2018 - -prescribe_rlen=0 # run length in namelist.config --> if 0 value from namelist.config is taken -fedit=1 - -#___HELP OUTPUT_________________________________________________________________ -script_name=job_ollie_chain -function usage { - echo "usage: $script_name [-cn ...] [-cs ...] [-ys ...] [-ye ...] [-wcl ...]" - echo " -cn number of chain cylces (default: 3)" - echo " -cs starting chain id (default: 1)" - echo " -ys starting year of model simulation (default: 1948)" - echo " -ye ending year of model simulation (default: 2009)" - echo " -h display help" - echo " -rl prescribe used run_length" - echo " -noedit no automatic editing of namelist.config" - echo - echo " --> for changing the wall-clock-time interactively use " - echo " sbatch --time=00:10:00 job_ollie_chain ..." - echo " --> for changing the number of task interactively use " - echo " sbatch --ntask=288 job_ollie_chain ..." - echo " --> for changing the job name interactively use " - echo " sbatch --job-name=whatever job_ollie_chain ..." - exit 1 -} - -#___OVERRIDE DEFAULT INPUT BY COMMANDLINE INPUT_________________________________ -while [ "$1" != "" ]; do - case $1 in - -cn | -chain_n ) shift ; chain_n=$1 ;; - -cs | -chain_s ) shift ; chain_s=$1 ;; - -ys | -year_s ) shift ; year_s=$1 ;; - -ye | -year_e ) shift ; year_e=$1 ;; - -rl | --run_length ) shift ; prescribe_rlen=$1 ;; - -noedit | --noedit ) fedit=0 ;; - -h | --help ) usage ; exit ;; - esac - shift -done - -#___EXTRACT WALL-CLOCK-TIME FROM JOBINFO________________________________________ -# either setted via #SBATCH time=... or at command line sbatch --time=... job_ollie_chain -# need here to extract to give the next job chain cycle as input -jobinfo=$(scontrol show job $SLURM_JOB_ID) -wcl=$( echo ${jobinfo##*"TimeLimit="} | cut -d " " -f 1 ) -ntasks=$( echo ${jobinfo##*"NumTasks="} | cut -d " " -f 1 ) - - -#___SET NAMELIST'S & EXECUTABLE IF NOT ALREADY EXIST____________________________ -set -x -ulimit -s unlimited -ln -s ../bin/fesom.x . # cp -n ../bin/fvom_ini.x -cp -n ../config/namelist.config . -cp -n ../config/namelist.forcing . -cp -n ../config/namelist.oce . -cp -n ../config/namelist.ice . - -#___SET CHAIN_ID________________________________________________________________ -if [ -f "file_chain_id" ]; then - chain_id=$( file_chain_id -fi - -#___PRINT INPUT INFO____________________________________________________________ -echo -e "\033[1;7;33m_____JOB CHAIN INFO_____________________________________\033[0m" -echo -e "\033[1;33m --> actual chain cycle: $chain_id \033[0m" -echo -e "\033[1;33m --> max. number of chain cycles: $chain_n \033[0m" -echo -e "\033[1;33m --> simulated time range: [ $year_s $year_e] \033[0m" -echo -e "\033[1;33m --> slurm: wall-clock-time = $wcl \033[0m" -echo -e "\033[1;33m --> slurm: ntask = $ntasks \033[0m" -if [ $prescribe_rlen -ne 0 ]; then - echo -e "\033[1;33m -->change run_length = $prescribe_rlen \033[0m" -fi - -#___CREATE SAVE DIR INFRASTRUCTURE______________________________________________ -# extract resultpath from namelist.config -dname_result_link_orig=$(grep "ResultPath=" namelist.config | grep -v '^!' | \ - cut -d "=" -f 2 | \ - cut -d "'" -f 2) -dname_result_link=$(echo ${dname_result_link_orig::-1}) - -# identify higher directory --> ResultPath -dname_result="$(dirname "$dname_result_link")/" - -# if ResultPath directory doesn't exist --> create it -if [ ! -d "${dname_result}" ]; then - echo -e "\033[33m --> ResultPath directory does not exist --> will create it \033[0m" - mkdir "${dname_result}" -fi - -# check if in namelist.config a chain path is given (that mean -# ResultPath=.../ResultDir/chain/) if not change namelist.config accordingly -check=${dname_result_link##*${dname_result}} -if [ $fedit -eq 1 ] && [ $check != "chain" ]; then - echo -e "\033[33m --> replace in namelist.config ResultPath with chain path \033[0m" - dname_result_link="${dname_result}chain" - sed -i "s|${dname_result_link_orig}|${dname_result_link}/|g" namelist.config -fi - -# identify real path in case a link is used -dname_result="$(realpath "$dname_result")/" - -# if directory for chain_id doesn't exist --> create it -if [ ! -d "${dname_result}/${chain_id}" ]; then - echo -e "\033[33m --> chain_id directory does not exist --> will create it \033[0m" - mkdir "${dname_result}/${chain_id}" -fi - -# link directory of chain_id with original linkdirectory from namelist.config -ln -sfn ${dname_result}${chain_id} $dname_result_link - -#___CHECK IF SIMULATION NEEDS TO BE INITIALISED OR CONTINUED____________________ -is_newsimul=1 -if [ -f "$dname_result_link/fesom.clock" ] ; then - aux_yr_clock=$(<${dname_result_link}/fesom.clock) - aux_yr_clock=$(echo ${aux_yr_clock} | cut -d" " -f 6) - if [ $aux_yr_clock -le $year_e ]; then is_newsimul=0 ; fi - - #___________________________________________________________________________ - if [ $fedit -eq 1 ] ; then - if [ $is_newsimul -eq 0 ] ; then - year_d=$(( $year_e - $aux_yr_clock + 1 )) - rlen=$(grep "run_length=" namelist.config | cut -d "=" -f 2 | cut -d " " -f 1) - # do not prescribe run length in job fle - if [ $prescribe_rlen -eq 0 ] ; then - if [ $rlen -ne $year_d ] ; then - sed -i " s/run_length=$rlen/run_length=$year_d/" namelist.config - echo -e "\033[1;33m --> change run_length to: $year_d \033[0m" - fi - # prescribe run length in job fle - else - aux_year_d=$prescribe_rlen - if [ $year_d -lt $aux_year_d ] ; then aux_year_d=$year_d ; fi - if [ $rlen -ne $year_d ] ; then - sed -i " s/run_length=$rlen/run_length=$aux_year_d/" namelist.config - echo -e "\033[1;33m --> change run_length to: $aux_year_d \033[0m" - fi - fi - fi - fi -else - #___________________________________________________________________________ - # set model run length in job_script and change namelist.config accordingly - # to match - if [ $fedit -eq 1 ] ; then - year_d=$(( $year_e - $year_s + 1 )) - rlen=$(grep "run_length=" namelist.config | cut -d "=" -f 2 | cut -d " " -f 1) - # do not prescribe run length in job fle - if [ $prescribe_rlen -eq 0 ] ; then - if [ $rlen -ne $year_d ] ; then - sed -i " s/run_length=$rlen/run_length=$year_d/" namelist.config - echo -e "\033[1;33m --> change run_length to: $year_d \033[0m" - fi - # prescribe run length in job file - else - aux_year_d=$prescribe_rlen - if [ $year_d -lt $aux_year_d ] ; then aux_year_d=$year_d ; fi - if [ $rlen -ne $year_d ] ; then - sed -i " s/run_length=$rlen/run_length=$aux_year_d/" namelist.config - echo -e "\033[1;33m --> change run_length to: $aux_year_d \033[0m" - fi - fi - fi -fi - -#___CREATE CLOCK & RESTART INFRASTRUCTURE FOR COLD/WARM START___________________ -# only touch clock file when a new simulation is supposed to start, if an old one -# should be continued dont touch it -if [ $is_newsimul -eq 1 ] ; then - - # --> make cold start - if [ $chain_id -eq 1 ] ; then - #_______________________________________________________________________ - # create cold start clock file - echo -e "\033[33m --> create cold start clock file \033[0m" - printf "0 1 ${year_s}\n0 1 ${year_s}" > $dname_result_link/fesom.clock - - #_______________________________________________________________________ - # in case yearnew in namelist.config was changed from 1948 - yearnew=$(grep "yearnew=" namelist.config | cut -d "=" -f 2) - if [ $yearnew -ne $year_s ]; then - sed -i " s/yearnew=$yearnew/yearnew=$year_s/" namelist.config - fi - - #___BACKUP NAMELIST.* FILES INTO RESULT DIRECTORY_______________________ - cp namelist.config namelist.oce namelist.ice namelist.forcing namelist.io \ - namelist.cvmix namelist.tra namelist.dyn ${dname_result}/. - cp fesom.x ${dname_result}/. - - #___BACKUP SRC FILES INTO RESULT DIRECTORY______________________________ - if [ ! -d "${dname_result}/src" ]; then mkdir "${dname_result}/src" ; fi - cp ../src/*.F90 ${dname_result}/src/. - - - # --> make warm start - else - #_______________________________________________________________________ - # create warm start clock file - stepperday=$(grep "step_per_day=" namelist.config | cut -d "=" -f 2 | cut -d " " -f 1 ) - aux_sec=$(( 86400 - 86400 / $stepperday )) - aux_day=365 - aux_yr=$(( $year_s - 1 )) - echo -e "\033[33m --> create warm start clock file \033[0m" - printf "${aux_sec} ${aux_day} ${aux_yr}\n0 1 ${year_s}" > $dname_result_link/fesom.clock - - #_______________________________________________________________________ - # chain id from previous spinup cycle - prev_chain_id=$(( $chain_id - 1 )) - - #_______________________________________________________________________ - # copy restart ocean files/directories from previous spinup cycle - prev_rfile=${dname_result}${prev_chain_id}/fesom.${year_e}.oce.restart - if [ -d "${prev_rfile}" ]; then - cp -r ${prev_rfile} ${dname_result_link}/fesom.${aux_yr}.oce.restart - elif [ -f "${prev_rfile}.nc" ]; then - cp ${prev_rfile}.nc ${dname_result_link}/fesom.${aux_yr}.oce.restart.nc - else - echo -e "\033[1;31m --> ERROR: could not find ocean restart file \033[0m" - exit - fi - - # copy restart ice files/files/directories from previous spinup cycle - prev_rfile=${dname_result}${prev_chain_id}/fesom.${year_e}.ice.restart - if [ -d "${prev_rfile}" ]; then - cp -r ${prev_rfile} ${dname_result_link}/fesom.${aux_yr}.ice.restart - elif [ -f "${prev_rfile}.nc" ]; then - cp ${prev_rfile}.nc ${dname_result_link}/fesom.${aux_yr}.ice.restart.nc - else - echo -e "\033[1;31m --> ERROR: could not find ice restart file \033[0m" - exit - fi - - #_______________________________________________________________________ - # adapt year new in namelist.config otherwise fesom is not doing a - # restart - aux=$(grep "yearnew=" namelist.config | cut -d "=" -f 2 | cut -d " " -f 1 ) - sed -i " s/yearnew=$aux/yearnew=$aux_yr/" namelist.config - - #_______________________________________________________________________ - # backup namelist.* & fesom.x in case they dont exist - if [ ! -f "${dname_result}/namelist.config" ]; then - cp namelist.config namelist.oce namelist.ice namelist.forcing namelist.io \ - namelist.cvmix namelist.tra namelist.dyn ${dname_result}/. - fi - if [ ! -f "${dname_result}/fesom.x" ]; then - cp fesom.x ${dname_result}/. - fi - #___BACKUP SRC FILES INTO RESULT DIRECTORY______________________________ - if [ ! -d "${dname_result}/src" ]; then mkdir "${dname_result}/src" ; fi - cp ../src/*.F90 ${dname_result}/src/. - fi -fi - -#___DETERMINE SLURM JOBID+OUTPUTFILE____________________________________________ -jobid=$(echo $SLURM_JOB_ID | cut -d"." -f1) -fname="fesom2.0_${SLURM_JOB_NAME}_${jobid}.out" - -#___PUT JOB IN QUEUE____________________________________________________________ -date -srun --mpi=pmi2 ./fesom.x >> ${fname} -err_srun=$? -echo -e "\033[33m --> err_srun=${err_srun} \033[0m" -date - -#___SHOW JOB INFORMATION________________________________________________________ -scontrol show job $SLURM_JOB_ID - -#___SETUP JOBCHAIN______________________________________________________________ -# check if complete cycle is finished only than increase chain_id -aux_yr_clock=$(<${dname_result_link}/fesom.clock) -aux_yr_clock=$(echo ${aux_yr_clock} | cut -d" " -f 6) - -# setup next chain job via dependence -if [ ${err_srun} -eq 0 ]; then - if [ $aux_yr_clock -lt $year_e ] || [ ${chain_id} -lt ${chain_n} ]; then - # aftercorr:job_id --> A task of this job array can begin execution after - # the corresponding task ID in the specified job has completed successfully - # (ran to completion with an exit code of zero). - echo -e "\033[33m --> setup next chain cycle \033[0m" - sbatch --time=$wcl --ntasks=$ntasks --job-name=${SLURM_JOB_NAME} --dependency=aftercorr:$SLURM_JOB_ID $script_name \ - -cn $chain_n -cs $chain_s -ys $year_s -ye $year_e -rl $prescribe_rlen - fi -fi - -#___CHECK FOR COMPLETNES________________________________________________________ -# check if complete cycle is finished only than increase chain_id -if [ $aux_yr_clock -gt $year_e ] && [ ${chain_id} -lt ${chain_n} ] ; then - chain_id=$(( $chain_id + 1 )) - echo $chain_id > file_chain_id -fi -