Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CMake Chem and Chem+KPP Build #2018

Open
wants to merge 25 commits into
base: develop
Choose a base branch
from
Open

Conversation

islas
Copy link
Collaborator

@islas islas commented Mar 11, 2024

TYPE: enhancement

KEYWORDS: cmake, chem, kpp

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
Current CMake build does not build chem or chem+kpp configurations

Solution:
Build kpp and associated tools, and cmake commands to facilitate simplified logic of the configure_wkc and compile_wkc scripts. As with all CMake builds, all auto-generated source code is placed in the out-of-source build directory.

Notable differences to make build :

  • Use of Bison instead of Yacc as it is more easily accessible for install and usage as well as backward compatible
  • Allow -j N parallel jobs to generate KPP sources up to a limit
  • Use KPP-generated source file original names (not renamed to module_kpp_*
  • Pass tuv_kpp a directory to locate where include file is to be generated, and allow control of file IO mode*
  • Allow integration decomp rewrite to specify file locations rather than hard-coded*
  • registry uses -DWRF_CHEM and -DWRF_KPP defines passed at command line instead of getenv() to match all other options*

*Affects make build in subtle ways but do not change user instructions

LIST OF MODIFIED FILES:
M CMakeLists.txt
M chem/CMakeLists.txt
A chem/KPP/CMakeLists.txt
M chem/KPP/compile_wkc
A chem/KPP/kpp/kpp-2.1/CMakeLists.txt
A chem/KPP/util/wkc/CMakeLists.txt
M chem/KPP/util/wkc/gen_kpp.c
M chem/KPP/util/wkc/protos_kpp.h
M chem/KPP/util/wkc/tuv_kpp.c
A chem/KPP/util/write_decomp/CMakeLists.txt
M chem/KPP/util/write_decomp/Makefile
M chem/KPP/util/write_decomp/integr_edit.c
M chem/chem_driver.F
M tools/CMakeLists.txt
M tools/data.h
M tools/registry.c

TESTS CONDUCTED:

  1. Reproduction of chem and chem+kpp regtests with cmake is possible now

RELEASE NOTE:
CMake Chem and Chem+KPP Build

@islas islas requested review from a team as code owners March 11, 2024 20:51
@weiwangncar
Copy link
Collaborator

The Jenkins test results:

Test Type              | Expected  | Received |  Failed
= = = = = = = = = = = = = = = = = = = = = = = =  = = = =
Number of Tests        : 23           24
Number of Builds       : 60           57
Number of Simulations  : 158           150        0
Number of Comparisons  : 95           86        0

Failed Simulations are: 
None
Which comparisons are not bit-for-bit: 
None

@islas islas requested review from a team as code owners August 6, 2024 02:34
@islas islas changed the base branch from develop to release-v4.6.1 August 6, 2024 03:08
@islas
Copy link
Collaborator Author

islas commented Aug 6, 2024

Requires #2056, #2053, #2086, #2087, and #2088

@islas islas changed the base branch from release-v4.6.1 to develop December 9, 2024 23:38
@amstokely amstokely self-requested a review December 16, 2024 17:20
amstokely
amstokely previously approved these changes Dec 16, 2024
Copy link
Collaborator

@amstokely amstokely left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While replacing the call to system(cp_command) with a safer alternative would be a good improvement, this system call was not introduced in this PR. Addressing it might be better suited for a future PR.


system(cp_command);
#ifndef NO_COPY
system(cp_command);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling system() with a command-line-provided argument introduces significant security risks, as it can allow command injection if the input is not carefully sanitized. Could we replace system(cp_command) with a safer alternative that avoids directly invoking the shell?

For example, the following implementation securely copies files using fork() and execlp():

#include <unistd.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

void copy_file(const char *source, const char *destination) {
    pid_t pid = fork();
    if (pid == 0) {
        // Child process: Securely execute the `cp` command
        execlp("cp", "cp", source, destination, NULL);
        perror("execlp failed"); // If exec fails
        exit(EXIT_FAILURE);
    } else if (pid > 0) {
        // Parent process: Wait for the child to complete
        int status;
        waitpid(pid, &status, 0);
        if (WIFEXITED(status)) {
            printf("Copy completed with exit code %d\n", WEXITSTATUS(status));
        } else {
            fprintf(stderr, "Copy process terminated abnormally\n");
        }
    } else {
        perror("fork failed");
        exit(EXIT_FAILURE);
    }
}

int main(int argc, char *argv[]) {
    if (argc != 2) {
        fprintf(stderr, "Usage: %s \"<source_file> <destination_file>\"\n", argv[0]);
        return EXIT_FAILURE;
    }

    // Parse the cp_command argument into source and destination
    char *cp_command = argv[1];
    char *source = strtok(cp_command, " ");
    char *destination = strtok(NULL, " ");

    if (!source || !destination) {
        fprintf(stderr, "Error: Invalid command. Provide source and destination files.\n");
        return EXIT_FAILURE;
    }

    // Call the copy_file function
    printf("Copying '%s' to '%s'...\n", source, destination);
    copy_file(source, destination);

    return EXIT_SUCCESS;
}

mgduda
mgduda previously approved these changes Dec 20, 2024
Copy link
Collaborator

@mgduda mgduda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything seems to compile successfully if I select chem and kpp as additional configuration options, though I actually have no idea how to run anything with chem+kpp to verify that the compiled model includes chemistry functionality.

@islas islas dismissed stale reviews from mgduda and amstokely via 7adbe6e December 21, 2024 00:08
@amstokely amstokely self-requested a review January 7, 2025 19:33
amstokely
amstokely previously approved these changes Jan 7, 2025
@mgduda
Copy link
Collaborator

mgduda commented Jan 10, 2025

I'm getting a compilation error that's probably due to something I'm doing wrong. On Derecho, starting with the default modules I first switch to the GNU compilers with ml gcc. Then, I merge the cmake-kpp branch into the develop branch and run configure_new with all default choices, except I configure the additional options 5 and 10 (for chem and kpp, respectively). Running compile_new fails with:

gcc: warning: /glade/derecho/scratch/duda/wrf_cmake/tools/gen_comms.stub: linker input file unused because linking not done
[  2%] Linking C executable registry_kpp
/usr/bin/ld: cannot find CMakeFiles/registry_kpp.dir/__/__/__/__/tools/gen_comms.stub.o: No such file or directory
collect2: error: ld returned 1 exit status
make[2]: *** [chem/KPP/util/wkc/CMakeFiles/registry_kpp.dir/build.make:417: chem/KPP/util/wkc/registry_kpp] Error 1
make[1]: *** [CMakeFiles/Makefile2:4205: chem/KPP/util/wkc/CMakeFiles/registry_kpp.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

@islas
Copy link
Collaborator Author

islas commented Jan 11, 2025

@mgduda I think I've isolated the issue to the version drop in #2132. Configuration should have looked something like

-- Configuring done (31.4s)
CMake Error at CMakeLists.txt:923 (add_library):
  Cannot find source file:

    /glade/work/aislas/wrf-model/wrf/_build/chem/module_kpp_cb05_sorg_aq_interface.F

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h
  .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc


-- Generating done (2.0s)
CMake Generate step failed.  Build files cannot be regenerated correctly.

Even after fixing these issues, the .stub file for registry generation still fails as the gcc compiler isn't being passed in -x c to override language detection via extension. This is a distinct problem in the cmake <= 3.19 (except 3.19.0) and was resolved in 3.20, along with backwards compatibility provided via CMake Policy CMP0119.

I've tested reverting back to cmake_minimum_required( VERSION 3.20 ) and that works again. As an alternative, we could move gen_comms.stubs -> gen_comm.stubs.c so that the stubs themselves can be compiled with the proper extension.

Note: the reason this wasn't caught in normal testing for the version requirement drop is the default for normal compilation is to use BASIC nesting which does not use the gen_comms.stub

Other parts of the cmake build rely on language
overrides being fed into the compiler. While this
could be adjusted to maintain the lower version
number, these adjustments to the source code are
outside the scope of these changes. Coupled with
the only-recent version drop, the more appropriate
approach is to revert the version back to the
higher working version and evaluate the necessary
code adjustments for cmake 3.19 to be viable at
a later time in a separate feature.
@islas
Copy link
Collaborator Author

islas commented Jan 13, 2025

@amstokely @mgduda Update pushed to (unfortunately) revert back to v3.20. I looked into converting gen_comms.stub to some other form, whether preprocessing / on-the-fly moving it to a .c file similar to the makefiles or outright git mv gen_comms.stub gen_comms.stub.c. Both these options were involved enough that I would like to address them outside this PR. Commit 5129e3f contains this reasoning as well

@mgduda mgduda self-requested a review January 14, 2025 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants