Fix clasic flang compilation with in-tree flang

[dmikushin]

This set of patches fixes various issues spotted during compilation of flang-classic with clang and LLVM in-tree flang.

For C/C++ it addresses #1374 and #1375 and fixes fatal clang issues, such as mismatching function pointer signature.

For Fortran it mostly adds a lot of intent(in) attributes.

[bryanpkc]

A few quick initial comments.

1. This PR contains a lot of patches that are unrelated to each other. Ideally, they should be submitted as separate PRs.
2. The Classic Flang runtime code is not intended to be compiled with LLVM Flang. The latter has its own runtime libraries. Why are you trying to do this?
3. This PR duplicates a lot of in-progress work we are already doing for release_16x branch of classic-flang-llvm-project. I suggest you wait until that is released if you really need LLVM 16 support.

[dmikushin]

Hi @bryanpkc , thanks for your comments!

To address 1 and 3, I will burst my PR into 13 separate PRs. I ask you to kindly review and merge all of them that are not yet covered by a PR merged from your side. Plans and progress are great, but too uncertain due to the corporate security policies regarding opensource contributions. Then you can rebase your changes on top of my merges and cover all the rest. I think this approach is fair, and will help all of us to have greater motivation and faster progress in getting better Flang.

[dmikushin]

The Classic Flang runtime code is not intended to be compiled with LLVM Flang. The latter has its own runtime libraries. Why are you trying to do this?

Here is my logic in simple statements. Kindly please point me to the wrong statement in it:

1. Classic Flang requires a Fortran compiler to be compiled.
2. Another Fortran compiler needs to be deployed to compile Flang
3. There are only two compilers in the wild that could theoretically handle this task: gfortran and in-tree LLVM Flang
4. The build instructions do not explain how to proceed about this precisely, so I make my decision

Therefore, I choose to deploy in-tree LLVM Flang. What is wrong with my logic?

[bryanpkc]

Classic Flang requires a Fortran compiler to be compiled.

The Classic Flang compiler is written in C/C++; a pre-existing Fortran compiler is not needed.

The Classic Flang runtime contains some Fortran files, but by design they are compiled with the Classic Flang compiler that has just been built.

As you can see in our CI workflows, the Classic Flang build does not require any pre-existing Fortran compiler.

[bryanpkc]

The build instructions do not explain how to proceed about this precisely, so I make my decision

Can you show me your attempt to follow our build instructions (what commands you tried and what they produced)? We can try to help you figure out how to build Classic Flang properly.

[dmikushin]

Hi @bryanpkc ,

I can follow your CI process by pulling one of containers being used for GitHub workflow:

docker run --rm -it ghcr.io/flang-compiler/ubuntu20-flang-release_15x
cd /home/github/usr/local/bin
./flang --version
flang version 15.0.3 (https://github.com/flang-compiler/classic-flang-llvm-project.git cd736e11b188a8f6ff14041abd818ad86f36b9bb)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/github/usr/local/bin/.

As you can see above, the CI container has the in-tree LLVM Flang built and installed.

Next step: in build_flang.sh you have:

cmake $CMAKE_OPTIONS \
      -DCMAKE_Fortran_COMPILER=$INSTALL_PREFIX/bin/flang \
      -DCMAKE_Fortran_COMPILER_ID=Flang \

which clearly refers to the previously mentioned Flang.

From this I conclude the CI uses in-tree Flang to compile the classic Flang.

The Classic Flang compiler is written in C/C++; a pre-existing Fortran compiler is not needed.

Therefore this is not true. Could you please ask someone who wrote this CI to come here and explain to us what is going on.

[kiranchandramohan]

Hi @bryanpkc ,

I can follow your CI process by pulling one of containers being used for GitHub workflow:

docker run --rm -it ghcr.io/flang-compiler/ubuntu20-flang-release_15x
cd /home/github/usr/local/bin
./flang --version
flang version 15.0.3 (https://github.com/flang-compiler/classic-flang-llvm-project.git cd736e11b188a8f6ff14041abd818ad86f36b9bb)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/github/usr/local/bin/.

As you can see above, the CI container has the in-tree LLVM Flang built and installed.

@dmikushin When classic-flang-llvm-project is built, it creates a symlink flang that points to clang.

lrwxrwxrwx 1 github github         5 Apr 19 17:24 flang -> clang

If you invoke this flang on a file then you can clearly see that it will try to call flang1 from classic-flang (flang-compiler/flang).

github@f9a13fbacc25:~/usr/local/bin$ ./flang /home/github/s.f90   
clang-15: error: unable to execute command: Executable "flang1" doesn't exist!

Next step: in build_flang.sh you have:

cmake $CMAKE_OPTIONS \
      -DCMAKE_Fortran_COMPILER=$INSTALL_PREFIX/bin/flang \
      -DCMAKE_Fortran_COMPILER_ID=Flang \

which clearly refers to the previously mentioned Flang.

You can consult the page (https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_COMPILER_ID.html) for the Compiler IDs. You can see that Classic Flang uses the Flang ID and in-tree Flang uses LLVMFlang ID.

[dmikushin]

Hi @kiranchandramohan , thank you for explaining this. Since classic Flang is in the CI image, does your explanation still mean that classic Flang is used to compile the classic Flang itself?

[bryanpkc]

Hi @kiranchandramohan , thank you for explaining this. Since classic Flang is in the CI image, does your explanation still mean that classic Flang is used to compile the classic Flang itself?

Clang is used to build the Classic Flang frontend, and the newly built Classic Flang frontend is used to build the Classic Flang runtime libraries. The compiler in the Docker container saves us from rebuilding Clang from source for every Classic Flang pull request; the build process doesn't depend on the Fortran frontend inside the Docker container.

Next step: in build_flang.sh you have:

cmake $CMAKE_OPTIONS \
      -DCMAKE_Fortran_COMPILER=$INSTALL_PREFIX/bin/flang \
      -DCMAKE_Fortran_COMPILER_ID=Flang \

which clearly refers to the previously mentioned Flang.

Actually, this clearly refers to the newly built frontend ($INSTALL_PREFIX/bin/flang) instead of the old one in the Docker container (/home/github/usr/local/bin/flang). It might be hard to understand, but when cmake is invoked, $INSTALL_PREFIX/bin/flang is only a symlink to clang, and flang1 and flang2 do not yet exist, so $INSTALL_PREFIX/bin/flang cannot actually compile any Fortran program. It can only start compiling the runtime code after flang1 and flang2 have been built halfway through the process.

[dmikushin]

Thank you both, I've finally sorted out with the compilation. @bryanpkc , I propose to add the following paragraph to the Wiki page to describe the proper build process to avoid any further confusion:

Classic Flang build must follow the following sequence.

First, the Classic Flang LLVM Project must be build with -DLLVM_ENABLE_CLASSIC_FLANG=ON and -DLLVM_ENABLE_PROJECTS="clang" options. These options and patches to the mainline LLVM applied in Classic Flang LLVM Project will prepare a toolchain that could incorporate Classic Flang, when it is built. Specifically, options and patches add support for legacy PGI options such as -Mreentrant and call the flang1 and flang2 stages, when Fortran compiler is invoked. Note: this scenario completely replaces ("predates") the official LLVM Flang: it can't be enabled and used together with the Classic Flang.

Second, the Flang project must be built with the LLVM toolchain prepared above. CMake configuration must include flang as a Fortran compiler, which refers to the Classic Flang frontend enabled by the patches. This Fortran compiler is not yet usable, so we also provide -DCMAKE_Fortran_COMPILER_ID=Flang, which instructs CMake to assume Fortran compiler works without checking it. During the build, flang1 and flang2 executables will be created to get a working Flang frontend, and then this frontend will be deployed to compile runtime libraries in Fortran language.

If you agree, please add it to the Wiki page.

[dmikushin]

Hello @bryanpkc , I'm waiting for the clarification above to be added to the Wiki page to close this PR. I can't edit the Wiki myself.

[kiranchandramohan]

I am supportive of improvements to the Wiki page. Particularly, clarifying the relation with llvm/flang. I have made a couple of edits.

I am not sure of a verbatim copy of the writeup that @dmikushin provided. Some of the material is already present on the wiki page, like the dependency on the classic-flang-llvm-project and that it supports additional flags etc. Going too much into the internals (existence of flang1, flang2, Mreentrant flag etc) is probably not required for people who might just want to build the project.

[dmikushin]

@kiranchandramohan , I think one critical misunderstanding the current Wiki page still creates is that there should exist some Fortran compiler, prior to the build of Flang. By reading the updated text I would still make the same mistake.

My detail about flang1 and flang2 is proposed to emphasize the fact that initially we only enable support for Fortran compiler in LLVM, and that $INSTALL_PREFIX/bin/flang is not yet a real usable Fortran compiler.

[bryanpkc]

Hi @dmikushin, I have edited the Building Flang wiki page again. Is this more understandable?

[bryanpkc]

This PR will not be accepted as-is; closing.