Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross compilation support for the libraries? #1202

Open
meow464 opened this issue Dec 23, 2021 · 44 comments
Open

Cross compilation support for the libraries? #1202

meow464 opened this issue Dec 23, 2021 · 44 comments

Comments

@meow464
Copy link

meow464 commented Dec 23, 2021

Is there cross compilation support? I only need the libraries, not the compiler.
No matter what parameters I try it keeps building for the host architecture (x86_64).

@bryanpkc
Copy link
Collaborator

What architecture do you want to cross-compile to?

@meow464
Copy link
Author

meow464 commented Dec 23, 2021

AArch64 (iOS)

@bryanpkc
Copy link
Collaborator

The current code base does not support cross-compilation of the frontend and the libraries, however we have implemented such a mechanism downstream. I'll see if I can clean things up and submit a PR with the change.

@meow464
Copy link
Author

meow464 commented Dec 24, 2021

That would be amazing!

@kaadam
Copy link
Collaborator

kaadam commented Dec 29, 2021

@bryanpkc I have a POC for enable cross-compilation in CMakefiles what I did some weeks ago [1], if you and the community are interested in, I can upload that.
[1] kaadam@a89ecd1

@meow464
Copy link
Author

meow464 commented Dec 31, 2021

Thanks, I will take a look at that too!

@meow464
Copy link
Author

meow464 commented Jan 2, 2022

This is what I'm doing and the error I'm getting.

build.sh (for libpgmath only)

# if [[ ! -d flang ]]; then                                                                                                                                                                                          
#     git clone -b macos https://github.com/isuruf/flang.git                                                                                                                                                         
# fi                                                                                                                                                                                                                 

export SDKROOT=$(xcrun --sdk iphoneos --show-sdk-path)

##########################################################                                                                                                                                                           
# This might not be needed. Investigate how to build only                                                                                                                                                            
# the flang runtime and not the compiler.                                                                                                                                                                            
# Will probably need to edit CMake files.                                                                                                                                                                            

FLAGS="-target arm64-apple-ios -miphoneos-version-min=9.0"
export CFLAGS=$FLAGS
export CXXFLAGS=$FLAGS
export FFLAGS=$FLAGS

CMAKE_OPTIONS=$CMAKE_OPTIONS" -DLLVM_FLANG_CPU_TARGET=aarch64 \                                                                                                                                                      
    -DLIBPGMATH_SYSTEM_PROCESSOR=aarch64"

##########################################################                                                                                                                                                           

(cd flang_iphonesimulator/runtime/libpgmath
 mkdir -p build_iphoneos && cd build_iphoneos
 cmake $CMAKE_OPTIONS ..
 make
 sudo make install)
setup.sh
INSTALL_PREFIX=/usr/local/stow/flang

# Targets to build should be one of: X86 PowerPC AArch64                                                                                                                                                             
CMAKE_OPTIONS="-DCMAKE_INSTALL_PREFIX=$INSTALL_PREFIX \                                                                                                                                                              
    -DLLVM_CONFIG=$INSTALL_PREFIX/bin/llvm-config \                                                                                                                                                                  
    -DCMAKE_CXX_COMPILER=$INSTALL_PREFIX/bin/clang++ \                                                                                                                                                               
    -DCMAKE_C_COMPILER=$INSTALL_PREFIX/bin/clang \                                                                                                                                                                   
    -DCMAKE_Fortran_COMPILER=$INSTALL_PREFIX/bin/flang \                                                                                                                                                             
    -DCMAKE_Fortran_COMPILER_ID=Flang \                                                                                                                                                                              
    -DCMAKE_BUILD_TYPE=Release \                                                                                                                                                                                     
    -DLLVM_TARGETS_TO_BUILD=aarch64"
The error

Note: the INSTALL_PREFIX folder has the name of the wrong architecture, but it's just the name.
The repo is was cloned into flang_iphonesimulator but I made no changes to it. It's just the name that is wrong.

-DCMAKE_INSTALL_PREFIX=/opt/flang/x86_64-apple-ios -DLLVM_TARGETS_TO_BUILD=AArch64 -DLLVM_TARGET_ARCH=AArch64 -DLLVM_DEFAULT_TARGET_TRIPLE=arm64-apple-ios -DCMAKE_OSX_SYSROOT=/Applications/Xcode.app/Contents/Devel
oper/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk -DLLVM_CONFIG=/usr/local/stow/flang/bin/llvm-config -DCMAKE_CXX_COMPILER=/usr/local/stow/flang/bin/clang++ -DLLVM_TABLEGEN=/usr/local/stow/flang/llvm-tblgen -DCLANG_TABLEGEN=/usr/local/stow/flang/clang-tblgen
-- The C compiler identification is AppleClang 12.0.5.12050022
-- The CXX compiler identification is AppleClang 12.0.5.12050022
-- The ASM compiler identification is Clang
-- Found assembler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - broken
CMake Error at /usr/local/Cellar/cmake/3.21.4/share/cmake/Modules/CMakeTestCCompiler.cmake:69 (message):
  The C compiler

    "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /Volumes/macos_extra/flang_iphonesimulator/runtime/libpgmath/build/CMakeFiles/CMakeTmp
    
    Run Build Command(s):/usr/bin/make -f Makefile cmTC_eb46e/fast && /Applications/Xcode.app/Contents/Developer/usr/bin/make  -f CMakeFiles/cmTC_eb46e.dir/build.make CMakeFiles/cmTC_eb46e.dir/build
    Building C object CMakeFiles/cmTC_eb46e.dir/testCCompiler.c.o
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc   -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk -MD -MT CMakeFiles/cmTC_eb46e.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_eb46e.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_eb46e.dir/testCCompiler.c.o -c /Volumes/macos_extra/flang_iphonesimulator/runtime/libpgmath/build/CMakeFiles/CMakeTmp/testCCompiler.c
    Linking C executable cmTC_eb46e
    /usr/local/Cellar/cmake/3.21.4/bin/cmake -E cmake_link_script CMakeFiles/cmTC_eb46e.dir/link.txt --verbose=1
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/cmTC_eb46e.dir/testCCompiler.c.o -o cmTC_eb46e 
    ld: building for iOS, but linking in object file built for iOS Simulator, file 'CMakeFiles/cmTC_eb46e.dir/testCCompiler.c.o' for architecture x86_64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    make[1]: *** [cmTC_eb46e] Error 1
    make: *** [cmTC_eb46e/fast] Error 2
    
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:30 (project)


-- Configuring incomplete, errors occurred!
See also "/Volumes/macos_extra/flang_iphonesimulator/runtime/libpgmath/build/CMakeFiles/CMakeOutput.log".
See also "/Volumes/macos_extra/flang_iphonesimulator/runtime/libpgmath/build/CMakeFiles/CMakeError.log".
make: *** No targets specified and no makefile found.  Stop.
make: *** No rule to make target `install'.  Stop.

As you can see CMake keeps trying to build for the triple x86_64-apple-ios (the iphone simulator on a x86 host)

EDIT: To clarify, that's not on kaadam's fork.

@meow464
Copy link
Author

meow464 commented Jan 8, 2022

I managed to make libpgmath target arch64 by commenting out two lines in flang/runtime/libpgmath/CMakeLists.txt:

# set(LIBPGMATH_SYSTEM_NAME "${CMAKE_SYSTEM_NAME}")
# set(LIBPGMATH_SYSTEM_PROCESSOR "${CMAKE_SYSTEM_PROCESSOR}")

Now I'm getting:

[  0%] Building C object lib/aarch64/tanh/CMakeFiles/aarch64_tanh.dir/gstanh4.c.o
clang-10: error: the clang compiler does not support '-mcpu=aarch64'

I've tried setting compiler flags with -DCMAKE_{C,CXX,Fortran}_FLAGS="-target arm64-apple-ios -miphoneos-version-min=9.0 -mcpu=generic" to no avail.

clang -target arm64-apple-ios -print-supported-cpus
clang version 10.0.1 (https://github.com/flang-compiler/classic-flang-llvm-project.git 4a566ddfa4157ed1a24920ab7673ccc01c46fd99)
Target: arm64-apple-ios
Thread model: posix
InstalledDir: /usr/local/bin
Available CPUs for this target:

        apple-a10
        apple-a11
        apple-a12
        apple-a13
        apple-a7
        apple-a8
        apple-a9
        apple-latest
        apple-s4
        apple-s5
        cortex-a35
        cortex-a53
        cortex-a55
        cortex-a57
        cortex-a65
        cortex-a65ae
        cortex-a72
        cortex-a73
        cortex-a75
        cortex-a76
        cortex-a76ae
        cyclone
        exynos-m3
        exynos-m4
        exynos-m5
        falkor
        generic
        kryo
        neoverse-e1
        neoverse-n1
        saphira
        thunderx
        thunderx2t99
        thunderx3t110
        thunderxt81
        thunderxt83
        thunderxt88
        tsv110

Use -mcpu or -mtune to specify the target's processor.
For example, clang --target=aarch64-unknown-linux-gui -mcpu=cortex-a35

Any help is appreciated.

@meow464
Copy link
Author

meow464 commented Jan 10, 2022

I fixed the problem above but now there are more problems. Please let me know if something is not clear.

The error
Consolidate compiler generated dependencies of target aarch64_tanh                                                                                                                                                   
[  1%] Built target aarch64_tanh                                                                                                                                                                                     
[  1%] Generating mth_128mask.cpp                                                                                                                                                                                    
[  1%] Built target mth_128mask                                                                           
Consolidate compiler generated dependencies of target mth_128mask_build                                                                                                                                              
[  1%] Building CXX object lib/common/CMakeFiles/mth_128mask_build.dir/mth_128mask.cpp.o                                                                                                                             
In file included from /Volumes/macos_extra/iOS/flang/runtime/libpgmath/build_iphoneos/lib/common/mth_128mask.cpp:17:                                                                                                 
In file included from /Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/mth_intrinsics.h:14:    
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/complex.h:28:                                                        
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/ccomplex:20:              
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/complex:243:                                                         
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/type_traits:417:                                                     
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:56:1: error: templates must have C++ linkage                                       
template <class _Tp> struct __libcpp_is_integral                     { enum { value = 0 }; };
^~~~~~~~~~~~~~~~~~~~                                                                                      
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/build_iphoneos/lib/common/mth_128mask.cpp:14:1: note: extern "C" language linkage specification begins here                                                         
extern "C" {                                                                                                                                                                                                         
^                                                                                                         
In file included from /Volumes/macos_extra/iOS/flang/runtime/libpgmath/build_iphoneos/lib/common/mth_128mask.cpp:17:                                                                                                 
In file included from /Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/mth_intrinsics.h:14:                                                                                                               
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/complex.h:28:                                                        
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/ccomplex:20:                                                         
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/complex:243:                                                         
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/type_traits:417:          
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:57:29: error: explicit specialization of undeclared template struct                
      '__libcpp_is_integral'                                                                                                                                                                                         
template <>          struct __libcpp_is_integral<bool>               { enum { value = 1 }; };                                                                                                                        
                            ^                   ~~~~~~                                       
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:58:29: error: explicit specialization of non-template struct '__libcpp_is_integral'
template <>          struct __libcpp_is_integral<char>               { enum { value = 1 }; };                                                                                                                        
                            ^                   ~~~~~~                                                                                                                                                               
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:58:29: error: redefinition of '__libcpp_is_integral'                               
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:57:29: note: previous definition is here                                           
template <>          struct __libcpp_is_integral<bool>               { enum { value = 1 }; };                                                                                                                        
                            ^                                                                             
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:59:29: error: explicit specialization of non-template struct '__libcpp_is_integral'
template <>          struct __libcpp_is_integral<signed char>        { enum { value = 1 }; };                                                                                                                        
                            ^                   ~~~~~~~~~~~~~                                                                                                                                                        
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:59:29: error: redefinition of '__libcpp_is_integral'
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:57:29: note: previous definition is here                                           
template <>          struct __libcpp_is_integral<bool>               { enum { value = 1 }; };                                                                                                                        
                            ^                                                                             
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:60:29: error: explicit specialization of non-template struct '__libcpp_is_integral'
template <>          struct __libcpp_is_integral<unsigned char>      { enum { value = 1 }; };             
                            ^                   ~~~~~~~~~~~~~~~                                           
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:60:29: error: redefinition of '__libcpp_is_integral'                               
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:57:29: note: previous definition is here                                           
template <>          struct __libcpp_is_integral<bool>               { enum { value = 1 }; }; 
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:61:29: error: explicit specialization of non-template struct '__libcpp_is_integral'
template <>          struct __libcpp_is_integral<wchar_t>            { enum { value = 1 }; };                                                                                                                        
                            ^                   ~~~~~~~~~                                                 
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:61:29: error: redefinition of '__libcpp_is_integral'                               
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:57:29: note: previous definition is here                                           
template <>          struct __libcpp_is_integral<bool>               { enum { value = 1 }; };                                                                                                                        
                            ^                                                                             
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:66:29: error: explicit specialization of non-template struct '__libcpp_is_integral'
template <>          struct __libcpp_is_integral<char16_t>           { enum { value = 1 }; };                                                                                                                        
                            ^                   ~~~~~~~~~~                                                                                                                                                           
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:66:29: error: redefinition of '__libcpp_is_integral'                               
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:57:29: note: previous definition is here                                           
template <>          struct __libcpp_is_integral<bool>               { enum { value = 1 }; };
                            ^                                                                             
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:67:29: error: explicit specialization of non-template struct '__libcpp_is_integral'
template <>          struct __libcpp_is_integral<char32_t>           { enum { value = 1 }; };                                                                                                                        
                            ^                   ~~~~~~~~~~                                                
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:67:29: error: redefinition of '__libcpp_is_integral'                               
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:57:29: note: previous definition is here                                           
template <>          struct __libcpp_is_integral<bool>               { enum { value = 1 }; };                                                                                                                        
                            ^                                                                                                                                                                                        
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:69:29: error: explicit specialization of non-template struct '__libcpp_is_integral'
template <>          struct __libcpp_is_integral<short>              { enum { value = 1 }; };                                                                                                                        
                            ^                   ~~~~~~~                                                                                                                                                              
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:69:29: error: redefinition of '__libcpp_is_integral'                               
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:57:29: note: previous definition is here                                           
template <>          struct __libcpp_is_integral<bool>               { enum { value = 1 }; };
                            ^                                                                                                                                                                                        
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:70:29: error: explicit specialization of non-template struct '__libcpp_is_integral'
template <>          struct __libcpp_is_integral<unsigned short>     { enum { value = 1 }; };                                                                                                                        
                            ^                   ~~~~~~~~~~~~~~~~                                                                                                                                                     
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:70:29: error: redefinition of '__libcpp_is_integral'                               
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:57:29: note: previous definition is here                                           
template <>          struct __libcpp_is_integral<bool>               { enum { value = 1 }; };             
                            ^                                                                                                                                                                                        
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/c++/v1/cstddef:71:29: error: explicit specialization of non-template struct '__libcpp_is_integral'
template <>          struct __libcpp_is_integral<int>                { enum { value = 1 }; };                                                                                                                        
                            ^                   ~~~~~                                                                                                                                                                
fatal error: too many errors emitted, stopping now [-ferror-limit=]                                                                                                                                                  
20 errors generated.                                                                                                                                                                                                 
make[2]: *** [lib/common/CMakeFiles/mth_128mask_build.dir/mth_128mask.cpp.o] Error 1                      
make[1]: *** [lib/common/CMakeFiles/mth_128mask_build.dir/all] Error 2                                                                                                                                               
make: *** [all] Error 2

The hypothesis:

  1. The file mth_128mask.cpp is generated from runtime/libpgmath/tools/mth_mask.awk. The top of mth_128mask.cpp looks like this:
#ifdef __cplusplus
extern "C" {
#endif

#include "mth_intrinsics.h" // Note how this file is included inside an extern C block
#include "mth_tbldefs.h"


static const vrs4_t Csp1_4={1.0, 1.0, 1.0, 1.0};
static const vrd2_t Cdp1_2={1.0, 1.0};
static const vrs8_t Csp1_8={1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0};
static const vrd4_t Cdp1_4={1.0, 1.0, 1.0, 1.0};
static const vrs16_t Csp1_16={1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,
                             1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0};
static const vrd8_t Cdp1_8={1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0};


#ifdef __cplusplus
}
#endif

The top of mth_intrinsics.h looks like this:

#ifdef __cplusplus
#include <cstdint>
#else
#include <stdint.h>
#endif
  1. Since mth_128mask.cpp is a cpp file, #ifdef __cplusplus is also triggered in the included mth_intrinsics.h file. We end up including a cpp file cstdint inside an extern C. This shouldn't happen, of course.

So far so good. If I move #include "mth_intrinsics.h" to outside the extern C block everything works. But a question arises: Why am I the first one hitting this problem?

Only in the ARM pathway mth_128mask.cpp is generated, for x86 and power it's mth_128mask.c, meaning #ifdef __cplusplus is never triggered. This explains why I didn't find this on MacOS.
But why didn't anyone find it on any other ARM platform? It seems this behaviour should be triggered on any ARM platform. I don't have an ARM machine so I can't test.

The relevant part of runtime/libpgmath/lib/common/CMakeLists.txt looks like this:

elseif(${LIBPGMATH_SYSTEM_PROCESSOR} MATCHES "aarch64")
  # Generate mth_128mask.cpp
  set(TARGET_NAME "mth_128mask")
  if(${LIBPGMATH_SYSTEM_NAME} MATCHES "Windows")
    add_custom_command(OUTPUT ${TARGET_NAME}.cpp DEPENDS ${LIBPGMATH_TOOLS_DIR}/mth_mask.awk PRE_BUILD
      COMMAND awk -v MAX_VREG_SIZE=128 -v TARGET=ARM64 -v TARGET_OS=WIN -f ${LIBPGMATH_TOOLS_DIR}/mth_mask.awk > ${TARGET_NAME}.cpp)
  else()
    add_custom_command(OUTPUT ${TARGET_NAME}.cpp DEPENDS ${LIBPGMATH_TOOLS_DIR}/mth_mask.awk PRE_BUILD
      COMMAND awk -v MAX_VREG_SIZE=128 -v TARGET=ARM64 -f ${LIBPGMATH_TOOLS_DIR}/mth_mask.awk > ${TARGET_NAME}.cpp)
  endif()
  add_custom_target(${TARGET_NAME} ALL DEPENDS "${TARGET_NAME}.cpp")
  libmath_add_object_library("${TARGET_NAME}.cpp" "${FLAGS}" "${DEFINITIONS}" "${TARGET_NAME}_build")
  add_dependencies("${TARGET_NAME}_build" ${TARGET_NAME})

  # Generate mth_128generic.c
  set(TARGET_NAME "mth_128generic")
  add_custom_command(OUTPUT ${TARGET_NAME}.c DEPENDS ${LIBPGMATH_TOOLS_DIR}/mth_generic_frp.awk PRE_BUILD
    COMMAND awk -v MAX_VREG_SIZE=128 -v TARGET=ARM64 -f ${LIBPGMATH_TOOLS_DIR}/mth_generic_frp.awk > ${TARGET_NAME}.c)
  add_custom_target(${TARGET_NAME} ALL DEPENDS "${TARGET_NAME}.c")
  libmath_add_object_library("${TARGET_NAME}.c" "${FLAGS}" "${DEFINITIONS}" "${TARGET_NAME}_build")
  add_dependencies("${TARGET_NAME}_build" ${TARGET_NAME})

@meow464
Copy link
Author

meow464 commented Jan 11, 2022

Problem above is "fixed" but now I get:

fltfenv.c error
-- Build files have been written to: /Volumes/macos_extra/iOS/flang/runtime/libpgmath/build_iphoneos
Consolidate compiler generated dependencies of target aarch64_tanh
[  1%] Built target aarch64_tanh
[  1%] Built target mth_128mask
Consolidate compiler generated dependencies of target mth_128mask_build
[  2%] Built target mth_128mask_build
[  2%] Built target mth_128generic
Consolidate compiler generated dependencies of target mth_128generic_build
[  3%] Built target mth_128generic_build
Consolidate compiler generated dependencies of target common_mth_cmplx
[ 16%] Built target common_mth_cmplx
[ 17%] Built target tmp-mth_alldefs
[ 18%] Built target tmp-mth_statsdefs
Consolidate compiler generated dependencies of target dispatch
[ 18%] Built target dispatch
Consolidate compiler generated dependencies of target common
[ 19%] Building C object lib/common/CMakeFiles/common.dir/fltfenv.c.o
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltfenv.c:537:10: error: implicit declaration of function 'feenableexcept' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  return feenableexcept(exc);
         ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltfenv.c:537:10: note: did you mean 'feraiseexcept'?
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/fenv.h:299:12: note: 'feraiseexcept' declared here
extern int feraiseexcept(int /* excepts */);
           ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltfenv.c:543:10: error: implicit declaration of function 'fedisableexcept' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  return fedisableexcept(exc);
         ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltfenv.c:543:10: note: did you mean 'feraiseexcept'?
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/fenv.h:299:12: note: 'feraiseexcept' declared here
extern int feraiseexcept(int /* excepts */);
           ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltfenv.c:549:10: error: implicit declaration of function 'fegetexcept' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  return fegetexcept();
         ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltfenv.c:549:10: note: did you mean 'fetestexcept'?
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/usr/include/fenv.h:301:12: note: 'fetestexcept' declared here
extern int fetestexcept(int /* excepts */);
           ^
3 errors generated.

Here is the relevant #ifdef in fltfenv.c

#else   /* #if (defined(TARGET_X8664) || defined(TARGET_X86) || defined(X86)) */

/*
 * aarch64 and POWER (not X86-64).
 *
 * Without loss of generality, use libc's implemenations of floating point
 * control/status get/set operations.
 */

#include <fenv.h>

But the functions in the error: feenableexcept, fedisableexcept and fegetexcept are GNU extensions and obviously not available on iOS or Windows.
This was taken into account for windows but the #if defined(TARGET_WIN_ARM64) assumes the #else case is always GNU and tries to the libc implementation.

#if defined(TARGET_WIN_ARM64) in fltfenv.c

#if defined(TARGET_WIN_ARM64)
/* TODO: Implement and test these functions */
int
__fenv_feenableexcept(int exc)
{
  // implement function.
  return 0;
}

int
__fenv_fedisableexcept(int exc)
{
  // implement function.
  return 0;
}

int
__fenv_fegetexcept(void)
{
  // implement function.
  return 0;
}

#else
int
__fenv_feenableexcept(int exc)
{
  return feenableexcept(exc);
}

int
__fenv_fedisableexcept(int exc)
{
  return fedisableexcept(exc);
}

int
__fenv_fegetexcept(void)
{
  return fegetexcept();
}
#endif

Fortunately FreeBSD implemennts all thoses three functions. freebsd/aarch64/fenv.h freebsd/aarch64/fenv.c

My questions are:

  1. Simply leaving the functions not implemented like on windows seems like a bad idea, right?
  2. Is it ok to mix 2-clause BSD code with LLVM code?
  3. How should I go about including the FreeBSD implementation? Should I make another file to preserve the license header?
  4. Isn't fltfenv.c becoming a bit too platform dependent for something in the common directory? It already implements the x86 version itself.

@meow464
Copy link
Author

meow464 commented Jan 14, 2022

I implemented the missing functions above myself.

When trying to build libpgmath for iOS I'm getting a bunch of warnings that I will fix afterwards and an error.

libpgmath error
[ 48%] Building C object lib/common/CMakeFiles/common.dir/floor.c.o
[ 49%] Building C object lib/common/CMakeFiles/common.dir/fltfenv.c.o
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltfenv.c:470:14: warning: pragma STDC FENV_ACCESS ON is not supported, ignoring pragma [-Wunknown-pragmas]
#pragma STDC FENV_ACCESS ON
             ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltfenv.c:538:14: warning: pragma STDC FENV_ACCESS ON is not supported, ignoring pragma [-Wunknown-pragmas]
#pragma STDC FENV_ACCESS ON
             ^
2 warnings generated.
[ 49%] Building C object lib/common/CMakeFiles/common.dir/fltmanip.c.o
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:129:42: warning: magnitude of floating-point constant too large for type 'long double'; maximum is 1.7976931348623157E+308 [-Wliteral-range]
        return (int)(ixh >> GET_SIGN) ? -_MAXLONGDOUBLE : _MAXLONGDOUBLE;
                                         ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:83:25: note: expanded from macro '_MAXLONGDOUBLE'
#define _MAXLONGDOUBLE (1.18973149535723176508575932662800702E+4932L)
                        ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:129:59: warning: magnitude of floating-point constant too large for type 'long double'; maximum is 1.7976931348623157E+308 [-Wliteral-range]
        return (int)(ixh >> GET_SIGN) ? -_MAXLONGDOUBLE : _MAXLONGDOUBLE;
                                                          ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:83:25: note: expanded from macro '_MAXLONGDOUBLE'
#define _MAXLONGDOUBLE (1.18973149535723176508575932662800702E+4932L)
                        ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:134:14: warning: magnitude of floating-point constant too large for type 'long double'; maximum is 1.7976931348623157E+308 [-Wliteral-range]
      return _MAXLONGDOUBLE;
             ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:83:25: note: expanded from macro '_MAXLONGDOUBLE'
#define _MAXLONGDOUBLE (1.18973149535723176508575932662800702E+4932L)
                        ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:136:13: warning: magnitude of floating-point constant too large for type 'long double'; maximum is 1.7976931348623157E+308 [-Wliteral-range]
    return -_MAXLONGDOUBLE;
            ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:83:25: note: expanded from macro '_MAXLONGDOUBLE'
#define _MAXLONGDOUBLE (1.18973149535723176508575932662800702E+4932L)
                        ^
4 warnings generated.
[ 49%] Building C object lib/common/CMakeFiles/common.dir/fpcvt.c.o
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fpcvt.c:421:5: error: implicit declaration of function '__io_set_errno' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    __io_set_errno(ERANGE);
    ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fpcvt.c:427:5: error: implicit declaration of function '__io_set_errno' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    __io_set_errno(ERANGE);
    ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fpcvt.c:432:7: error: implicit declaration of function '__io_set_errno' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
      __io_set_errno(ERANGE);
      ^
3 errors generated.
make[2]: *** [lib/common/CMakeFiles/common.dir/fpcvt.c.o] Error 1
make[1]: *** [lib/common/CMakeFiles/common.dir/all] Error 2
make: *** [all] Error 2

__io_set_errno is declared in runtime/include/stdioInterf.h and defined in runtime/flangrti/iostdinit.c. It used to be included indirectly by pgstdinit.h but this file has been deleted.

The file fpcvt.c where the error is raised is included for all architectures but but only causing problems here. I'm not able to figure out why.

@meow464
Copy link
Author

meow464 commented Jan 15, 2022

For now I simply declared the prototype for the function above.

The next error is again a bit weird. It seems it should be triggered on any arm64 platform.

error
[ 70%] Building CXX object lib/common/sincos/CMakeFiles/gdsincos2.dir/gsincos.cpp.o
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/sincos/gsincos.cpp:82:5: error: use of undeclared identifier 'sincosf'; did you mean 'sincos'?
    SINCOS(x[i], &ts[i], &tc[i]);
    ^~~~~~
    sincos
<command line>:7:16: note: expanded from here
#define SINCOS sincosf
               ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/sincos/gsincos.cpp:71:17: note: 'sincos' declared here
extern "C" void sincos(double, double*, double*) throw();
                ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/sincos/gsincos.cpp:82:18: error: cannot initialize a parameter of type 'double *' with an rvalue of type 'float *'
    SINCOS(x[i], &ts[i], &tc[i]);
                 ^~~~~~
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/sincos/gsincos.cpp:71:39: note: passing argument to parameter here
extern "C" void sincos(double, double*, double*) throw();
                                      ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/sincos/gsincos.cpp:96:9: error: use of undeclared identifier 'sincosf'; did you mean 'sincos'?
        SINCOS(x[i], &ts[i], &tc[i]);
        ^~~~~~
        sincos
<command line>:7:16: note: expanded from here
#define SINCOS sincosf
               ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/sincos/gsincos.cpp:71:17: note: 'sincos' declared here
extern "C" void sincos(double, double*, double*) throw();
                ^
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/sincos/gsincos.cpp:96:22: error: cannot initialize a parameter of type 'double *' with an rvalue of type 'float *'
        SINCOS(x[i], &ts[i], &tc[i]);
                     ^~~~~~
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/sincos/gsincos.cpp:71:39: note: passing argument to parameter here
extern "C" void sincos(double, double*, double*) throw();
                                      ^
4 errors generated.
make[2]: *** [lib/common/sincos/CMakeFiles/gdsincos2.dir/gsincos.cpp.o] Error 1
make[1]: *** [lib/common/sincos/CMakeFiles/gdsincos2.dir/all] Error 2
make: *** [all] Error 2

The error is on the file runtime/libpmath/lib/common/sincos/gsincos.cpp. As you can see the function SINCOS(VFLOAT, FLOAT, FLOAT) is missing and nowhere to be found in the code base. But gsincos.cpp does have the suspiciously familiar prototype extern "C" void sincos(double, double*, double*) throw();.

It turns out gsincos.cpp is only used for arm64. x86 and power use the very similar file gsincos.c which does have the same call to SINCOS(VFLOAT, FLOAT, FLOAT) except the familiar prototype from above is extern void SINCOS(FLOAT, FLOAT *, FLOAT *); (on the exact same line number too).

@kiranchandramohan
Copy link
Collaborator

Good to see that you are making progress here. Is sincos expected to come from libm?

@meow464
Copy link
Author

meow464 commented Jan 16, 2022

sincos should be available on math.h, yes. But to be honest I don't fully understand all the code in runtime/libpggmath/lib/common/sincos/ yet.

As for the overall status: I can build libpgmath but: 1. some targets are still missing the arm64 compiler flags and thus built for x86 (the host) and; 2. there are some warnings about long double to double implicit conversion on arm64.

@d-parks
Copy link

d-parks commented Jan 16, 2022

WRT SINCOS not being defined:
The error is on the file runtime/libpmath/lib/common/sincos/gsincos.cpp. As you can see the function SINCOS(VFLOAT, FLOAT, FLOAT) is missing and nowhere to be found in the code base

SINCOS is a CPP object macro that is defined in CMakeLists.txt.

@meow464
Copy link
Author

meow464 commented Jan 16, 2022

I completely missed that, thanks! I only needed to add the prototype for sincosf(float, float*, float*).

@meow464
Copy link
Author

meow464 commented Jan 17, 2022

I'm getting a bunch of implicit long double -> double conversion warnings:

warnings
[ 50%] Building C object lib/common/CMakeFiles/common.dir/fltmanip.c.o                                    
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:129:42: warning: magnitude of floating-point constant too large for type 'long double'; maximum is 1.7976931348623157E+308 [-Wliteral-range]  
        return (int)(ixh >> GET_SIGN) ? -_MAXLONGDOUBLE : _MAXLONGDOUBLE;                                 
                                         ^                                                                
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:83:25: note: expanded from macro '_MAXLONGDOUBLE'                                                                                             
#define _MAXLONGDOUBLE (1.18973149535723176508575932662800702E+4932L)                                                                                                                                                
                        ^                                                                                                                                                                                            
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:129:59: warning: magnitude of floating-point constant too large for type 'long double'; maximum is 1.7976931348623157E+308 [-Wliteral-range]  
        return (int)(ixh >> GET_SIGN) ? -_MAXLONGDOUBLE : _MAXLONGDOUBLE;                                 
                                                          ^                                                                                                                                                          
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:83:25: note: expanded from macro '_MAXLONGDOUBLE'                                                                                             
#define _MAXLONGDOUBLE (1.18973149535723176508575932662800702E+4932L)                                     
                        ^                                                                                                                                                                                            
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:134:14: warning: magnitude of floating-point constant too large for type 'long double'; maximum is 1.7976931348623157E+308 [-Wliteral-range]  
      return _MAXLONGDOUBLE;                                                                              
             ^                                                                                            
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:83:25: note: expanded from macro '_MAXLONGDOUBLE'                                                                                             
#define _MAXLONGDOUBLE (1.18973149535723176508575932662800702E+4932L)                                     
                        ^                                                                                 
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:136:13: warning: magnitude of floating-point constant too large for type 'long double'; maximum is 1.7976931348623157E+308 [-Wliteral-range]  
    return -_MAXLONGDOUBLE;                                                                                                                                                                                          
            ^                                                                                                                                                                                                        
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/fltmanip.c:83:25: note: expanded from macro '_MAXLONGDOUBLE'                                                                                             
#define _MAXLONGDOUBLE (1.18973149535723176508575932662800702E+4932L)

It seems that on apple long double and double are the same type even if arm64 theoretically supports float128. Ref1 Ref2. For now I simply disabled TARGET_SUUPPORTS_QUADFP, please let me know if this is not the correct course of action.

@meow464
Copy link
Author

meow464 commented Jan 18, 2022

I'm still getting quite a few warnings about user defined types in C-linkage functions.

Looking at their code I'm not sure if this is really a problem. They are all structs with operator overloads and constructors, which in principles C code wouldn't even know how to call.

The warnings.
[ 65%] Building CXX object lib/common/acos/fma3/CMakeFiles/common_acos_fma3.dir/vdacos2.cpp.o             
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/acos/fma3/vdacos2.cpp:19:20: warning: '__fvd_acos_fma3' has C-linkage specified, but returns user-defined type '__m128d' (aka '__s128d') which is        
      incompatible with C [-Wreturn-type-c-linkage]                                                       
extern "C" __m128d __fvd_acos_fma3(__m128d);                                                              
                   ^                                                                                      
1 warning generated.                                                                                      
[ 65%] Building CXX object lib/common/acos/fma3/CMakeFiles/common_acos_fma3.dir/vsacos4.cpp.o
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/acos/fma3/vsacos4.cpp:18:19: warning: '__fvs_acos_fma3' has C-linkage specified, but returns user-defined type '__m128' (aka '__s128f') which is         
      incompatible with C [-Wreturn-type-c-linkage]                                                       
extern "C" __m128 __fvs_acos_fma3(__m128 const a);                                                        
                  ^                                                                                       
1 warning generated.                                                                                      
[ 65%] Built target common_acos_fma3                                                                      
[ 66%] Building CXX object lib/common/asin/fma3/CMakeFiles/common_asin_fma3.dir/sdasin.cpp.o
[ 66%] Building CXX object lib/common/asin/fma3/CMakeFiles/common_asin_fma3.dir/ssasin.cpp.o              
[ 67%] Building CXX object lib/common/asin/fma3/CMakeFiles/common_asin_fma3.dir/vdasin2.cpp.o
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/asin/fma3/vdasin2.cpp:19:20: warning: '__fvd_asin_fma3' has C-linkage specified, but returns user-defined type '__m128d' (aka '__s128d') which is        
      incompatible with C [-Wreturn-type-c-linkage]                                                       
extern "C" __m128d __fvd_asin_fma3(__m128d);                                                              
                   ^                                                                                      
1 warning generated.                                                                                      
[ 67%] Building CXX object lib/common/asin/fma3/CMakeFiles/common_asin_fma3.dir/vsasin4.cpp.o
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/asin/fma3/vsasin4.cpp:20:19: warning: '__fvs_asin_fma3' has C-linkage specified, but returns user-defined type '__m128' (aka '__s128f') which is         
      incompatible with C [-Wreturn-type-c-linkage]                                                       
extern "C" __m128 __fvs_asin_fma3(__m128 const a);                                                        
                  ^                                                                                       
1 warning generated.                                                                                      
[ 67%] Built target common_asin_fma3                                                                      
[ 67%] Building C object lib/common/exp/CMakeFiles/common_exp.dir/gsexp4.c.o        
[ 68%] Building C object lib/common/exp/CMakeFiles/common_exp.dir/gdexp2.c.o                              
[ 68%] Built target common_exp                                                                            
[ 68%] Building CXX object lib/common/exp/fma3/CMakeFiles/common_exp_fma3.dir/ssexp.cpp.o                 
[ 69%] Building CXX object lib/common/exp/fma3/CMakeFiles/common_exp_fma3.dir/sdexp.cpp.o
[ 69%] Building CXX object lib/common/exp/fma3/CMakeFiles/common_exp_fma3.dir/vdexp2.cpp.o
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/exp/fma3/vdexp2.cpp:19:20: warning: '__fvd_exp_fma3' has C-linkage specified, but returns user-defined type '__m128d' (aka '__s128d') which is           
      incompatible with C [-Wreturn-type-c-linkage]                                                       
extern "C" __m128d __fvd_exp_fma3(__m128d);                                                               
                   ^                                                                                      
1 warning generated.
[ 69%] Building CXX object lib/common/exp/fma3/CMakeFiles/common_exp_fma3.dir/vsexp4.cpp.o
/Volumes/macos_extra/iOS/flang/runtime/libpgmath/lib/common/exp/fma3/vsexp4.cpp:17:19: warning: '__fvs_exp_fma3' has C-linkage specified, but returns user-defined type '__m128' (aka '__s128f') which is            
      incompatible with C [-Wreturn-type-c-linkage]                                                       
extern "C" __m128 __fvs_exp_fma3(__m128);                                                                 
                  ^                                                                                       
extern "C" __m128 __fvs_exp_fma3(__m128);                                                                                                                                                                   
                  ^                                                                                       
1 warning generated.                                                                                      

For now I will just move on to flang itself and come back later.

@d-parks
Copy link

d-parks commented Jan 18, 2022

Hi. I'm away from my desk reading this on my phone - so I might be missing something.

But if I'm following you correctly, you're trying to build for aarch64. The errors you're seeing are from an x86_64 target.

@meow464
Copy link
Author

meow464 commented Jan 18, 2022

Thanks, I will check if there is any if(aarch64) missing.

The build process does finish and link successfully, what makes you think those are x86 errors?

@bryanpkc
Copy link
Collaborator

But the functions in the error: feenableexcept, fedisableexcept and fegetexcept are GNU extensions and obviously not available on iOS or Windows.

We implemented our own AArch64-specific routines in fltfenv.c as well.

__io_set_errno is declared in runtime/include/stdioInterf.h and defined in runtime/flangrti/iostdinit.c. It used to be included indirectly by pgstdinit.h but this file has been deleted.

I remember that I ran into this problem as well. IIRC the use of __io_set_errno can be deleted and replaced with calling the libc version of the function.

@michaeldupton
Copy link

is there a set of code I can test on an apple M1?
I am hitting lots of the issues mentioned.

@meow464
Copy link
Author

meow464 commented Jan 22, 2022

It's not working yet but I'm making progress. libpgmath already builds, I'm now tackling the flang runtime itself, which I expect to be easier.
After that it should be usable on MacOS. For iOS I still need to automate the build of the Xcode framework and build arm64+x86 fat binaries.

@meow464
Copy link
Author

meow464 commented Jan 25, 2022

Everything builds statically, I'm now trying to solve some linking problems.

Linking error
[  0%] Linking C shared library ../../lib/libompstub.dylib
cd /Volumes/macos_extra/iOS/flang/build_iphoneos/runtime/ompstub && /usr/local/Cellar/cmake/3.21.4/bin/cmake -E cmake_link_script CMakeFiles/ompstub_shared.dir/link.txt --verbose=1
/usr/local/stow/flang/bin/clang  -fPIC -Werror=date-time -Werror=unguarded-availability-new -w -ffunction-sections -fdata-sections -Werror -Wno-error=sign-compare -O3 -DNDEBUG -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk -miphoneos-version-min=9.0 -dynamiclib -Wl,-headerpad_max_install_names -L/usr/local/opt/[email protected]/lib -Wl,-z,defs -o ../../lib/libompstub.dylib -install_name @rpath/libompstub.dylib CMakeFiles/ompstub_shared.dir/init_nomp.c.o CMakeFiles/ompstub_shared.dir/ompstubs.c.o   -L/usr/local/stow/flang/lib 
ld: unknown option: -z
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [lib/libompstub.dylib] Error 1
make[1]: *** [runtime/ompstub/CMakeFiles/ompstub_shared.dir/all] Error 2
make: *** [all] Error 2

The problem seems to be with -z,defs, this flag does exist on Linux and according to man ld: " Report unresolved symbol references from regular object files. This is done even if the linker is creating a non-symbolic shared library. This option is the inverse of -z undefs.".

But it does not exist on mac, the equivalient seems to be -no_weak_imports: Error if any symbols are weak imports (i.e. allowed to be unresolved (NULL) at runtime). Useful for config based projects that assume they are built and run on the same OS version.

I can't figure out where this flag is being set.

@meow464
Copy link
Author

meow464 commented Jan 26, 2022

The problem does not happen when -DCMAKE_SYSTEM_NAME=Darwin and I still cannot find where those flags are coming from. This makes me think they were indeed auto inserted by CMake.

Now what would be a good way around this? Should I just set the system name to Darwin instead of iOS? Ideally we should be able to distinct between both. iOS for example doesn't support system() calls and building the compiler itself makes no sense.

@kiranchandramohan
Copy link
Collaborator

Now what would be a good way around this? Should I just set the system name to Darwin instead of iOS?
I guess for a local workaround that should be OK. Or are you asking whether this is acceptable in the flang project?

Anyway, great progress here.

@meow464
Copy link
Author

meow464 commented Jan 26, 2022

I would like to make pull request, yes.

@kiranchandramohan
Copy link
Collaborator

The llvm-project cmake sets the System Name to Darwin in iOS, so I think it should be OK.
https://github.com/llvm/llvm-project/blob/db0631096e59c71b6f6b034306327b42d9184184/llvm/cmake/platforms/iOS.cmake#L3

@meow464
Copy link
Author

meow464 commented Jan 26, 2022

I can see they also have SET(IOS True). Thanks very much!

@meow464
Copy link
Author

meow464 commented Jan 31, 2022

We are 90% done, now there is only 90% to go. 🎉 🎉 🎉

All the libraries build and link statically and dynamically. But I still need to:

  1. Test on a real device;
  2. Build Xcode frameworks;
  3. Fix a bunch of warnings;
  4. Build it for the iOS x86 simulator (I only have a x86 machine so this is kinda important to me);
  5. Change a bunch of #ifdefs and hopefully it will just work on the M1 (this could be the second item of the list, the more people testing the better).

A three years old issue was solved 20 days ago. Talk about luck. I will leave it here for reference.

@kiranchandramohan
Copy link
Collaborator

@meow464 This is great. M1 support will be super. Thanks for pushing forward with this.

@meow464
Copy link
Author

meow464 commented Feb 10, 2022

OpenBLAS compiles and links successfully!
But I still have to test it on a real device.

@meow464
Copy link
Author

meow464 commented Feb 13, 2022

I'm trying to make OpenBLAS work with numpy on iOS. It's not yet ready for a pull request but but I will leave the instructions here for now in case someone else needs to test it.

  1. Install flang with the wiki instructions;
  2. Save the three files below;
  3. Run build-llvm.sh and build-flang.sh in sequence. Don't forget to add -jN to make (N is how many cores you want the build process to use). Everything will be installed under INSTALL_PREFIX=/usr/local/flang-iOS;
  4. To build for iOS simply use flang -L/usr/local/flang-iOS/lib -isysroot $(xcrun --sdk iphoneos --show-sdk-path)

Things to be aware of:

  1. ATM it only builds for arm64;
  2. Enabling bitcode embedding means disabling dylibs, see build-flang-iphoneos.sh

setup.sh

INSTALL_PREFIX=/usr/local/flang-iOS

# Targets to build should be one of: X86 PowerPC AArch64
CMAKE_OPTIONS="-DCMAKE_INSTALL_PREFIX=$INSTALL_PREFIX \
    -DLLVM_CONFIG=$INSTALL_PREFIX/bin/llvm-config \
    -DCMAKE_CXX_COMPILER=$INSTALL_PREFIX/bin/clang++ \
    -DCMAKE_C_COMPILER=$INSTALL_PREFIX/bin/clang \
    -DCMAKE_Fortran_COMPILER=/usr/local/bin/flang \
    -DCMAKE_Fortran_COMPILER_ID=Flang \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_TARGETS_TO_BUILD=X86;AArch64"

build-llvm.sh

. setup.sh

if [[ ! -d classic-flang-llvm-project ]]; then
    git clone -b release_100 https://github.com/flang-compiler/classic-flang-llvm-project.git
fi

cd classic-flang-llvm-project
mkdir -p build && cd build
cmake $CMAKE_OPTIONS -DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++ \
      -DLLVM_ENABLE_CLASSIC_FLANG=ON -DLLVM_ENABLE_PROJECTS="clang;openmp" ../llvm
make
sudo make install

build-flang-iphoneos.sh

. setup.sh

# use the branches ios or ios_bitcode
if [[ ! -d flang ]]; then
    git clone -b ios https://github.com/meow464/flang.git
fi

CMAKE_OPTIONS_COMMON=$CMAKE_OPTIONS" -DLLVM_FLANG_CPU_TARGET=apple-a9 \
   -DCMAKE_SYSTEM_PROCESSOR=arm64 \
   -DCMAKE_CROSSCOMPILING=TRUE \
   -DCMAKE_SYSTEM_NAME=Darwin \
   -DIOS=TRUE \
   -DCMAKE_OSX_SYSROOT=$(xcrun --sdk iphoneos --show-sdk-path) \ 
   -DCMAKE_OSX_DEPLOYMENT_TARGET=9.0 \
   -DCMAKE_OSX_ARCHITECTURES=arm64"

CMAKE_OPTIONS_FLANG=$CMAKE_OPTIONS_COMMON" -DCMAKE_MACOSX_BUNDLE=OFF"

(cd flang/runtime/libpgmath
 mkdir -p build_iphoneos && cd build_iphoneos
 cmake $CMAKE_OPTIONS_COMMON ..
 make
 sudo make install)

cd flang
mkdir -p build_iphoneos && cd build_iphoneos
cmake $CMAKE_OPTIONS_FLANG -DFLANG_LLVM_EXTENSIONS=ON ..
make
sudo make install

@meow464
Copy link
Author

meow464 commented Feb 20, 2022

Problem: I'm running into a EXC_BAD_ACCESS with this call stack (forgive the formatting but the Xcode debugger UI is not copy/paste friendly):

  1. lapacke_dgesdd.c: info = LAPACKE_dgesdd_work( matrix_layout, jobz, m, n, a, lda, s, u, ldu, vt, ldvt, &work_query, lwork, iwork );

  2. lapacke_dgesdd_work.c: LAPACK_dgesdd( &jobz, &m, &n, a, &lda_t, s, u, &ldu_t, vt, &ldvt_t, work, &lwork, iwork, &info );

  3. ilaenv.f -> INTEGER FUNCTION ILAENV( ISPEC, NAME, OPTS, N1, N2, N3, N4 ): SUBNAM = NAME

  4. fntcharsup.c -> void Ftn_str_copy_klen(int n, char *to, int64_t to_len, ...): *to_p++ = *from;

Where Ftn_str_copy_klen is a variable args function and seems to be the implementation of the assignment SUBNAME=NAME.
Going step by step on lldb I can see that from is NULL, in fact it never receives a value here from = va_arg(ap, char *);.

All of this led me to this commit. I was simply using the flang, flang1 and flang2 build for x86 but it seems some code generation code wasn't being triggered?
The files in the commit have changed quite a bit but there is a very similar #ifdef TARGET_LLVM_ARM64. I will try building flang, flang1 and flang2 for x86 with TARGET_LLVM_ARM64 and see how it goes.

Please let me know if this makes sense. I'm missing something obvious or about to shoot myself on the foot?

@meow464
Copy link
Author

meow464 commented Feb 22, 2022

It didn't work :(
I still get a EXC_BAD_ACCESS on this line because *from is NULL.

I don't know how to solve this.

EDIT: Sometimes I get a SIGSEGV but I still don't know how to solve it.

@meow464
Copy link
Author

meow464 commented Feb 22, 2022

Note: I copied the source of the function below.

If I put a breakpoint at the function declaration I can actually proceed through by going step by step, from is no longer NULL but now I get a EXC_BAD_ACCESS at memcpy(qq->str, from, (size_t)from_len);

The only thing I can think of is one thread stepping on each other's foot?

Ftn_str_copy_klen
void
Ftn_str_copy_klen(int n, char *to, int64_t to_len, ...)
{
  va_list ap;
  char *from;
  int64_t from_len;
  int idx2;
  int cnt;
  typedef struct {
    char *str;
    int64_t len;
    int dyn;
  } SRC_STR;
  SRC_STR *src_p, *qq;
  SRC_STR aa[4]; /* statically allocate for 4 source strings */
  int src_p_allocd;
  char *to_p;
  char *to_end;
  char *from_end;
  int any_allocd;

  if (to_len <= 0) {
    return;
  }
  if (n <= (sizeof(aa) / sizeof(SRC_STR))) {
    qq = src_p = aa;
    src_p_allocd = 0;
  } else {
    qq = src_p = (SRC_STR *)_mp_malloc((size_t)sizeof(SRC_STR) * n);
    src_p_allocd = 1;
  }
  va_start(ap, to_len);
#ifdef DEBUG
  printf("to_len = %d\n", to_len);
#endif
  to_end = to - 1;
  any_allocd = idx2 = 0;
  for (cnt = n; cnt > 0; cnt--, qq++) {
    from = va_arg(ap, char *);
    from_len = va_arg(ap, int64_t);
#ifdef DEBUG
    printf("from_len = %ld\n", from_len);
#endif
    if (from_len < 0)
      from_len = 0;
    qq->str = from;
    qq->len = from_len;
    qq->dyn = 0;
    to_end += from_len;
    from_end = from + (from_len - 1);
    if ((from >= to && from <= to_end) ||
        (from_end >= to && from_end <= to_end))
      if (from_len) {
        qq->str = _mp_malloc((size_t)from_len);
        memcpy(qq->str, from, (size_t)from_len);
        qq->dyn = 1;
#ifdef DEBUG
        printf("string %d overlaps\n", n - cnt);
        printf("mallocd %08x\n", qq->str);
#endif
        any_allocd = 1;
      }
    idx2 += from_len;
    if (idx2 >= to_len)
      break;
  }
  va_end(ap);

  qq = src_p;
  to_p = to;
  to_end = to + to_len; /* position after the end of the destination */
  for (cnt = n; cnt > 0; cnt--, qq++) {
    from = qq->str;
    for (from_len = qq->len; from_len > 0; from_len--, from++) {
      *to_p++ = *from;
      if (to_p == to_end)
        goto exit_return;
#ifdef DEBUG
      printf("from_char = %c\n", *from);
#endif
    }
  }

exit_return:
  while (to_p < to_end) { /* remember, to_end is 1 after end */
    /* blank fill to right */
    *to_p++ = ' ';
  }

  if (any_allocd) {
    idx2 = 0;
    qq = src_p;
    for (cnt = n; cnt > 0; cnt--, qq++) {
      if (qq->dyn) {
        _mp_free(qq->str);
#ifdef DEBUG
        printf("freed   %08x\n", qq->str);
#endif
      }
      idx2 += qq->len;
      if (idx2 >= to_len)
        break;
    }
  }

  if (src_p_allocd)
    _mp_free(src_p);
}

@bryanpkc
Copy link
Collaborator

What does the assembly code (i.e. compiled code for the Fortran function ILAENV) look like before the call to Ftn_str_copy_klen? Is it passing the expected number of arguments?

@meow464
Copy link
Author

meow464 commented Feb 23, 2022

I'm knowledgeable enough to tell if this is correct. It doesn't look particularly wrong.

    0x100a5d3cc <+132>:  ldr    x4, [sp, #0x68]
    0x100a5d3d0 <+136>:  mov    w8, #0x1
    0x100a5d3d4 <+140>:  add    x1, sp, #0x50             ; =0x50
    0x100a5d3d8 <+144>:  mov    w0, #0x1
    0x100a5d3dc <+148>:  mov    w2, #0x10
    0x100a5d3e0 <+152>:  str    w8, [sp, #0x64]
    0x100a5d3e4 <+156>:  bl     0x1009dd2b8               ; f90_str_copy_klen at ftncharsup.c:286

Inside f90_str_copy_klen, even after va_start(ap, to_len), ap remains \0.
(edited) Note: ap is a string on iOS

Note: This is not compiled with TARGET_LLVM_ARM64 as mentioned 3 comments above.

@meow464
Copy link
Author

meow464 commented Feb 25, 2022

It also happens when:

  1. Compiling flang, flang1, flang2 with -DTAREGT_LLVM_ARM64 -DTARGET_LLVM_ARM -DTARGET_ARM64 -DTARGET_ARM;
  2. Manually calling f90_str_copy_klen from objective-c.

With obj-c the function at least seems to receive the string parameter but the error still happens on the same line.
I got a SIGBUS once. I guess I will keep trying to pinpoint the origin.

Anyone using flang on ARM by any chance?

@bryanpkc
Copy link
Collaborator

bryanpkc commented Mar 5, 2022

It seems that a proper port of commit 5d46b68 for AArch64 Darwin iOS is necessary. Hopefully someone else with hardware access can help look into it.

@meow464
Copy link
Author

meow464 commented Mar 6, 2022

(I didn't give up I just been busy. I got here, I'm fixing this)

The function can be called by C code compiled with clang-10. This makes me think llvm11 probably wouldn't be helpful.
Some interesting things: The C code seems to use an unusual set of registers x8, x9, x10 (only x0-x7 are for argument passing?). The flang code does use x0-x7 but the code for f90_str_copy_klen still tries to manipulate x8, x9, x10.
The code for f90_str_copy_klen is not included below.

Assembly generated by flang and register inspection
    0x104d7f074 <+132>:  ldr    x4, [sp, #0x68]
    0x104d7f078 <+136>:  mov    w8, #0x1
    0x104d7f07c <+140>:  add    x1, sp, #0x50             ; =0x50
    0x104d7f080 <+144>:  mov    w0, #0x1
    0x104d7f084 <+148>:  mov    w2, #0x10
->  0x104d7f088 <+152>:  str    w8, [sp, #0x64]
    0x104d7f08c <+156>:  bl     0x104ce6660               ; f90_str_copy_klen at ftncharsup.c:286

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

General Purpose Registers:
        x0 = 0x0000000000000001  ; `n` number of va_args?
        x1 = 0x000000016b11ee30  ; `to` char array?
        x2 = 0x0000000000000010  ; length of `to` array?
        x3 = 0x0000000104d87719  lapack-test3`.C349_dgebrd_ ; `from`char array
        x4 = 0x0000000000000006  ; length of `from`array
        ; values seem to repeat on registers below
        x5 = 0x0000000104d87714  lapack-test3`.C351_dgebrd_
        x6 = 0x0000000104d87714  lapack-test3`.C351_dgebrd_
        x7 = 0x0000000000000006
        x8 = 0x0000000000000001
        x9 = 0x0000000104d87ea0  lapack-test3`.C324_dlamch_ + 7
       x10 = 0x0000000104d7f074  lapack-test3`ilaenv_ + 132
       x11 = 0x000000000000000b
       x12 = 0x000000028254c080
       x13 = 0x000000001c43e800
       x14 = 0x000000001c400000
       x15 = 0x000000000000004c
       x16 = 0x00000000000008fd
       x17 = 0x00000000000008fd
       x18 = 0x0000000000000000
       x19 = 0x000000016b11f294
       x20 = 0x000000016b11f298
       x21 = 0x0000000104d87714  lapack-test3`.C351_dgebrd_
       x22 = 0x0000000104d87714  lapack-test3`.C351_dgebrd_
       x23 = 0x0000000104d87720  lapack-test3`.C294_dgebrd_
       x24 = 0x000000016b11f130
       x25 = 0x000000016b11f130
       x26 = 0x000000016b11f280
       x27 = 0x000000016b11f12c
       x28 = 0x000000016b11f294
        fp = 0x000000016b11f2f0
        lr = 0x0000000104d5f398  lapack-test3`dgebrd_ + 148
        sp = 0x000000016b11ede0
        pc = 0x0000000104d7f088  lapack-test3`ilaenv_ + 152
      cpsr = 0x80000000
Assembly code generated by clang-10 with register inspection
    int n=1;
    int64_t to_len = 6;
    char to[to_len];

    char *from = "ABCDEF";
    int from_len = 6;
    f90_str_copy_klen(n, &to, to_len, from, from_len);
    NSLog(@(from));

->  0x100cb98a4 <+132>: ldur   w0, [x29, #-0x1c]
    0x100cb98a8 <+136>: ldur   x2, [x29, #-0x28]
    0x100cb98ac <+140>: ldur   x10, [x29, #-0x40]
    0x100cb98b0 <+144>: ldur   w9, [x29, #-0x44]
    0x100cb98b4 <+148>: mov    x8, x9
    0x100cb98b8 <+152>: sub    sp, sp, #0x10             ; =0x10
    0x100cb98bc <+156>: mov    x9, sp
    0x100cb98c0 <+160>: str    x10, [x9]
    0x100cb98c4 <+164>: str    x8, [x9, #0x8]
    0x100cb98c8 <+168>: bl     0x100cba678               ; f90_str_copy_klen at ftncharsup.c:286

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

General Purpose Registers:
        x0 = 0x0000000000000001  ; number of va_arg
        x1 = 0x000000016f14b690  ; `to` char array?
        x2 = 0x0000000000000006  ; length of `from`
        x3 = 0x000000016f14b958  ; seems to be memory junk
        x4 = 0x0000000000000000
        x5 = 0x0000000000000000
        x6 = 0x0000000000000000
        x7 = 0x0000000000000000
        x8 = 0x0000000000000006 ; length of `from` argument
        x9 = 0x000000016f14b680 ; stack pointer
       x10 = 0x0000000100d5549c  "ABCDEF" ; this is `from`. why is it here?
       x11 = 0x0000000000000002
       x12 = 0x0000000000000002
       x13 = 0x0000000000000000
       x14 = 0x0000000000000001
       x15 = 0x000000000000004d
       x16 = 0x00000001f2436ea4  libsystem_pthread.dylib`___chkstk_darwin
       x17 = 0x6ae100016f148c00
       x18 = 0x0000000000000000
       x19 = 0x0000000100f64060
       x20 = 0x0000000100cb9820  lapack-test3`main at main.m:30
       x21 = 0x0000000100f14070  dyld`dyld4::sConfigBuffer
       x22 = 0x0000000000000000
       x23 = 0x0000000000000000
       x24 = 0x0000000000000000
       x25 = 0x0000000000000000
       x26 = 0x0000000000000000
       x27 = 0x0000000000000000
       x28 = 0x0000000000000000
        fp = 0x000000016f14b720
        lr = 0x0000000100cb987c  lapack-test3`main + 92 at main.m:43:5
        sp = 0x000000016f14b680
        pc = 0x0000000100cb98c8  lapack-test3`main + 168 at main.m:47:5
      cpsr = 0x20000000

It's way past midnight now but I will keep investigating.

@meow464
Copy link
Author

meow464 commented Mar 9, 2022

The darwin aarch64 ABI differs a bit from the default ARM64. It's not implemented in LLVM yet but it is implemented in clang and rustc so I will take inspiration from them.

@meow464
Copy link
Author

meow464 commented Apr 12, 2022

I'm back.

On the AAPCS arguments for variadic functions are passed on the registers (x0-x7) until they are full and only then added to the stack.

On Apple only the normal arguments are assigned to registers and all variadic arguments are put on the stack.

You can see on my comment above how flang is trying to put all parameters on registers, while clang correctly puts only the first 3 on registers.

The function responsible for deciding what to do with vargs seems to be ll_abi_classify_va_arg_dtype. On aarch64 it's implemented here.

"ll_abi_classify_va_arg_dtype" prototype and doc string
/* Classify a dtype for va_arg().
 *
 * This is needed for x86-64 and aarch64 which use a separate register save
 * area in varargs functions.
 *
 * Return the number of general-purpose and and floating point registers needed
 * to pass dtype in num_gp, num_fp. Return (0,0) for an argument that must be
 * passed in memory.
 *
 * The returned value is the 'map' argument for __builtin_va_genargs. It is one
 * of GP_XM(0), XM_GP(1), or XM_XM(2). See rte/pgc/hammer/src/va_arg.c.
 */
unsigned ll_abi_classify_va_arg_dtype(LL_Module* module, DTYPE dtype, 
                                      unsigned *num_gp, unsigned *num_fp);

As documented, I set *num_gp=0 and num_fp=0:

New ll_abi_classify_va_arg_dtype
unsigned
ll_abi_classify_va_arg_dtype(LL_Module *module, DTYPE dtype, unsigned *num_gp,
                             unsigned *num_fp)
{
  *num_gp = 0;
  *num_fp = 0;

  return 0;
}

I also set the CMakeLists.txt to use the modified aarch/ll_abi.cpp.

But yet, the problem persists and the generated code looks the exact same: parameters on registers and nothing on the stack.

Registers
        x0 = 0x0000000000000001  ; `n` number of variadics
        x1 = 0x000000016f90ee30   ; seems to be '*to`
        x2 = 0x0000000000000010  ;  seems to be `to_len`
        x3 = 0x0000000100597729  lapack-test3`.C349_dgebrd_ ; seems to be `from`
        x4 = 0x0000000000000006 ; seems to be `from_len`
        x5 = 0x0000000100597724  lapack-test3`.C351_dgebrd_
        x6 = 0x0000000100597724  lapack-test3`.C351_dgebrd_
        x7 = 0x0000000000000006
        x8 = 0xcbb1c2270d090016
        x9 = 0x0000000100597eb0  lapack-test3`.C324_dlamch_ + 7
      x10 = 0x000000010058e8b4  lapack-test3`ilaenv_ + 132
      x11 = 0x000000000000000b
      x12 = 0x000000028291c880
      x13 = 0x000000000c836800
      x14 = 0x000000000c800000
      x15 = 0x000000000000004c
      x16 = 0x00000000000008fd
      x17 = 0x00000000000008fd
      x18 = 0x0000000000000000
      x19 = 0x000000016f90f294
      x20 = 0x000000016f90f298
      x21 = 0x0000000100597724  lapack-test3`.C351_dgebrd_
      x22 = 0x0000000100597724  lapack-test3`.C351_dgebrd_
      x23 = 0x0000000100597730  lapack-test3`.C294_dgebrd_
      x24 = 0x000000016f90f130
      x25 = 0x000000016f90f130
      x26 = 0x000000016f90f280
      x27 = 0x000000016f90f12c
      x28 = 0x000000016f90f294
        fp = 0x000000016f90edd0
         lr = 0x000000010058e8d0  lapack-test3`ilaenv_ + 160
       sp = 0x000000016f90ecd0
       pc = 0x00000001004f66ac  lapack-test3`f90_str_copy_klen + 44 at ftncharsup.c:305:7
    cpsr = 0x80000000
Calling assembly code in ilaenv
lapack-test3`ilaenv_:
    0x10058e830 <+0>:    sub    sp, sp, #0xb0             ; =0xb0
    0x10058e834 <+4>:    stp    x24, x23, [sp, #0x70]
    0x10058e838 <+8>:    stp    x22, x21, [sp, #0x80]
    0x10058e83c <+12>:   stp    x20, x19, [sp, #0x90]
    0x10058e840 <+16>:   stp    x29, x30, [sp, #0xa0]
    0x10058e844 <+20>:   str    x7, [sp, #0x68]
    0x10058e848 <+24>:   ldr    w8, [x0]
    0x10058e84c <+28>:   sub    w8, w8, #0x1              ; =0x1
    0x10058e850 <+32>:   cmp    w8, #0xf                  ; =0xf
    0x10058e854 <+36>:   b.hi   0x10058eb1c               ; <+748>
    0x10058e858 <+40>:   adr    x9, #0x9658               ; .C324_dlamch_ + 7
    0x10058e85c <+44>:   nop
    0x10058e860 <+48>:   adr    x10, #0x28                ; <+88>
    0x10058e864 <+52>:   ldrb   w11, [x9, x8]
    0x10058e868 <+56>:   add    x10, x10, x11, lsl #2
    0x10058e86c <+60>:   mov    x21, x6
    0x10058e870 <+64>:   mov    x22, x5
    0x10058e874 <+68>:   mov    x19, x4
    0x10058e878 <+72>:   mov    x20, x3
    0x10058e87c <+76>:   mov    x23, x0
    0x10058e880 <+80>:   mov    x3, x1
    0x10058e884 <+84>:   br     x10
    0x10058e888 <+88>:   ldr    x7, [sp, #0x68]
    0x10058e88c <+92>:   ldr    x8, [sp, #0xb0]
    0x10058e890 <+96>:   mov    x0, x23
    0x10058e894 <+100>:  mov    x1, x3
    0x10058e898 <+104>:  mov    x3, x20
    0x10058e89c <+108>:  mov    x4, x19
    0x10058e8a0 <+112>:  mov    x5, x22
    0x10058e8a4 <+116>:  mov    x6, x21
    0x10058e8a8 <+120>:  str    x8, [sp]
    0x10058e8ac <+124>:  bl     0x10058f7b4               ; iparmq_
    0x10058e8b0 <+128>:  b      0x10058ebc8               ; <+920>
    0x10058e8b4 <+132>:  ldr    x4, [sp, #0x68]
    0x10058e8b8 <+136>:  mov    w8, #0x1
    0x10058e8bc <+140>:  add    x1, sp, #0x50             ; =0x50
    0x10058e8c0 <+144>:  mov    w0, #0x1
    0x10058e8c4 <+148>:  mov    w2, #0x10
    0x10058e8c8 <+152>:  str    w8, [sp, #0x64]
    0x10058e8cc <+156>:  bl     0x1004f6680               ; f90_str_copy_klen at ftncharsup.c:286

It really looks like I accidentally didn't swap openblas.a for the new version but I sure did. I deleted the old one, ran the test app without it to make sure I would get a missing symbol error and it was not hidden in some cache then added the new version and tried again.

All ideas are welcomed.

@meow464
Copy link
Author

meow464 commented Apr 12, 2022

Hmm I can't find where ll_abi_classify_va_arg_dtype is being called from...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants