run_pytest.sh: "RuntimeError: DRM_IOCTL_AMDXDNA_CREATE_BO IOCTL failed (err=22): Invalid argument" with manually compiled xdna

[LakshyAAAgrawal]

Hi,

I am trying to setup the Riallto on a Minisforum UM790 Pro using the guide at https://github.com/AMDResearch/Riallto/tree/main/scripts/linux. Following the issues I faced with xdna installation as reported on (#67), I manually built and installed the xdna driver, as per instructions from https://github.com/amd/xdna-driver?tab=readme-ov-file#build.

After this, I launch the jupyter notebook using the launch_jupyter.sh <notebooks directory> command and then run tests for sanity check using run_pytest.sh, however many of the tests are failing, all with the same error message:

Autocomplete not enabled for XRT tools
XILINX_XRT        : /opt/xilinx/xrt
PATH              : /opt/xilinx/xrt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LD_LIBRARY_PATH   : /opt/xilinx/xrt/lib
PYTHONPATH        : /opt/xilinx/xrt/python
RTNETLINK answers: File exists
================================================================================================ test session starts =================================================================================================
platform linux -- Python 3.12.3, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/riallto/Riallto
plugins: anyio-4.6.0, cov-5.0.0
collected 160 items                                                                                                                                                                                                  

tests/test_address_32bit_issue.py F                                                                                                                                                                            [  0%]
tests/test_aietiles.py ^C

====================================================================================================== FAILURES ======================================================================================================
________________________________________________________________________________________________ test_32bit_transfers ________________________________________________________________________________________________

manage_testing = None

    def test_32bit_transfers(manage_testing):
        """ test to check if 32bit transfers are aligned correctly. """
        check_npu()
    
        set_value_src= """
    #include <stdint.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <aie_api/aie.hpp>
    
    extern "C" {
    
    void set_value(uint32_t *in_buffer, uint32_t *out_buffer){
        for(int i =0; i<16; i++) {
            out_buffer[i] = i;
        }
    }
    
    } // extern C
        """
    
        def setval_behavioral(obj):
             obj.out_buffer.array = np.array([range(16)])
    
        set_value = Kernel(set_value_src,setval_behavioral)
    
        A_buff = np.zeros(shape=(4,4), dtype=np.uint32)
        B_buff = np.zeros(shape=(4,4), dtype=np.uint32)
    
        set_value.tloc = (0,2)
    
        class SimpleInt32App(AppBuilder):
            def callgraph(self, x):
                return set_value(x)
    
        trace_app = SimpleInt32App()
        trace_app.build(A_buff)
    
        A = np.random.randint(0, 256, (4, 4), dtype=np.uint32)
        B = np.random.randint(0, 256, (4, 4), dtype=np.uint32)
    
>       app = AppRunner('SimpleInt32App.xclbin')

tests/test_address_32bit_issue.py:54: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
npu/runtime/apprunner.py:100: in __init__
    self.device =  self._get_device()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <npu.runtime.apprunner.AppRunner object at 0x7b6347a49b20>

    def _get_device(self):
        """ Checks to see if there is already an AppRunner that exists.
        if there is gets the device from previously allocated AppRunner or
        else creates a new device.
        """
    
        for obj in gc.get_objects():
            try:
                if isinstance(obj, type(self)) and (obj != self):
                    if getattr(obj, "device", None):
                        return obj.device
            except Exception as e:
                warn(f"Encountered an exception during isinstance check: {e}")
                continue
>       return ipr.device(0)
E       RuntimeError: DRM_IOCTL_AMDXDNA_CREATE_BO IOCTL failed (err=22): Invalid argument

npu/runtime/apprunner.py:135: RuntimeError
------------------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------------------
Building the set_value kernel...
Building the xclbin...
Successfully Building Application... SimpleInt32App.xclbin & SimpleInt32App.seq delivered
============================================================================================== short test summary info ===============================================================================================
FAILED tests/test_address_32bit_issue.py::test_32bit_transfers - RuntimeError: DRM_IOCTL_AMDXDNA_CREATE_BO IOCTL failed (err=22): Invalid argument
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/usr/lib/python3.12/subprocess.py:1196: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
================================================================================================= 1 failed in 35.34s =================================================================================================
riallto_pytest

Could you please help me understand and solve the issue? I have a debug version of the xdna-driver installed, and can provide the relevant debug information required.

[LakshyAAAgrawal]

Since I had deviated from the official guide, and instead manually compiled the xdna-driver, I believe there was a version mismatch between the module version on the host machine and the docker container. I modified the "setup_riallto_linux.sh" script to use my manually built xdna and xrt deb files, and was able to resolve the above issue. Hence, I am closing this discussion thread.

However, the issue reported earlier, of the npu modules not being installed properly (or rather, the size of the module being zero), still persists which I had to solve by compiling manually, instead of using the setup_riallto_linux.sh script. Hence, I will leave the thread at #67 open for now.