Skip to content

Xmake v2.6.6 Released, Support Distributed Compilation and Build Cache

Jump to bottom
ruki edited this page May 25, 2022 · 1 revision

Xmake is a lightweight cross-platform build utility based on Lua.

It is very lightweight and has no dependencies because it has a built-in Lua runtime.

It uses xmake.lua to maintain project builds and its configuration syntax is very simple and readable.

We can use it to build project directly like Make/Ninja, or generate project files like CMake/Meson, and it also has a built-in package management system to help users solve the integrated use of C/C++ dependent libraries.

Xmake = Build backend + Project Generator + Package Manager + [Remote|Distributed] Build + Cache

Although not very precise, we can still understand Xmake in the following way:

Xmake ~= Make/Ninja + CMake/Meson + Vcpkg/Conan + distcc + ccache/sccache

Introduction of new features

In this version, we have added a lot of heavyweight new features:

  • Distributed compilation
  • Local compilation cache
  • Remote compilation cache

With these features, we can compile large C/C++ projects faster.

In addition, they are completely cross-platform, support not only gcc/clang but also msvc, and there is no any third-party dependency except the compiler, which is very convenient to use.

Therefore, using Xmake is equivalent to using distcc/ccache/sccache at the same time.

Compared with these third-party tools, Xmake fully supports Windows and msvc, which eliminates platform differences, independent process calls, and the overhead of additional daemon processes.

In addition to these features, the new version of Xmake also adds support for compiling Keil/C51 projects, as well as support for the nvc/nvc++/nvfortran compilers in the nvidia-hpc-sdk toolchain.

Support user authentication for remote compilation

In the last version, we initially supported remote compilation, but did not provide user authentication support, which would bring some security issues. Therefore, in this version, we added user authentication support.

At present, Xmake mainly provides the following authentication mechanisms. In addition, it is also effective for distributed compilation and remote caching.

  1. Token authentication
  2. Password authentication
  3. Trusted host verification

Token authentication

This is also the default recommended method, which is more secure, more convenient to configure and connect, and does not need to enter a password every time you connect.

When we execute the xmake service command, a server and client configuration file will be generated by default, and a default token will be automatically generated, so the local direct connection does not require any configuration.

Server authentication configuration

The server can configure multiple tokens for authorizing connections to different user hosts.

Of course, we can also share single token.

$ cat ~/.xmake/service/server.conf
{
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    remote_build = {
        listen = "0.0.0.0:9691",
        workdir = "/Users/ruki/.xmake/service/server/remote_build"
    },
    tokens = {
        "e438d816c95958667747c318f1532c0f"
    }
}
Client authentication configuration

The client only needs to add the token on the server to the corresponding client configuration.

$ cat ~/.xmake/service/client.conf
{
    remote_build = {
        connect = "127.0.0.1:9691",
        token = "e438d816c95958667747c318f1532c0f"
    }
}
Manually generate new token

We can also execute the following command to manually generate a new token and add it to the server configuration ourselves.

$ xmake service --gen-token
New token a7b9fc2d3bfca1472aabc38bb5f5d612 is generated!

Password authentication

We also provide password authentication. Compared with token authentication, it requires users to enter a password every time they connect, and can only be connected after the verification is passed.

Server authentication configuration

For password authentication, we do not need to manually configure the token, just execute the following command to add a user. During the adding process, the user will be prompted to enter a password.

$ xmake service --add-user=ruki
Please input user ruki password:
123456
Add user ruki ok!

Then, xmake will generate a new token from the username and password and add it to the token list of the server configuration.

$ cat ~/.xmake/service/server.conf
{
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    remote_build = {
        listen = "0.0.0.0:9691",
        workdir = "/Users/ruki/.xmake/service/server/remote_build"
    },
    tokens = {
        "e438d816c95958667747c318f1532c0f",
        "7889e25402413e93fd37395a636bf942"
    }
}

Of course, we can also delete the specified user and password.

$xmake service --rm-user=ruki
Please input user ruki password:
123456
Remove user ruki ok!
Client authentication configuration

For the client, we no longer need to set the token of the server. We only need to add the user name that needs to be connected in the connection configuration to enable password authentication.

The format is: user@address:port

$ cat ~/.xmake/service/client.conf
{
    remote_build = {
        connect = "[email protected]:9691"
  }
}

If the username is removed and the token is not configured, it is anonymous mode. If the server is not configured with a token, the authentication is completely disabled and the connection is made directly.

Trusted host verification

In addition, in order to further improve security, we also provide server-side trusted host verification.

If the server-configured known_hosts list is configured with the ip address of the client host that can be connected.

Then only these hosts can successfully connect to this server, and other hosts' connections to it will be prompted to be untrusted and refuse the connection, even if token and password authentication are OK.

$ cat ~/.xmake/service/server.conf
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    server = {
        tokens = {
            "4b928c7563a0cba10ff4c3f5ca0c8e24"
        },
        known_hosts = { "127.0.0.1", "xx.xx.xx.xx"}
    }
}

Connect to remote server

Next, we only need to enter the root directory of the project that needs to be compiled remotely, and execute the xmake service --connect command to connect.

If it is the token authentication mode, then no additional password input is required, and the connection is directly connected.

$ xmake create test
$ cd test
$ xmake service --connect
<remote_build_client>: connect 192.168.56.110:9091 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 192.168.56.110:9091 ..
Scanning files ..
Comparing 3 files ..
    [+]: src/main.cpp
    [+]: .gitignore
    [+]: xmake.lua
3 files has been changed!
Archiving files ..
Uploading files with 1372 bytes ..
<remote_build_client>: sync files ok!

If it is password authentication, the user will be prompted to enter the password to continue the connection.

$ xmake service --connect
Please input user root password:
000000
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 127.0.0.1:9691 ..
Scanning files ..
Comparing 3 files ..
    [+]: xmake.lua
    [+]: .gitignore
    [+]: src/main.cpp
3 files has been changed!
Archiving files ..
Uploading files with 1591 bytes ..
<remote_build_client>: sync files ok!

If the password is incorrect, an error message will be displayed.

$ xmake service --connect
Please input user root password:
123
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connect 127.0.0.1:9691 failed, user and password are incorrect!

Distributed compilation support

Xmake provides a built-in distributed compilation service, usually it can cooperate with local compilation cache and remote compilation cache to achieve optimal compilation acceleration.

Also, it is fully cross-platform supported, we not only support gcc/clang, but also Windows and msvc well.

For cross-compilation, as long as the cross-toolchain supports, we do not require the system environment of the server. Even if the server resources of linux, macOS and Windows are mixed, distributed compilation can be well realized.

Start service

We can specify the --distcc parameter to enable the distributed compilation service. Of course, if this parameter is not specified, xmake will enable all server-configured services by default.

$ xmake service --distcc
<distcc_build_server>: listening 0.0.0.0:9093 ..

We can also start the service and echo detailed log information.

$ xmake service --distcc -vD
<distcc_build_server>: listening 0.0.0.0:9093 ..

Start the service in daemon mode

$ xmake service --distcc --start
$ xmake service --distcc --restart
$ xmake service --distcc --stop

Configure the server

We first, run the xmake service command, it will automatically generate a default server.conf configuration file, stored in ~/.xmake/service/server.conf.

$ xmake service
generating the config file to /Users/ruki/.xmake/service/server.conf ..
an token(590234653af52e91b9e438ed860f1a2b) is generated, we can use this token to connect service.
generating the config file to /Users/ruki/.xmake/service/client.conf ..
<distcc_build_server>: listening 0.0.0.0:9693 ..

Then, we edit it, fixing the server's listening port (optional).

$ cat ~/.xmake/service/server.conf
{
    distcc_build = {
        listen = "0.0.0.0:9693",
        workdir = "/Users/ruki/.xmake/service/server/distcc_build"
    },
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    tokens = {
        "590234653af52e91b9e438ed860f1a2b"
    }
}

Configure the client

The client configuration file is in ~/.xmake/service/client.conf, where we can configure the server address that the client needs to connect to.

We can configure multiple server addresses and corresponding tokens in the hosts list.

For distributed compilation, it is recommended to use the token authentication mode, because the password mode requires a password to be entered for each server connection, which is very cumbersome.

$cat ~/.xmake/service/client.conf
{
    distcc_build = {
        hosts = {
            {
                connect = "127.0.0.1:9693",
                token = "590234653af52e91b9e438ed860f1a2b"
            }
        }
    }
}

Connect to the server

After configuring the authentication and server address, you can enter the following command to connect the current project to the configured server.

We need to enter --distcc when connecting to specify that only distributed services are connected.

$ cd projectdir
$ xmake service --connect --distcc
<client>: connect 127.0.0.1:9693 ..
<client>: 127.0.0.1:9693 connected!

We can also connect to multiple services at the same time, such as distributed compilation and remote compilation cache services.

$ xmake service --connect --distcc --ccache

!> If there is no parameter, the default connection is the remote compilation service.

Distributed compilation project

After connecting to the server, we can perform distributed compilation like normal local compilation, for example:

$ xmake
...
[ 93%]: ccache compiling.release src/demo/network/unix_echo_client.c ----> local job
[ 93%]: ccache compiling.release src/demo/network/ipv6.c
[ 93%]: ccache compiling.release src/demo/network/ping.c
[ 93%]: distcc compiling.release src/demo/network/unix_echo_server.c. ----> distcc job
[93%]: distcc compiling.release src/demo/network/http.c
[ 93%]: distcc compiling.release src/demo/network/unixaddr.c
[ 93%]: distcc compiling.release src/demo/network/ipv4.c
[ 94%]: distcc compiling.release src/demo/network/ipaddr.c
[94%]: distcc compiling.release src/demo/math/fixed.c
[94%]: distcc compiling.release src/demo/libm/float.c
[ 95%]: ccache compiling.release src/demo/libm/double.c
[ 95%]: ccache compiling.release src/demo/other/test.cpp
[ 98%]: archiving.release libtbox.a
[99%]: linking.release demo
[100%]: build ok!

Among them, the words with distcc are remote compilation tasks, and the others are local compilation tasks. By default, xmake also enables local compilation caching to cache distributed compilation results to avoid frequent requests to the server.

In addition, we can also open the remote compilation cache and share the compilation cache with others to further accelerate the compilation of multi-person collaborative development.

Disconnect server

$ xmake service --disconnect --distcc

Specify the number of parallel compilation tasks

Let's briefly introduce the number of parallel tasks currently calculated by default based on the number of host cpu cores:

local default_njob = math.ceil(ncpu * 3 / 2)

Therefore, if distributed compilation is not enabled, the default maximum number of parallel compilation tasks is this default_njob.

If distributed compilation is enabled, the default number of parallel compilation tasks is:

local maxjobs = default_njob + server_count * server_default_njob
Modify the number of local parallel tasks

We only need to pass -jN to specify the number of local parallel tasks, but it will not affect the number of parallel tasks on the server side.

$ xmake -jN
Modify the number of parallel tasks on the server side

If you want to modify the number of parallel tasks on the server, you need to modify the configuration file of the client.

$cat ~/.xmake/service/client.conf
{
    distcc_build = {
        hosts = {
            {
                connect = "127.0.0.1:9693",
                token = "590234653af52e91b9e438ed860f1a2b",
                njob = 8 <------- modify here
            },
            {
                connect = "192.168.01:9693",
                token = "590234653af52e91b9e438ed860f1a2b",
                njob = 4
            }
        }
    }
}

For each server host, add the njob = N parameter configuration to specify the number of parallel jobs that this server can provide.

Distributed compilation of Android projects

The distributed compilation service provided by xmake is completely cross-platform and supports Windows, Linux, macOS, Android, iOS and even cross-compilation.

If you want to compile the Android project, you only need to add the toolchains toolchain configuration in the server configuration, and provide the path of the NDK.

$ cat ~/.xmake/service/server.conf
{
    distcc_build = {
        listen = "0.0.0.0:9693",
        toolchains = {
            ndk = {
                ndk = "~/files/android-ndk-r21e" <------------ here
            }
        },
        workdir = "/Users/ruki/.xmake/service/server/distcc_build"
    },
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    tokens = {
        "590234653af52e91b9e438ed860f1a2b"
    }
}

Then, we can compile the Android project in a distributed way like normal local compilation, and even configure multiple Windows, macOS, Linux and other different server hosts as resources of the distributed compilation service to compile it.

Just download the NDK for the corresponding platform.

$ xmake f -p android --ndk=~/files/xxxx
$ xmake

Distributed compilation of iOS projects

Compiling iOS projects is easier, because Xmake can usually automatically detect Xcode, so just switch the platform to ios like a normal local.

$ xmake f -p iphoneos
$ xmake

Distributed cross compilation configuration

If we want to distribute cross-compilation, we need to configure the toolchain sdk path on the server, for example:

$ cat ~/.xmake/service/server.conf
{
    distcc_build = {
        listen = "0.0.0.0:9693",
        toolchains = {
            cross = {
                sdkdir = "~/files/arm-linux-xxx" <------------ here
            }
        },
        workdir = "/Users/ruki/.xmake/service/server/distcc_build"
    },
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    tokens = {
        "590234653af52e91b9e438ed860f1a2b"
    }
}

Among them, under toolchains, each item corresponds to a toolchain, here is configured as cross = {} cross toolchain, corresponding to toolchain("cross").

In the toolchain, we can configure sdkdir, bindir, cross, etc., corresponding to the interface configuration of set_sdkdir, set_bindir and set_cross in toolchain("cross").

If cross toolchain comparisonSpecification, we usually only need to configure sdkdir, xmake can automatically detect it.

And client-side compilation only needs to specify the sdk directory.

$ xmake f -p cross --sdk=/xxx/arm-linux-xxx
$ xmake

Clean server cache

The compilation of each project on the server side will generate some cache files, which are stored separately according to the project granularity.

We can use the following command to clear the cache corresponding to each server for the current project.

$ xmake service --clean --distcc

Some optimizations

  1. Cache server-side compilation results to avoid repeated compilation
  2. Local cache, remote cache optimization, avoid unnecessary server communication
  3. Server load balancing scheduling, rational allocation of server resources
  4. Small files are compiled directly locally after preprocessing, which is usually faster
  5. Real-time compression and transmission of large files, based on lz4 fast compression
  6. Internal state maintenance, compared to independent tools such as distcc, avoids frequent independent process loading and time-consuming, and avoids additional communication with the daemon process

Local compilation cache support

By default, Xmake will enable the local cache. The version before 2.6.5 uses the external ccache by default, and after 2.6.6, Xmake provides a built-in cross-platform local cache solution.

Compared with third-party independent processes such as ccache, xmake's internal state maintenance is easier to optimize, and it also avoids frequent independent process loading and time-consuming, and avoids additional communication with the daemon process.

In addition, the built-in cache can better support cross-platform, and msvc on Windows can also support well, while ccache only supports gcc/clang.

Of course, we can also disable the cache with the following command.

$ xmake f --ccache=n

Note: Regardless of whether the built-in local cache is used, the configuration name is --ccache=, which means the c/c++ build cache, not just the name of the ccache tool.

If we want to continue to use other external caching tools, we can also configure it in the following way.

$ xmake f --ccache=n --cxx="ccache gcc" --cc="ccache gcc"
$ xmake

Remote compilation cache support

In addition to local caching, we also provide remote caching services, similar to mozilla's sscache, which is usually not used if it is only for personal development.

However, if a large-scale project is developed collaboratively by multiple people within the company, distributed compilation and local caching alone are not enough. We also need to cache the compiled object files to a separate server for sharing.

In this way, even if other people compile it for the first time, they do not need to compile it distributedly every time, and directly pull the cache from the remote to speed up the compilation.

In addition, the remote cache service provided by Xmake is also supported by all platforms, not only gcc/clang but also msvc.

Start service

We can specify the --ccache parameter to enable the remote compilation cache service.

Of course, if this parameter is not specified, xmake will enable all server-configured services by default.

$ xmake service --ccache
<remote_cache_server>: listening 0.0.0.0:9092 ..

We can also start the service and echo detailed log information.

$ xmake service --ccache -vD
<remote_cache_server>: listening 0.0.0.0:9092 ..

Start the service in daemon mode

$ xmake service --ccache --start
$ xmake service --ccache --restart
$ xmake service --ccache --stop

Configure the server

We first, run the xmake service command, it will automatically generate a default server.conf configuration file, stored in ~/.xmake/service/server.conf.

$ xmake service
generating the config file to /Users/ruki/.xmake/service/server.conf ..
an token(590234653af52e91b9e438ed860f1a2b) is generated, we can use this token to connect service.
generating the config file to /Users/ruki/.xmake/service/client.conf ..
<remote_cache_server>: listening 0.0.0.0:9692 ..

Then, we edit it, fixing the server's listening port (optional).

$ cat ~/.xmake/service/server.conf
{
    distcc_build = {
        listen = "0.0.0.0:9692",
        workdir = "/Users/ruki/.xmake/service/server/remote_cache"
    },
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    tokens = {
        "590234653af52e91b9e438ed860f1a2b"
    }
}

Configure the client

The client configuration file is in ~/.xmake/service/client.conf, where we can configure the server address that the client needs to connect to.

We can configure multiple server addresses and corresponding tokens in the hosts list.

$cat ~/.xmake/service/client.conf
{
    remote_cache = {
            connect = "127.0.0.1:9692,
            token = "590234653af52e91b9e438ed860f1a2b"
        }
    }
}

Connect to the server

After configuring the authentication and server address, you can enter the following command to connect the current project to the configured server.

We need to enter --ccache when connecting to specify that only the remote compilation cache service is connected.

$ cd projectdir
$ xmake service --connect --ccache
<client>: connect 127.0.0.1:9692 ..
<client>: 127.0.0.1:9692 connected!

We can also connect to multiple services at the same time, such as distributed compilation and remote compilation cache services.

$ xmake service --connect --distcc --ccache

!> If there is no parameter, the default connection is the remote compilation service.

Disconnect server

$ xmake service --disconnect --ccache

Clean server cache

We can also use the following command to clear the cache on the remote server corresponding to the current project.

$ xmake service --clean --ccache

And if we execute xmake clean --all, when the remote service is connected, all caches will be automatically cleaned up.

Some optimizations

  1. Pull the snapshot of the remote cache and send it back to the local through bloom filter + lz4, which is used to quickly determine whether the cache exists and avoid frequently querying the server cache information
  2. With the local cache, you can avoid frequent requests to the remote server and pull the cache.
  3. Internal state maintenance, compared with independent tools such as sscache, avoids frequent independent process loading and time-consuming, and avoids additional communication with the daemon process

Keil/C51 project support

We only need to bind to the c51 toolchain, Xmake can automatically detect the Keil/C51 SDK toolchain environment installed on the system, and then use it to compile.

target("hello")
    add_rules("c51.binary")
    set_toolchains("c51")
    add_files("src/main.c")

Of course, if you don't set the toolchain through set_toolchains("c51"), we can also manually switch to the c51 toolchain through xmake f --toolchain=c51.

Changelog

New features

  • #2327: Support nvc/nvc++/nvfortran in nvidia-hpc-sdk
  • Add path instance interfaces
  • #2334: Add lz4 compress module
  • #2349: Add keil/c51 project support
  • #274: Distributed compilation support
  • Use builtin local cache instead of ccache

Changes

  • #2309: Support user authorization for remote compilation
  • Improve remote compilation to support lz4 compression

Bugs fixed

  • Fix lua stack when select package versions

中文

English

Clone this wiki locally