When does Meson decide to use a response file?

[lhearachel]

I am maintaining a project where we depend a lot on the specific ordering of files as an input to a particular tool for, e.g., bytecode generation, archive packing, etc. We currently are passing upwards of 500 files as command line arguments to our tools for these processes. Given the structure of our file tree, we worry about running into command line argument limits in the future.

To that end:

1. At what point does Meson decide to use a response file for excess command line args?
2. How many arguments are passed as part of the response file?
3. Are response files only used for compilers/linkers? Can we configure them for use to custom tools?

One solution that we have thought about to this problem is generating an input file for passage to our tools, ideally from a static list of files in the meson.build file. My solution to that was something like this:

custom_target('<target>.order',
    capture: true,
    output: '<target>.order',
    command: [ 'printf', '%s\n', <filenames> ]
)

But this feels like it will still run into the command line argument limit in the future. Is there a similar procedure that Meson supports which would circumvent that limit?

[jpakkane]

I don't think this behaviour is documented. Nor is it anything more special than "use rsp files if the line is too long and we know the compiler supports it". This is problematic to support automatically, since some tools support rsp files and others don't. You also might need to do different quoting with the rsp files as they are nonstandard and so on.

I guess a kwarg like use_rsp_file could be added and when set to true, Meson will transparently convert mycommand foo bar baz zzz into mycommand @targetname.rspfile. But that is functionality that does not exist.

[lhearachel]

What would it take to implement this? Could I contribute such a solution? Is there any reference material for how compilers are told to use response files? We'd really like to be able to extend this functionality to tools that are known to support it, especially homegrown ones.

[jpakkane]

The only reference material is ninjabackend.py, sadly. You can search for rspfile to find the methods that set up rsp files.

[bruchar1]

I did two attempts to allow the use of response files in custom_targets: #12272 and #11822. I'd be happy to rebase and finish one or the other, if I have somebody to review and merge it...

[lhearachel]

#12272 would be extremely useful for our use case!