My pre-render script is suddenly causing an "Argument list too long" error, causing the quarto render to abort.

[Analect]

Description

I'm hitting this problem with a private gitlab-hosted repo, containing circa 1500 documents that get rendered with quarto. I'm not able to share this set-up, however the approach I'm taking is similar to what is shown publically here, where I'm scraping some document meta-data and pushing this to files in a data folder which are then published as resources in the rendered docs. I'm experimenting with alternative ways to work with the document meta-data and wanted to leverage the pre-render capability within quarto.

project:
  type: website
  resources:
    - "data/**/*"
    - "package/**/*"
    - "coi-serviceworker.min.js"
  pre-render:
    - scripts/metadata-scrape.py
    - scripts/load_data_kuzu.py
  render:
    - "*.qmd"
    ...

The gitlab-runner that is doing the rendering is based on this docker image, a debian 12 OS, with quarto 1.5.57 on-board. If I comment-out the pre-render scripts, then things run fine. However, when I enable the scripts/metadata-scrape.py on the gitlab repo (similar in pattern to this scripts/metadata-scrape.py, only longer to handle custom meta-data), I'm getting this Argument list too long error. Can you shed any light on why this might be happening when quarto is handling pre-render scripts.

Per this, I tried to set a longer command-line buffer with ulimit -s 65536 on the VM running this dockerized gitlab-runner, but also included in the .gitlab-ci.yml so that it gets applied within the runner itself (see image above), but none of these have helped.

[mcanouil]

Did you try to execute your Python script alone?

[Analect]

Yes, the script runs fine locally ... and if I run quarto preview, with the pre-render scripts enabled, that works too.

[mcanouil]

Your issue seems to be with your environment.
Did you try running your script alone in your environment instead of as a pre-render script?

[cscheid]

... huh. This is probably a limitation of the gitlab runner environment that we didn't foresee, because of the large number of input files. I think we'll need to communicate the input files through means other than an environment variable.

[Analect]

@cscheid ... I have been able to render this repo containing 1500+ docs for the past year. It's sometimes slow (60+ minutes), but it works. Once I introduced these pre-render scripts, then I started getting this Argument list too long. To @mcanouil point above, I tried to exclude environment factors by running up a container running the same image that is used to process my quarto repo and was able run the various scripts. There is definitely something happening with the CI job at the start of the quarto render process when a first pre-render script is being called. How does this input files env variable system currently work?

[cderv]

For reference, issue has been opened about this

• Large projects might cause "Argument list too long" error on pre-render scripts #10828

[Analect]

Thanks @cderv ... I hadn't noticed that.