Avoid too long Git command lines on Windows [WinError 206] #542
Conversation
|
Note: this isn't passing tests yet. I realise that my changes are munging the file paths and changing behaviour. Working on a better change |
akaihola
added
bug
|
I will test this properly tomorrow but looking at the diff it’s unlikely. I don’t have a problem with gitignore. The issue is that the list of files passed to the git diff call is too large and causes the command to exceed the max length for windows. All those files are Python files so are expected to be there. |
|
Just tried with the latest master and the error persists. To elaborate, the list of files passed to |
Svenito
force-pushed
the
cmd-length-limit-error
branch
from
096acb8 to
63a5b67
Compare
akaihola
changed the title
|
@Svenito I will now rebase this on |
On windows the command can _only_ be 32762 characters. This limit is reached when processing a large project. Instead of passing the whole list of files to the `git_get_modified_python_files` function, pass the paths instead with the repo root as cwd.
Use the black_compat equivalent instead
The test creates a bunch of files in a git repo which causes the subprocess call for getting modified files to exceed the max for Windows see [the docs](http://msdn.microsoft.com/en-us/library/ms682425%28VS.85%29.aspx) Also filter out the files that have been modified so that the list only includes the files that are supposed to be processed.
this reduces the number of files for the subprocess test from 1100 to 210. A bit more sane. Unfortunately we also fall foul of the Windows path length limit here, so we can't have any number of directories. Also mark the test for this bug to be skipped on non windows systems as it's not an issue there.
Use a single nested directory, not one for each file Refactor getting changed files to reduce number of temporary variables
Rename the large file list count test to long command test. Don't skip the long command test on other platforms
akaihola
force-pushed
the
cmd-length-limit-error
branch
from
9fd18cb to
c8c61e5
Compare
|
Lovely. Thank you. Glad to see it going in. |
As per #490 when running darker on Windows with a large codebase, subprocess fails to run with a large list of files.
This occurs because
git_get_modified_python_filesis called with a list of all files that are returned byfilter_python_files. For a large project this is a long list of files.What I attempted to do in this commit is to only process changed files, seeing as this is all we care about anyway. The filtering of the files is moved earlier in the process and is passed the directory or paths set on the commandline, instead of the list of files returned from the
filter_python_filescall. The resulting set is then intersected with all the files Black would process to generate a list of files that should be processed.While this alleviates the symptom, it could potentially still break if a large amount of files are modified between the current and previous revisions. That said, this is still an improvement on passing the entire list of files to
git_get_modified_python_files.