mingw-w64-i686-oniguruma-5.9.6-1-any.pkg.tar.xz maximum size exceeded (??)

[nicowilliams]

https://ci.appveyor.com/project/stedolan/jq/build/1.0.17/job/xhouqgbcvmmx0xuy

:: Retrieving packages ...
error: failed retrieving file 'mingw-w64-i686-oniguruma-5.9.6-1-any.pkg.tar.xz' from repo.msys2.org : Maximum file size exceeded
error: failed retrieving file 'mingw-w64-i686-oniguruma-5.9.6-1-any.pkg.tar.xz' from downloads.sourceforge.net : Maximum file size exceeded
error: failed retrieving file 'mingw-w64-i686-oniguruma-5.9.6-1-any.pkg.tar.xz' from www2.futureware.at : Maximum file size exceeded
warning: failed to retrieve some files
error: failed retrieving file 'mingw-w64-x86_64-oniguruma-5.9.6-1-any.pkg.tar.xz' from repo.msys2.org : Maximum file size exceeded
error: failed retrieving file 'mingw-w64-x86_64-oniguruma-5.9.6-1-any.pkg.tar.xz' from downloads.sourceforge.net : Maximum file size exceeded
error: failed retrieving file 'mingw-w64-x86_64-oniguruma-5.9.6-1-any.pkg.tar.xz' from www2.futureware.at : The requested URL returned error: 404
warning: failed to retrieve some files
error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.

How do we fix this?

[FeodorFitsner]

Found few links that might be helpful:
https://bbs.archlinux.org/viewtopic.php?id=184864
https://www.reddit.com/r/archlinux/comments/2axo9x/unable_to_update_using_pacman_maximum_file_size/
https://chakraos.org/forum/viewtopic.php?id=12970

[nicowilliams]

@FeodorFitsner Well, OK, that's kinda depressing. It seems people break the packaging metadata all the time? And it leaves the rest of us to twiddle our thumbs and hope someone who can fix it does fix it. Great. Maybe we should put Oniguruma inside jq.

[nicowilliams]

Trying this:

+  - C:\msys64\usr\bin\bash -lc "pacman --needed --noconfirm -Syyu"
+  - C:\msys64\usr\bin\bash -lc "pacman --needed --noconfirm -Sc"

[nicowilliams]

That's causing pacman to go very slowly. I wonder if I'll hit a timeout.

https://ci.appveyor.com/project/nicowilliams/jq/build/job/xb160ymi5xr1hnf9

If it's just a matter of upgrading pacman, can AppVeyor just do that a priori? Probably not. Can caching help?

[nicowilliams]

E.g.,

error: failed retrieving file 'mingw-w64-i686-gcc-fortran-5.3.0-1-any.pkg.tar.xz' from downloads.sourceforge.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
warning: failed to retrieve some files
downloading mingw-w64-i686-gcc-objc-5.3.0-1-any.pkg.tar.xz...

Who knows when this will finish.

[nicowilliams]

Is the slowness something to do with the appveyor build environment? Is there anything I can do to make this faster?

[FeodorFitsner]

Look, we are going to update build worker images this week (Node.js, git, etc.). Give me a list of commands to update the whole thing.

[nicowilliams]

@FeodorFitsner

Thanks. I think it's this:

C:\msys64\usr\bin\bash -lc "pacman --needed --noconfirm -Syyu"
C:\msys64\usr\bin\bash -lc "pacman --needed --noconfirm -Sc"

(From the first link you pasted.)

I don't know yet that it will work. I'm watching it go (though since it's like watching paint dry, i'll look again in a few hours; I'm guessing it will have timed out). https://ci.appveyor.com/project/nicowilliams/jq/build/job/xb160ymi5xr1hnf9

[FeodorFitsner]

I don't think it's because of slowness of build environment. You are building on VM with 2 vCPUs and 7 GB of RAM. Access to download locations maybe slow (depending on location/mirror, especially SF has been always a pain) - I don't know if you can control mirrors.

[nicowilliams]

The need to upgrade strikes me as odd, FYI. If you didn't touch the pacman.conf then everything should have kept working, and that it broke would mean that the pacman repo maintainers are doing something very wrong.

[FeodorFitsner]

I haven't touched pacman.conf - I even don't know where it's located :)

[FeodorFitsner]

We can try re-installing msys64 entirely, if that help.

[nicowilliams]

I don't either! Heh.

[nicowilliams]

Is chocolatey available? Maybe I should use that...

[FeodorFitsner]

You can give Chocolatey a try, of course.

[nicowilliams]

Alright, I'll try it tonight or tomorrow. Thanks.

[FeodorFitsner]

OK, look at this issue: #352 - this is how it was installed in the first place. Also, people recommend for update doing this: #597 - look how mirror is changed from SF.

[FeodorFitsner]

You can logon via RDP. Remove c:\msys64 completely (I don't remember if there is an uninstaller) then install it again using commands from those issue and see how that goes for your build.

If that fixes your issue then we definitely need to reinstall it.

[nicowilliams]

Oh, maybe the problem is the -Sy* options.

[nicowilliams]

In fairness, I copied this from some other appveyor.yml, cargo-cult style. (Because let's face, it, there's only so many hours in the day. But there's always a price to cargo-culting things.)

[nicowilliams]

It seems this is unfixable with pacman. Here's what I get (I've tried every reasonable combination of pacman commands) from pacman -Syyu mingw-w64-i686-oniguruma mingw-w64-x86_64-oniguruma:

https://ci.appveyor.com/project/nicowilliams/jq/build/job/mp0mx2uftiqf0tqp

error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.
Command exited with code 1

Chocolatey doesn't have a package handy for Oniguruma. So I can't use that. I can't use pacman because it broke.

I guess we just have to grab the oniguruma tarball (which is now hard to find) and build it in the appveyor config. This fails:

bash -lc "exec 0</dev/null && cd $APPVEYOR_BUILD_FOLDER && mkdir deps && cd oniguruma-5.9.6 && touch NEWS ChangeLog && autoreconf -i && ./configure --prefix=$OLDPWD/deps/ && make install"
main::scan_file() called too early to check prototype at /usr/bin/aclocal-1.11 line 644.
aclocal-1.11: couldn't open directory 'm4': No such file or directory
autoreconf: aclocal failed with exit status: 1
Command exited with code 1

To say that this is frustrating is an understatement. I can't even find who's responsible for pacman so I can file an issue. All I know is that the build was working, then something changed in the pacman mirrors, and now I can't build. None of this is AppVeyor's fault, just some anonymous third party's.

@FeodorFitsner I don't know what commands you'd run to update pacman, beyond this: pacman -Syyu, but I've no idea if it will work for you -- for me this always fails.

[nicowilliams]

@FeodorFitsner Maybe you should just re-do the MSYS and pacman installs. Maybe that will work (until it breaks again).

[nicowilliams]

Oh, and update-core fails too:

error: failed retrieving file 'msys2-runtime-devel-2.4.0.16836.5ea1fb5-1-x86_64.pkg.tar.xz' from downloads.sourceforge.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
warning: failed to retrieve some files
error: failed to commit transaction (download library error)
Errors occurred, no packages were upgraded.
==> ERROR: 'pacman' failed to update core packages.
Command exited with code 1

[nicowilliams]

Tomorrow I may try to fix pacman.conf per https://www.reddit.com/r/archlinux/comments/2axo9x/unable_to_update_using_pacman_maximum_file_size/ to use curl and not enforce the known file length. I suspect that too won't work.

[nicowilliams]

Changing the XferCommand in /etc/pacman.conf helped, but now the checksums don't match and -sensibly!- there is no way to disable that check. It seems that the pkgs really did change in the repositories but corresponding changes to the databases weren't made.

Oh, hmmm, looking more carefully at the pacman --debug output... it seems that the pacman executable only likes to run from the Windows cmd shell, not from either a Cygwin or MSYS bash shell... Retrying... very different output this time, but I ran pacman -Syyu ..., so it's going pretty slow.

[nicowilliams]

Nope, didn't work.

I managed to find the name of the pkg maintainer: @Alexpux. And I found this: Alexpux/MSYS2-pacman#12.

@Alexpux The summary is that the oniguruma pkgs for MSYS have the wrong checksum now, so https://github.com/stedolan/jq's AppVeyor build fails. Any help would be greatly appreciated.

[FeodorFitsner]

I'll re-install it and publish a test image for you to try.

[nicowilliams]

Thanks. Let me know here when you've done that. If you run into the same problem then give up (EDIT: but let me know either way!). I'll just figure out how to build Oniguruma on Windows and be done.

[FeodorFitsner]

That's weird! I've reinstalled msys2 completely using these instructions: #597 (comment)

Running on "test" image (it's not public yet, but it's available under your account), but installing "mingw-w64-i686-oniguruma" fails with the same error: https://ci.appveyor.com/project/FeodorFitsner/simple-console/build/1.0.1627

I tend to believe this is something package-specific.

[nicowilliams]

@FeodorFitsner Yup! The breakage is upstream. I'll just figure out how to build Oniguruma on Windows from the jq appveyor.yml (I'll have to patch some things, no doubt), and in the worst case scenario I'll use a cross-compiled, pre-built version (since I know how to do that) that I could probably host on github.

We can close this issue, but I'll let you do it as you might want to apply some label to indicate that this is not an AppVeyor CI issue but an issue elsewhere.

Thanks for trying!

[FeodorFitsner]

OK. Anyway, we've updated MSYS 2 which is good :)

[jankatins]

@nicowilliams oniguruma got an updated package into msys, 6 days ago, maybe that will help?

-> https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-oniguruma

[jankatins]

... and it does: jqlang/jq#1132