Bundle command does not work for versions after ruby/[email protected] for windows

[kappratiksha]

Failed workflow - https://github.com/GoogleCloudPlatform/functions-framework-ruby/runs/7795974033?check_suite_focus=true
The above workflow is with @v1 and it fails when we run "bundle install && gem install --no-document toys" with the error - bundle: command not found

The workflow runs fine for v1.112.0 : https://github.com/GoogleCloudPlatform/functions-framework-ruby/runs/7813891709?check_suite_focus=true
However fails for versions after that.
Here is the run for v1.113.0: https://github.com/GoogleCloudPlatform/functions-framework-ruby/runs/7813819674?check_suite_focus=true

[eregon]

Interesting, maybe the bundle.bat or so is missing on Windows in RubyInstaller2 builds?
@MSP-Greg Could you take a look?

(and it worked in 1.112.0 because of gem install bundler essentially, but that should not be necessary)

[eregon]

What's odd is it does work in this repo's tests: https://github.com/ruby/setup-ruby/runs/7705639587?check_suite_focus=true#step:17:1

Maybe the bundle install && gem install --no-document toys (in shell: bash) is causing it.
Could you split that in 2 steps?

BTW you can also use bundler-cache: true for faster gem installation.

[MSP-Greg]

@kappratiksha

I think you need to remove shell: bash from the Actions workflow step. The copy of 3.0.4 that I have does not have a plain bundle file, which is what's needed to run in the MSYS2 bash shell.

EDIT: I just checked my copy of 3.1.2, and it does have the file. Not.sure.why. @larskanis ?

[eregon]

I'm trying to add a which bundle test (inside shell: bash) in https://github.com/eregon/setup-ruby/runs/7843304859?check_suite_focus=true
Let's see for which cases it works/fails (currently running, not done yet).

[MSP-Greg]

JFYI, I can't look into it right now, but at one point bin files were installed that would run from a bash script or a Windows command. They had a .bat or .cmd extension.

So there was no bundle file, just the 'dual-purpose' file. That obviously caused problems with cross-platform scripts and workflows that just used bundle...

EDIT: Just checked, Windows Ruby 3.0.4 has the dual-purpose scripts, 3.1.2 has the normal scripts, where bundle.bat starts ruby the bundle script.

[eregon]

So the versions which fail are 3.0, 3.1, head, mswin.
I guess those all don't have a bin/bundle file, yet previous versions did.
And gem install bundler creates bin/bundle.
So I guess that's a RI2 or CRuby bug?

[MSP-Greg]

I guess those all don't have a bin/bundle file, yet previous versions did.

mswin has the file, buts it's the dual-purpose file. ruby-loco's mingw and ucrt builds have a lot of custom code in them. It's not broke, so I've never removed it.

But, its mswin build is standard script, similar to what's used in ruby/ruby CI. So, I think the issue is with CRuby. But, I need to look at it, because making it work is a bit tricky if one is running the bundle command from a ruby that isn't the env Ruby.

[MSP-Greg]

@kappratiksha

See GoogleCloudPlatform/functions-framework-ruby#142 for a fix.

[msteinbeck]

@MSP-Greg

My job also fails due to missing bundle: https://github.com/msteinbeck/tinyspline/runs/7902080504?check_suite_focus=true

@kappratiksha

See GoogleCloudPlatform/functions-framework-ruby#142 for a fix.

What is the actual fix? I don't cannot it.

[dentarg]

@msteinbeck I think the fix is to remove this line: https://github.com/msteinbeck/tinyspline/blob/b96a2d334c4be5282078d42a61d9105136f63cab/.github/workflows/ci.yml#L585 (i.e. don't use bash)

Also, you can remove the line with bundle install if you add bundler-cache: true to the setup-ruby step as then this action will run bundle install as part of caching the gems

[msteinbeck]

Removing shell: bash doesn't fix this issue but adding bundler-cache: true does.

[eregon]

Issue for RubyInstaller2: oneclick/rubyinstaller2#299 and for CRuby: https://bugs.ruby-lang.org/issues/18970

I guess for now we'll to workaround in setup-ruby by force-installing Bundler on Windows :/

[MSP-Greg]

@eregon

Fixed this in mswin. To keep the install portable, bash binstubs start with a shebang line, which is the same as the ucrt & mingw builds.

When running the 'Windows' binstubs (cmd or bat files), they will run with the Ruby exe file in the same folder. I haven't figured out a way to do so with the bash binstubs, so they use the ENV/PATH ruby.

[eregon]

When running the 'Windows' binstubs (cmd or bat files), they will run with the Ruby exe file in the same folder. I haven't figured out a way to do so with the bash binstubs, so they use the ENV/PATH ruby.

The :""||{ ""=> %q<-*- ruby -*-... header seems actually valid Bash/cmd/ruby: https://bugs.ruby-lang.org/issues/18970#note-2
So I guess the main issue for bin/bundle of mswin is it's not seen as executable by ls -l in bash.

[MSP-Greg]

So I guess the main issue for bin/bundle of mswin is it's not seen as executable by ls -l in bash.

I thought the exec flag was ignored in Windows. JFYI, I added a step to the mswin build CI to test the bash binstubs. Script is here, most recent CI log is here.

I guess I could fix it in the build scripts...

[eregon]

I thought the exec flag was ignored in Windows

Yeah, but apparently there is some handling in Bash or so to decide if a file is executable (maybe it checks if there is a .cmd file next to it, I'm unsure how it works), and which seems to only consider executable files.

[MSP-Greg]

which seems to only consider executable files

Correct. Fixed.

$ which bundle
/c/ruby-ucrt/bin/bundle

$ PATH=/c/ruby-mswin/bin:$PATH

$ which bundle
/c/ruby-mswin/bin/bundle

EDIT: I forgot to mention...

When I tried to get the existing bash binstub working, it appeared to work fine in a cmd shell, but when using PowerShell, it opened a cmd window. So, given that many Windows developers probably use PowerShell, I decided to use a standard bash binstub.

[eregon]

When I tried to get the existing bash binstub working, it appeared to work fine in a cmd shell, but when using PowerShell, it opened a cmd window. So, given that many Windows developers probably use PowerShell, I decided to use a standard bash binstub.

By the existing bash binstub, you mean the :""||{ ""=> %q<-*- ruby -*-..., right? I think that's an additional reason to not use that special header for https://bugs.ruby-lang.org/issues/18970 then.

[MSP-Greg]

@eregon

I'll move to the Ruby issue 18970.

I just ran this in my fork, and all the ruby-loco builds passed. Actions has been squirrelly recent, and I've seen intermittent failures. Not sure what the issue is, might be network related.

Anyway, any thoughts on not installing bundler on the ruby-loco builds? I really don't like installing Bundler in master builds...

[eregon]

Anyway, any thoughts on not installing bundler on the ruby-loco builds? I really don't like installing Bundler in master builds...

OK, I'll remove that and only do it for 3.0+ releases, and head but not other head builds.

[MSP-Greg]

Thanks.

[MSP-Greg]

Using Windows bash shell (either Git or MSYS2), which does not return information as expected. It will show a x executable flag, but that flag is ignored when running commands, which is based solely on Path if the command is run without reference to a folder.

Some console output below, /c/ruby-ucrt/bin/bundle is in Path, and I added puts RUBY_DESCRIPTION to both bundle bash files:

$ ruby -v
ruby 3.2.0dev (2022-08-25T15:49:50Z master b2d0f78869) [x64-mingw-ucrt]

$ which bundle
/c/ruby-ucrt/bin/bundle

$ bundle
ruby 3.2.0dev (2022-08-25T15:49:50Z master b2d0f78869) [x64-mingw-ucrt]
Could not locate Gemfile

$ /c/ruby-mswin/bin/bundle
ruby 3.2.0dev (2022-08-25T15:49:50Z master b2d0f78869) [x64-mswin64_140]
Could not locate Gemfile

$ ls -l /c/ruby-mswin/bin/bundle
-rw-r--r-- 1 <user> <user> 653 Aug 25 22:13 /c/ruby-mswin/bin/bundle

$ PATH=/c/ruby-mswin/bin:$PATH

$ ruby -v
ruby 3.2.0dev (2022-08-25T15:49:50Z master b2d0f78869) [x64-mswin64_140]

$ bundle
ruby 3.2.0dev (2022-08-25T15:49:50Z master b2d0f78869) [x64-mswin64_140]
Could not locate Gemfile

$ which bundle
/c/ruby-ucrt/bin/bundle

$ where bundle
C:\ruby-mswin\bin\bundle
C:\ruby-mswin\bin\bundle.bat
C:\ruby-ucrt\bin\bundle
C:\ruby-ucrt\bin\bundle.bat

[eregon]

I'm not sure what to understand from this output but it seems somehow the ruby-mswin bundle is not recognized as executable, and most likely because of that is also then ignored by which. But bundle will still pick up the ruby-mswin bundle after it was prepended to PATH.
So I guess the conclusion is "don't use which on Windows it's kinda broken/inconsistent with what will get run"?
One problem is it's not rare that Ruby or shell programs use which, and so it can cause actual issues within Bash on Windows.

BTW I noticed that ruby-mswin fails the which bundle test: https://github.com/ruby/setup-ruby/runs/8049444333?check_suite_focus=true

What's needed to make ruby-mswin seen as executable/found by which?

[MSP-Greg]

What's needed to make ruby-mswin seen as executable/found by which?

The Windows file system does not natively support 'executable'. Using CLI or Ruby's chmod to set it doesn't raise an error, but doesn't seem to change anything. If one googles it, a lot of info about how chmod isn't reliable on Windows, either with the MSYS2 or Git shells.

Obviously some files do show 'executable' when using ls -l, and it matches what which finds. I spent too much time screwing around with it, but maybe later this weekend I'll see if a fresh look helps...

[MSP-Greg]

@eregon

Found an interesting item:

MSYS2, from which Git for Windows is built on, virtually adds the "execution bit" to file permissions for *.exe files and for files that starts with a shebang, for example #!/usr/bin/env bash, but they are never stored in the file system.

So, the current bash files in mswin start with the following:

{
  bindir=$(dirname "$0")
  exec "$bindir/ruby" "-x" "$0" "$@"
}
#!/usr/bin/env ruby

I changed that to:

#!
{
  bindir=$(dirname "$0")
  exec "$bindir/ruby" "-x" "$0" "$@"
}
#!/usr/bin/env ruby

Now, which works, and the code is called correctly by the Windows binstub, which use ruby -x. I only changed the mswin bash binstubs, as I left the mingw & ucrt builds the same. And, they start with a normal shebang line.

So, I haven't changed it yet, but I may change all the ruby-loco builds to the above...