-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC submission for override_gem_dependency_versions #13
Conversation
|
||
The proposed solution clearly documents that the problematic gem is `puma` and that we are explicitly overriding it's version constraints for the `rack` gem. | ||
|
||
The impact of this is verbose and possibly unsightly code. The benefit, however, is that version overrides are explicitly documented which will help prevent excessive issues being lodged to bundler and to gem authors. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it also could make the implementation of the feature much more difficult
Have you seen rubygems/bundler#5670, by any chance? |
@segiddins I hadn't seen your PR, but I had seen people suggest the |
Definitely! I just wanted to make sure you had seen a (mostly) working implementation of an alternative, especially since it might be a good jumping-off point for playing around with the different DSL options |
@segiddins it's great that there is a working implementation. What exactly isn't working the way you would expect? Also, did I read correctly that an exact version must be supplied? Whats the rationale behind this over using version constraints and negotiating with other gems? |
I think my thinking at the time was that it would help keep resolution more predictable -- by restricting it to only exact versions, you would never end up with a conflict on that particular gem. It also requires the override to be very explicit (I need this version), versus just changing the requirements around (which could end up matching different versions over time, with no change in the Gemfile). Basically, I wanted this to be a very precise hammer to break out of the normal behavior of Bundler, and I wanted it to be targeted, so there would be minimal room for surprising (or edge-case-y behavior) |
@segiddins yeah sure that makes sense. |
I understand the use case for this but at the same time this scares me completely 😱 |
* Maintaining overridden version constraints within the Gemfile saves a significant amount of development cost when compared to maintaining a forked repository | ||
* Explicit syntax within the Gemfile with custom notes and obvious output messages when running `bundle install` can help to mitigate developers not knowing that a version conflict exists | ||
|
||
# Rationale and Alternatives |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is a good example of the problem in action: salesforce-marketingcloud/FuelSDK-Ruby#115.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm sorry, but I don't get it. But what makes it actual good example?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would you prefer to maintain an entire fork of a gem just to update it's Gemfile, when you could just add an override in your own Gemfile?
The good things about overriding in your own Gemfile are:
gem install
can explicitly show that a problematic gems dependencies are being forced to be a version that isn't technically supported by that gem. This couldn't happen if you maintain a fork obviously. Maybe you can tell it's not the original gem because you can see the link to a GitHub repo, but it doesn't tell people why you are doing this and what the condition is for moving back to the original gem. Maybe you could add a code comment the explains all this, but that's not very semantic. You couldn't use a code tool to read the comments to tell you when you should start using the original gem again.gem install
can explicitly show that none of the problematic gem's dependencies' versions are being forced. If after agem update
this message is being show, it indicates that the original gem author has finally updated their problematic gem dependency version constraints, which means you can finally remove the override from your Gemfile. This wouldn't happen if you maintain a fork. You would need to constantly actually look at the original gem for updates. That is such a manual chore.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would you prefer to maintain an entire fork of a gem just to update it's Gemfile, when you could just add an override in your own Gemfile?
I would prefer to maintain an entire fork of a unmaintained gem or actually try to find out alternative to this gem to remove abandoned gem in all codebases I do maintain.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer to maintain an entire fork of a unmaintained gem or actually try to find out alternative to this gem to remove abandoned gem in all codebases I do maintain.
Ok, but why?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer to maintain an entire fork of a unmaintained gem or actually try to find out alternative to this gem to remove abandoned gem in all codebases I do maintain.
For the cases where that apply to us, we have found maintaining and tracking occasionally updated (but not fully abandoned) gems, ends up being a thankless chore that creates more problems than it solves. This of course doesn't mean it's right for everyone else.
This is also a common feature in other dependency managers:
- NPM (javascript): https://nodejs.org/en/blog/npm/managing-node-js-dependencies-with-shrinkwrap/
- NPM (javascript): https://www.npmjs.com/package/npm-force-resolutions
- Yarn (javascript): https://classic.yarnpkg.com/en/docs/selective-version-resolutions/
- Pip (python): https://pip.pypa.io/en/stable/user_guide/#constraints-files
- Maven (java): https://www.baeldung.com/maven-version-collision
- Gradle (java): https://docs.gradle.org/current/userguide/resolution_rules.html
The above cover the top languages from Github and all have a solution. I would argue this is not an esoteric need.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wrote up my view on the problem and options for solutions that are currently available, 1 that is this RFC and 1 other that is not available.
I hope this helps to frame the debate.
Options for handling out of date gems
Out of date gems have been a problem for a long time. As the ruby ecosystem evolves and ages, they are and will become more common. This is especially prevalent in areas where the state of the art has moved on (for example SOAP).
Unfortunately, these 'stuck' gems often have dependencies on other more common and evolving gems. This would not be a problem if the un-evolved gems had loose dependency constraints. However, the recommendation for gem writers is to be relatively tight on constraints to ensure compatibility.
We now have stuck gems depending on other gems. Since only one version of a gem can be installed in bundler. This then constrains the use of newer gems that share a downstream gem dependency (but not the dependency version).
There are a number of options for handling these out of date gems. Here I list them out and highlight their pros and cons.
Finally, I give my recommendation for consideration.
Here are the many options that developers have to deal with out of date gems:
- Create an issue
- Create issue and develop a PR
- Monkeypatch
- Fork and use git path
- Fork and use private registry
- Fork and change name and alias name
- Take over maintenance of the gem
- Force version
- Write your own
Here is a detailed description of each of them and their pros and cons:
1. Create an issue
Create an issue on the relevant repo and wait for the maintainer to handle it and release a new version. This and the next option are the happy path, the way things should work.
This option and the following option are usually the first port of call, since they are also the easiest.
Pros
- Keeps gem up to date for the community
- Easy for the user
Cons
- No help if the gem isn't being maintained
- Have to wait for maintainer to agree to and then make the change a priority
- Assumes gem is maintained in timely manner
- Maintainer has to do the work
2. Create issue and submit a PR
Create the issue but also provide a solution in the form of a PR. Again, this is the happy path, but isn't always available.
Pros
- Keeps gem up to date for the community
- A ready made solution is in place so the maintainer has less work
- Relatively easy for some users
Cons
- No help if the gem isn't being maintained
- Have to wait for maintainer to make it a priority
- Assumes gem is maintained in timely manner
- You have to do the work, and if you don't understand the code, figure it out.
3. Monkeypatch
Create a gem or an internal piece of code that overrides the buggy or out of date behavior in the gem that needs updating. An example of this is Savon Fixes which I created.
Pros
- Make the change without having to get the maintainers to do anything
- If a gem, can be provided to the community
Cons
- Could cause missing out on gem updates
- Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
- Fragile (especially if a gem; a monkeypatch gem is not straightforward)
- Doesn't work for out of date dependencies as
gemspecs
can't be monkeypatched - Depending on load order, can be very tricky to verify that it is working if deeply embedded in a chain of gems
4. Fork and use git path
This is the most common way of forking. Create the fork, make the patch and point your gem file at your git repo and the relevant commit/branch.
Pros
- Can make targeted changes of whatever nature is needed
Cons
- Maintaining a full code base (you have to do the work, and if you don't understand the code, figure it out)
- Requires discipline. I've seen additional functionality added to the forked gem, thus creating barriers to tracking upstream.
- Could cause missing out on gem updates
- Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
- Can't have all gems in a registry
5. Fork and use private registry
Create the fork, make the patch, add in your proprietary CI/CD and registry config/code and point your Gemfile
at the internal registry.
Pros
- Can make targeted changes of whatever nature is needed
Cons
- Maintaining a full code base (you have to do the work, and if you don't understand the code, figure it out)
- Requires discipline. I've seen additional functionality added to the forked gem, thus creating barriers to tracking upstream.
- Could cause missing out on gem updates
- Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
- Having to add internal CI config/code to the public repo
- Having to create new version number to avoid clash with public version (so multiple sources aren't ambiguous)
- If there are intermediate gems, having to make sure they will accept the new version number
6. Public fork and change name
In this case you are forking the gem, and then changing it's name so that you can publish to Rubygems
under the alternate name. Then your Gemfile
points to the newly named gem.
Pros
- Can use standard publishing mechanisms (
Rubygems
) - Community can benefit from changes (if aware)
Cons
- In order for intermediate gems in a dependency chain to use, would require
alias_gem_name
fuctionality toBundler
which is not currently available - Multiple versions of same gem with similar names will exist
- Could cause missing out on original gem updates
7. Take over maintenance of the gem
In this case, become the maintainer of the gem as the previous maintainer no longer has the resources to spend the time on maintenance. Then you can make the necessary updates.
Pros
- Keeps gem up to date for the community
- Gem is maintained again
- Strong way to give back to the community
Cons
- Big task to take on
- Maintaining a full code base (you have to do the work, and if you don't understand the code, figure it out)
- When maintainer has disappeared, taking over maintainership is not easy
- Maintainer may not be willing to give up ownership, even as they no longer make changes. (I have an example of this.)
- To be a universal approach, requires more robust or frequent 'change of ownership' process for
Rubygems.org
8. Force version
Add a capability Bundler
which allows an option in a Gemfile
. This option lets the user force the version of a dependency which then overrides that dependency in the gemspecs
of all dependent gems.
Pros
- Allows gem dependencies to be forced when the gem is still working (other than the out of date dependencies)
- Works on dependencies at any level in a dependency chain
- Javascript (nvm and yarn), Python (pip) and Java (Maven, Gradle), the top 3 Github languages, offer this capability
Cons
- Could cause missing out on gem updates
- Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
- Change to
Bundler
api (though opt in) - Does not solve cases where functionality/API changes are required
9. Write your own
This just means avoiding the 'stuck' gem entirely.
Pros
- Full control
Cons
- Not feasible in situations where the capabilities are not a strength of yours (in our case, SOAP)
- Does not solve problems for intermediate gems in a dependency chain
- If private, no benefit to or from the community
- If public, have to popularize your version so that all gem maintainers switch
Recommendation
Given the complexity and how painful many of the options are, I believe adding 8) Force Version to Bundler
is the best solution for the dependency side of these problems.
Additionally, I think the Bundler
team should consider an alias gem name
function that would allow users to redirect to a forked version of a gem which they can make public without having to fork entire chains of gems.
References
Dependency managers offering override functionality:
-
NPM (javascript): https://nodejs.org/en/blog/npm/managing-node-js-dependencies-with-shrinkwrap/
-
NPM (javascript): https://www.npmjs.com/package/npm-force-resolutions
-
Yarn (javascript): https://classic.yarnpkg.com/en/docs/selective-version-resolutions/
-
Pip (python): https://pip.pypa.io/en/stable/user_guide/#constraints-files
-
Maven (java): https://www.baeldung.com/maven-version-collision
-
Gradle (java): https://docs.gradle.org/current/userguide/resolution_rules.html
-
Top languages in Github: https://madnight.github.io/githut/#/pull_requests/2020/3
-
Google search for 'bundler override gem dependency site:stackoverflow.com' returns 1,360 results: https://www.google.com/search?q=bundler+override+gem+dependency+site:stackoverflow.com
@colby-swandale it shouldn't :) It should be a perfectly reasonable thing to do. Just imagine that this was implemented and the user could override a gem dependency version. When performing a
Or 2. A gem dependency version constraint is getting overridden and the chosen version is within the bounds of the gem authors constraint. A friendly text can be displayed telling the user to remove the override.
This is much safer and easier than the current hack of forking repos just to fix stupid version constraints imposed by gem authors. When the user forks a repo, they need to manually check the original gem for updates. They could potentially miss out on security updates or what not if they aren't diligent. However, being able to add an override, the user gets nice messages indicating whether the override can be removed or not. A simple It couldn't be simpler or more intuitive if you tried ;) |
This is the best design because it is very explicit and clearly documents what is happening. | ||
It requires project maintainers to have a clear understanding of the conflicts they are encountering. | ||
|
||
Other designs are as follows: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After looking at the original design by @segiddins and now ported to bundler/master here: rubygems/rubygems#4178, I think the force_version
syntax is better. I couldn't really see any advantages to the more complex implementation other than a very specific message. My vote is that force_version
provides most of the benefit with a simpler syntax.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is sample output from the force_version
PR:
$ dbundle
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Using public_suffix 4.0.6
Using builder 3.2.4
Using mini_portile2 2.4.0
Using bundler 2.3.0.dev
Using rack 2.2.3
Using socksify 1.7.1
Using json 1.8.6
Using jwt 1.5.6
Using nori 2.6.0
Using addressable 2.7.0
Using gyoku 1.3.1
Using nokogiri 1.10.10
Using httpi 2.4.5
Using akami 1.3.1
Using wasabi 3.6.1
Using savon 2.12.1 [version forced]
Using sfmc-fuelsdk-ruby 1.3.0
Bundle complete! 2 Gemfile dependencies, 17 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
This also means I no longer have to maintain a forked version of sfmc-fuelsdk-ruby
.
Here is the Gemfile
:
source 'https://rubygems.org'
gem "sfmc-fuelsdk-ruby", "1.3.0"
gem "savon", "2.12.1", force_version: true
Here is the Gemfile.lock
:
GEM
remote: https://rubygems.org/
specs:
addressable (2.7.0)
public_suffix (>= 2.0.2, < 5.0)
akami (1.3.1)
gyoku (>= 0.4.0)
nokogiri
builder (3.2.4)
gyoku (1.3.1)
builder (>= 2.1.2)
httpi (2.4.5)
rack
socksify
json (1.8.6)
jwt (1.5.6)
mini_portile2 (2.4.0)
nokogiri (1.10.10)
mini_portile2 (~> 2.4.0)
nori (2.6.0)
public_suffix (4.0.6)
rack (2.2.3)
savon (2.12.1)
akami (~> 1.2)
builder (>= 2.1.2)
gyoku (~> 1.2)
httpi (~> 2.3)
nokogiri (>= 1.8.1)
nori (~> 2.4)
wasabi (~> 3.4)
sfmc-fuelsdk-ruby (1.3.0)
json (~> 1.8, >= 1.8.1)
jwt (~> 1.0, >= 1.0.0)
savon (= 2.2.0)
socksify (1.7.1)
wasabi (3.6.1)
addressable
httpi (~> 2.0)
nokogiri (>= 1.4.2)
PLATFORMS
x86_64-darwin-19
DEPENDENCIES
savon (= 2.12.1)
sfmc-fuelsdk-ruby (= 1.3.0)
BUNDLED WITH
2.3.0.dev
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My implementation in rubygems/rubygems#4178 is now fulling working. This is a port of @segiddins original POC. I have created additional tests, as well as run it against our complex rails project at http://https://github.com/simplybusiness where it has worked well. @br3nt, would welcome your thoughts.
This feature would have a lot of value for us as a company. It is also opt in so no one is :forced
to use it ;-).
@br3nt I've added a bunch of comments and a working implementation. I can't PR your PR but feel free to incorporate my feedback into the content of the RFC, especially the fact this is a common capability in package managers. Not exactly sure how this moves forward, but hopefully a fully functioning implementation helps! |
Having spoken with the maintainers of How to override gem dependenciesNote This attempts to document that recommended approach for overriding gem dependencies which are not being updated by gem maintainers. This documentation is based on what has been recommended by the core When a gem has dependencies that are not being updated (or really any changes that should be made) in a timely manner, the community should attempt to move forward by proposing the changes. The best way to do this is to
Note This documentation only seeks to provide a solution for (Github)[https://www.github.com] (or at least Steps
This will provide you with a patched version of the gem which can then be replaced when the PR is eventually incorporated into the original gem. |
Suggestion for DSL:
This syntax could co-exist with yielding a decorated spec for more elaborate tinkering. |
This RFC was AFAIK widely rejected by various RubyGems maintainers at various places (Slack, Issues, ...). Would it be ok to close this RFC and mark as rejected? |
I'm OK with it being closed, but I think it would be good to incorporate the documentation I wrote here into the official documentation. |
I don't think this should be part of official documentation. We should suggest to reach the project and ask for update or takeover over the maintenance (you can use also new adoption center at rubygems.org). Forking the project and using custom version is really last option here and it should be done only temporarily (not as a final solution) meanwhile the project is revived and maintained again or until you remove this unmaintained dependency from your project. |
The problem always was the second level dependency. Not the first level dependency. I'm not familiar with the adoption center so maybe that really is the answer. Because as it stood, we were forking loads of gems to just update dependencies. Anyways, all this was discussed at length at the time. |
Seriously, why would anyone want to reject and close this issue?? This is a real problem that will only ever increase as more gems become abandoned. Of course current gem authors don’t agree with this… they can’t imagine a time when their gem will hit the graveyard. But sure enough it’s highly likely that at some point their gem will become abandoned and the poor saps who have a dependency on those gems will want a way to override the crazy overly restrictive constraints that gem authors often impose. Unless someone has come up with a better way of solving this real problem, I propose the community work together and push this solution to completion. At the very least, the community needs this feature, or a feature like it, to get past all the security bugs from old abandoned gems that are still useful. Prove me wrong or link to a better option. |
Just as an add on to this… forks, merge requests, and any other such hacks are not viable solutions to get around dependency constraints from poorly maintained or abandoned gems. Not only does this add extra development burden to every project using the abandoned or poorly maintained gem. It also fragments support! If everyone followed this advice, there could be 1000s of forks for a single gem all making the same change to its constraints. Just crazy! It’s wishful thinking that an abandoned or poorly maintained gem will get a new maintainer. The community needs a simple easy way to override dependencies for the brief period required to replace the problematic gem. |
I seriously don’t understand what all the fuss is about. There hasn’t been a good argument against this proposal. |
I just read about the Bundler teams recommendations documented in a comment by @lukaso on 1 Feb 2021. This is seriously bad advice and bad development practices for the reasons I stated above. Just because the Bundler team support and endorse this practice doesn’t make it good or right. Just because it works doesn’t make it a good solution. How many times do you think an abandoned and poorly maintained gem will be forked just so it’s dependencies can be fixed? Max of once for every app using the gem. Now what happens when people use one of these forks so they don’t have to fork it them selves? What happens when that fork becomes abandoned or poorly maintained? Let’s call this scenario forking hell. I’ll say it again… it’s terrible advice and bad practice. We have a far superior and versatile solution in this proposal. The community just needs to push it to completion. |
Our take is that encouraging fixing this upstream (for everyone) and not just for yourself actually saves work for everyone. Yes, the first person that proposes a solution upstream has to do some extra work, but once there's a working PR, the changes needed for every other user are about the same either with a new feature, or with existing features. Basically, gem 'other_gem' do |s|
s.override_runtime_dependency('json', '>= 1.7.7', '<3')
end vs gem 'other_gem', github: "https://github.com/<other_gem_org>/<other_gem_name>/pull/<pull_number>" I worked with ruby libraries for many years and in real life there's no such issue as "fork hell", there's hardly ever competing PRs proposing the same dependency changes, and when that happens they get unified quickly. Let's close this and focus on other things, thanks for the discussion! |
In regards to this vs This recommendation works with the constraints system following the same rules and behaviours. That is, the constraint of the dependency is overridden with a new constraint. I think this would be more useful for gem authors who want to slowly ween off an abandoned gem dependency. Imagine if a gem author used By using this approach, a looser constraint can be applied as an override. This would allow the underlying dependency to continue to develop and the end app will get the newer versions. Perfect! |
The notion that this is closed as unnecessary when it's supported in virtually every other package manager, is quite frankly, bonkers. Many of us use this exact feature in other package managers, and it works perfectly without the feared explosions and mayhem alluded to here. It's absolutely brilliant to be able to add a At this point, I am much closer to using a forked version of bundler with @lukaso's PR than I am a fork of every problematic dependency/sub-dependency in my life.
You guys, the work on rubygems/rubygems#4178 appears nearly finished, either as-is or with a bit more discussion. I can understand maintainers' apprehension about a breaking change, but again, this is an opt-in change that no one is being forced to use, but so many would benefit from. Please consider reopening this, I've spent dozens of hours trying to find workarounds for this, only to find my own comments gathering dust, posted at 3-year intervals. I'm even happy to be a part of the process myself if it would help move it along. |
Your underlying points are fairly reasonable, but the way you have chosen to present them is so off-putting that I, for one, have lost all interest in trying to engage. If there are any future possible contributors reading this, the existing team is not actively against anything like this feature. As discussed in this ticket and elsewhere, it is the effort needed to develop this feature and support it when it causes issues that means it is not a high priority at this time. |
This is the RFC that has been requested by @indirect in bundler#4552.
Summary
Enable the version constraints of gem dependencies to be explicitly overridden for gems listed in a Gemfile.
The purpose of this RFC is to get feedback on use cases, format/syntax/DSL, and feature options.
Motivation
This feature has been requested many times since 2011.
Project authors want the ability to specify newer versions of gems in their Gemfile even if the version is being constrained by another gem.
There several causes for this issue:
s.add_dependency 'json', '= 1.7.7'
s.add_dependency 'json', '>= 1.7.7', '<= 2'
The common approach to this problem is to fork the problematic gem, relax the dependencies specified in the gemspec, and reference the fork in the Gemfile. This is a very heavyweight and time intensive solution to a problem that could be easily solved by a well-designed solution to override dependencies from within the Gemfile itself.
Obviously, overriding version constraints of gems would be AT YOUR OWN RISK, however, a properly defined DSL and output messages should mitigate and address these issues.
Most of the time, this kind of override should NOT be a permanent solution, and as such, SHOULD be accompanied with some type of message that is additionally displayed to indicate why the override has occurred.
Guide-level explanation
Say you want to use the latest version of the
json
gem (currently2.1.0
), however, another gem you are using constrains the version tos.add_dependency 'json', '>= 1.7.7', '<= 2'
. To override this version constraint so that version2.1.0
is installed:This will allow bundler to install the
2.1.0
version of thejson
gem.The output when running
bundle install
will look like the following:Most of the time, this should only be a temporary fix as we expect the gem author to release a new version with updated dependency constraints.
To provide additional information a note can be included:
The output when running
bundle install
will look like the following: