-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC submission for override_gem_dependency_versions #13
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,123 @@ | ||
- Feature Name: override_gem_dependency_version | ||
- Start Date: 2018-07-10 | ||
- RFC PR: | ||
- Bundler Issue: | ||
|
||
# Summary | ||
|
||
Enable the version constraints of gem dependencies to be explicitly overridden for gems listed in a Gemfile. | ||
|
||
The purpose of this RFC is to get feedback on use cases, format/syntax/DSL, and feature options. | ||
|
||
# Motivation | ||
|
||
This feature has been requested many times since 2011. | ||
|
||
Project authors want the ability to specify newer versions of gems in their Gemfile even if the version is being constrained by another gem. | ||
|
||
There several causes for this issue: | ||
- The version dependency specified in the gemspec is unnecessarily strict: `s.add_dependency 'json', '= 1.7.7'` | ||
- The gem is slow to update or is no-longer maintained to the point that version constraints do not keep up with the version updates of their dependencies: `s.add_dependency 'json', '>= 1.7.7', '<= 2'` | ||
|
||
The common approach to this problem is to fork the problematic gem, relax the dependencies specified in the gemspec, and reference the fork in the Gemfile. This is a very heavyweight and time intensive solution to a problem that could be easily solved by a well-designed solution to override dependencies from within the Gemfile itself. | ||
|
||
Obviously, overriding version constraints of gems would be *AT YOUR OWN RISK*, however, a properly defined DSL and output messages should mitigate and address these issues. | ||
|
||
Most of the time, this kind of override should *NOT* be a permanent solution, and as such, *SHOULD* be accompanied with some type of message that is additionally displayed to indicate why the override has occurred. | ||
|
||
# Guide-level explanation | ||
|
||
Say you want to use the latest version of the `json` gem (currently `2.1.0`), however, another gem you are using constrains the version to `s.add_dependency 'json', '>= 1.7.7', '<= 2'`. To override this version constraint so that version `2.1.0` is installed: | ||
|
||
```ruby | ||
gem 'json' | ||
gem 'other_gem' do |s| | ||
s.override_runtime_dependency('json', '>= 1.7.7', '<3') | ||
end | ||
``` | ||
|
||
This will allow bundler to install the `2.1.0` version of the `json` gem. | ||
|
||
The output when running `bundle install` will look like the following: | ||
|
||
``` | ||
Installing json 2.1.0 (forced version, conflicts with other_gem) | ||
Installing other_gem 1.0.0 | ||
``` | ||
|
||
Most of the time, this should only be a temporary fix as we expect the gem author to release a new version with updated dependency constraints. | ||
|
||
To provide additional information a note can be included: | ||
|
||
```ruby | ||
gem 'json' | ||
gem 'other_gem' do |s| | ||
s.override_runtime_dependency('json', '>= 1.7.7', '<3', 'waiting on pull request #5') | ||
end | ||
``` | ||
|
||
The output when running `bundle install` will look like the following: | ||
|
||
``` | ||
Installing json 2.1.0 (forced version, conflicts with other_gem: waiting on pull request #5) | ||
Installing other_gem 1.0.0 | ||
``` | ||
|
||
# Reference-level explanation (TODO) | ||
|
||
This is the technical portion of the RFC. Explain the design in sufficient detail that: | ||
|
||
- Its interaction with other features is clear. | ||
- It is reasonably clear how the feature would be implemented. | ||
- Corner cases are dissected by example. | ||
|
||
The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. | ||
|
||
# Drawbacks | ||
|
||
These are the common drawbacks and counter-arguments mentioned by the community: | ||
* This issue can already be solved by forking gem repositories and updating the dependencies yourself | ||
* Allowing this means there is no longer a canonical source of truth about what a gem's dependencies are | ||
* When these kinds of changes cause resolver errors (and they will), it will be extremely hard to diagnose | ||
* The potential that people will complain that Bundler is broken when it’s actually something they did themselves | ||
* The potential that people will complain to the gem author that the gem is broken when it’s actually something they did themselves | ||
|
||
These are all valid points of view, however, there are also counter-arguments to each of these points which are also equally valid: | ||
* The most obvious place to manage version dependencies (even overridden version constraints) is in the Gemfile and not in a forked repository | ||
* Forking an entire project (and subsequently managing upstream changes, etc.) is an unnecessarily tall order for something that can certainly be fixed with a simple, at-your-own-risk dependency override | ||
* This feature becomes more and more important with the number of gems growing. .NET has it. Java has it. Nothing bad happened to them because of it | ||
* Maintaining overridden version constraints within the Gemfile saves a significant amount of development cost when compared to maintaining a forked repository | ||
* Explicit syntax within the Gemfile with custom notes and obvious output messages when running `bundle install` can help to mitigate developers not knowing that a version conflict exists | ||
|
||
# Rationale and Alternatives | ||
|
||
This is the best design because it is very explicit and clearly documents what is happening. | ||
It requires project maintainers to have a clear understanding of the conflicts they are encountering. | ||
|
||
Other designs are as follows: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. After looking at the original design by @segiddins and now ported to bundler/master here: rubygems/rubygems#4178, I think the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here is sample output from the $ dbundle
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Using public_suffix 4.0.6
Using builder 3.2.4
Using mini_portile2 2.4.0
Using bundler 2.3.0.dev
Using rack 2.2.3
Using socksify 1.7.1
Using json 1.8.6
Using jwt 1.5.6
Using nori 2.6.0
Using addressable 2.7.0
Using gyoku 1.3.1
Using nokogiri 1.10.10
Using httpi 2.4.5
Using akami 1.3.1
Using wasabi 3.6.1
Using savon 2.12.1 [version forced]
Using sfmc-fuelsdk-ruby 1.3.0
Bundle complete! 2 Gemfile dependencies, 17 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed. This also means I no longer have to maintain a forked version of Here is the source 'https://rubygems.org'
gem "sfmc-fuelsdk-ruby", "1.3.0"
gem "savon", "2.12.1", force_version: true Here is the
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. My implementation in rubygems/rubygems#4178 is now fulling working. This is a port of @segiddins original POC. I have created additional tests, as well as run it against our complex rails project at http://https://github.com/simplybusiness where it has worked well. @br3nt, would welcome your thoughts. This feature would have a lot of value for us as a company. It is also opt in so no one is |
||
|
||
```ruby | ||
gem "puma", "5.1" # depends on rack <= 2.0 | ||
gem "rack", "3.0", force_version: true | ||
``` | ||
|
||
This version forces the `rack` gem to explicitly force the install of a version matching `3.0`. | ||
This syntax will work fine, but it is not documenting which is the problematic gem preventing the requested version of `rack`. | ||
|
||
The equivalent using the proposed solution would look like this: | ||
|
||
```ruby | ||
gem "puma", "5.1" do |s| | ||
s.override_runtime_dependency "rack", "~> 3.0" | ||
end | ||
gem "rack", "3.0" | ||
``` | ||
|
||
The proposed solution clearly documents that the problematic gem is `puma` and that we are explicitly overriding it's version constraints for the `rack` gem. | ||
|
||
The impact of this is verbose and possibly unsightly code. The benefit, however, is that version overrides are explicitly documented which will help prevent excessive issues being lodged to bundler and to gem authors. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it also could make the implementation of the feature much more difficult |
||
|
||
# Unresolved questions | ||
|
||
* What is the best mechanism to show override messages to the user? | ||
* What is the best syntax to include a custom override message? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is a good example of the problem in action: salesforce-marketingcloud/FuelSDK-Ruby#115.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm sorry, but I don't get it. But what makes it actual good example?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would you prefer to maintain an entire fork of a gem just to update it's Gemfile, when you could just add an override in your own Gemfile?
The good things about overriding in your own Gemfile are:
gem install
can explicitly show that a problematic gems dependencies are being forced to be a version that isn't technically supported by that gem. This couldn't happen if you maintain a fork obviously. Maybe you can tell it's not the original gem because you can see the link to a GitHub repo, but it doesn't tell people why you are doing this and what the condition is for moving back to the original gem. Maybe you could add a code comment the explains all this, but that's not very semantic. You couldn't use a code tool to read the comments to tell you when you should start using the original gem again.gem install
can explicitly show that none of the problematic gem's dependencies' versions are being forced. If after agem update
this message is being show, it indicates that the original gem author has finally updated their problematic gem dependency version constraints, which means you can finally remove the override from your Gemfile. This wouldn't happen if you maintain a fork. You would need to constantly actually look at the original gem for updates. That is such a manual chore.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer to maintain an entire fork of a unmaintained gem or actually try to find out alternative to this gem to remove abandoned gem in all codebases I do maintain.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, but why?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the cases where that apply to us, we have found maintaining and tracking occasionally updated (but not fully abandoned) gems, ends up being a thankless chore that creates more problems than it solves. This of course doesn't mean it's right for everyone else.
This is also a common feature in other dependency managers:
The above cover the top languages from Github and all have a solution. I would argue this is not an esoteric need.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wrote up my view on the problem and options for solutions that are currently available, 1 that is this RFC and 1 other that is not available.
I hope this helps to frame the debate.
Options for handling out of date gems
Out of date gems have been a problem for a long time. As the ruby ecosystem evolves and ages, they are and will become more common. This is especially prevalent in areas where the state of the art has moved on (for example SOAP).
Unfortunately, these 'stuck' gems often have dependencies on other more common and evolving gems. This would not be a problem if the un-evolved gems had loose dependency constraints. However, the recommendation for gem writers is to be relatively tight on constraints to ensure compatibility.
We now have stuck gems depending on other gems. Since only one version of a gem can be installed in bundler. This then constrains the use of newer gems that share a downstream gem dependency (but not the dependency version).
There are a number of options for handling these out of date gems. Here I list them out and highlight their pros and cons.
Finally, I give my recommendation for consideration.
Here are the many options that developers have to deal with out of date gems:
Here is a detailed description of each of them and their pros and cons:
1. Create an issue
Create an issue on the relevant repo and wait for the maintainer to handle it and release a new version. This and the next option are the happy path, the way things should work.
This option and the following option are usually the first port of call, since they are also the easiest.
Pros
Cons
2. Create issue and submit a PR
Create the issue but also provide a solution in the form of a PR. Again, this is the happy path, but isn't always available.
Pros
Cons
3. Monkeypatch
Create a gem or an internal piece of code that overrides the buggy or out of date behavior in the gem that needs updating. An example of this is Savon Fixes which I created.
Pros
Cons
gemspecs
can't be monkeypatched4. Fork and use git path
This is the most common way of forking. Create the fork, make the patch and point your gem file at your git repo and the relevant commit/branch.
Pros
Cons
5. Fork and use private registry
Create the fork, make the patch, add in your proprietary CI/CD and registry config/code and point your
Gemfile
at the internal registry.Pros
Cons
6. Public fork and change name
In this case you are forking the gem, and then changing it's name so that you can publish to
Rubygems
under the alternate name. Then yourGemfile
points to the newly named gem.Pros
Rubygems
)Cons
alias_gem_name
fuctionality toBundler
which is not currently available7. Take over maintenance of the gem
In this case, become the maintainer of the gem as the previous maintainer no longer has the resources to spend the time on maintenance. Then you can make the necessary updates.
Pros
Cons
Rubygems.org
8. Force version
Add a capability
Bundler
which allows an option in aGemfile
. This option lets the user force the version of a dependency which then overrides that dependency in thegemspecs
of all dependent gems.Pros
Cons
Bundler
api (though opt in)9. Write your own
This just means avoiding the 'stuck' gem entirely.
Pros
Cons
Recommendation
Given the complexity and how painful many of the options are, I believe adding 8) Force Version to
Bundler
is the best solution for the dependency side of these problems.Additionally, I think the
Bundler
team should consider analias gem name
function that would allow users to redirect to a forked version of a gem which they can make public without having to fork entire chains of gems.References
Dependency managers offering override functionality:
NPM (javascript): https://nodejs.org/en/blog/npm/managing-node-js-dependencies-with-shrinkwrap/
NPM (javascript): https://www.npmjs.com/package/npm-force-resolutions
Yarn (javascript): https://classic.yarnpkg.com/en/docs/selective-version-resolutions/
Pip (python): https://pip.pypa.io/en/stable/user_guide/#constraints-files
Maven (java): https://www.baeldung.com/maven-version-collision
Gradle (java): https://docs.gradle.org/current/userguide/resolution_rules.html
Top languages in Github: https://madnight.github.io/githut/#/pull_requests/2020/3
Google search for 'bundler override gem dependency site:stackoverflow.com' returns 1,360 results: https://www.google.com/search?q=bundler+override+gem+dependency+site:stackoverflow.com