Closed
Description
The main reasons:
- It's used incorrectly more often than not
- It's not even practical for development nowadays (unless
--development-all
is used) - It's superseded by Bundler's groups
- There's no distinction between "dev deps", "test deps", "build deps" and "deploy deps" - and these distinctions are getting more and more important every day (especially "test deps")
- Development workstations don't need the "minimalism" provided by
gem install --dev
- Too easy to accidentally add/remove a dependency from the wrong section (runtime vs dev)
- Putting development-related gems in a gemspec doesn't make any sense from a dev ops point of view
- Average development needs are constantly growing and there's already too much stuff in the gemspec
- Bundler supports easy platform switching during development - while gemspecs are only really useful for managing releases of builds
- Organizing dependencies between gemspec and Bundler's Gemfile is simply too confusing - and deprecating this method would really help guide people in the right direction
Summary: gem development and testing is not the responsibility of RubyGems - and should never be, because otherwise it's just wasting precious developer hours.
Activity
drbrain commentedon Dec 18, 2014
I've never heard this complaint before. Can you link me to similar complaints?
If you are developing for gem a you do not need development dependencies for gem b. I don't understand what --development-all would fix here. I guess this ties into complaint number 1 or is a bug in RubyGems. Can you clarify?
Bundler's groups don't get exported as metadata on gem servers so they can't supersede the Gem::Specification.
I see this as an argument for adding more dependency types, not reducing the existing number of types.
Deployment does.
Do you mean "development dependencies should be listed in the Gemfile, runtime dependencies should be listed in the specification"?
If so this violates the DRY principle and I am highly unlikely to make such a change to RubyGems.
I haven't seen any complains about this. Can you clarify?
I don't see this as compelling, it is not the only context that RubyGems must consider. (See also my response about DRY?)
What do you mean by "too much stuff"?
Note that RubyGems relatively recently added a generic metadata hash to allow authors to be more descriptive with the content in their specifications, so the trend is to add more features to a specification, not less.
RubyGems does not download the full specification when determining which gem to install and instead downloads only a subset of the information available. The full specification is compressed inside the gem and typically consumes 512KB (a single tar block).
I don't understand how this applies to development dependencies. See also the following
How is #gemspec insufficient? This seems to be the best way for using bundler along with a Gemfile:
You have not yet convinced me it is the right direction.
This would be a breaking API change that could not be introduced until RubyGems 3. There is no schedule for such a release.
e2 commentedon Dec 19, 2014
You'll find many gems with incorrectly/incompletely specified dev deps:
Example 1:
$ gem install cucumber --dev Fetching: builder-3.2.2.gem (100%) (...) Fetching: cucumber-1.3.17.gem (100%) ERROR: Error installing cucumber: cucumber requires diff-lcs (>= 1.1.3)
Example 2:
gem install --dev nenv Fetching: diff-lcs-1.2.5.gem (100%) Successfully installed diff-lcs-1.2.5 Fetching: rspec-3.1.0.gem (100%) ERROR: Error installing nenv: rspec requires rspec-core (~> 3.1.0)
Note how ridiculous it is for a developer to include all dependencies of rspec in their gemspec (or be expected to already have them installed).
Example 3:
$ gem install --dev aruba Fetching: builder-3.2.2.gem (100%) (...) Fetching: cucumber-1.3.17.gem (100%) ERROR: Error installing aruba: cucumber requires gherkin (~> 2.12)
Example 4:
(This one actually baffles me to no end)
All 4 examples suggest incorrect usage of the 'development gem' in the gemspec.
Can you point me to a gem that uses
rspec
AND can be installed on a clean system with the--dev
flag?Of course, the above gems rely on bundler resolving the dependencies.
No one is probably using
gem install --dev
(which doesn't matter if people are using Gemfiles anyway).Some people define
rake
as a dependency twice - once in the gemspec and once in the Gemfile, to "workaround" problems running rake on Travis (more about that later).So the usage is obviously incorrect - but the problem is mitigated by using Bundler.
Exactly. My point being: there are only two options for dealing with dev deps in
gem
and both aren't practical:gem install --dev
- fails to install dev deps if they have dependenciesgem install --development-all
- installs every dev dep's deps, but installs a lot more tooI don't even understand what the use case is for that's still relevant nowadays.
It's the only other command (AFAIK) that makes uses of the 'add_development_dependency' entries.
You're right - and while it would do what
gem install --dev
fails to do, that's not what it's for.Yes. As far as I understand,
gem install --dev
fails to install deps of deps by design.So it's a problem either way:
gem install --dev
doesn't do what people expect = problemWhat's the point of exporting development deps as metadata? I mean practically?
Runtime deps and everything else in the specification - no argument there.
Example: https://rubygems.org/gems/aruba
How is that information relevant? It isn't complete.
Take RSpec for example: https://rubygems.org/gems/rspec
No deps listed, but even with --development-all can't be installed.
From: http://guides.rubygems.org/patterns/#declaring-dependencies
So we have 4 types of dependencies:
Nowadays, tests have become integral to development so much, that they are often more important than developer tools.
Test dependencies and dev dependencies are semantically different (ask any dev op).
It doesn't make sense to install Pry on a CI (where there's no terminal anyway), while doing so may break the build for unecessary reasons (I won't even mention platform deps). With JRuby and Rubinius you likely even won't be able to install the dev tools - no matter how precisely they are defined in the gemspec.
If the gemspec had "add_deployment_dependency" and "add_test_dependency" I wouldn't argue - because even as metadata it could be used.
I may be able to install test deps on CI on every platform, but maybe only one platform supports deployment.
I agree the minimalism is useful - which is precisely why the name "development" defined on RubyGems page as "typically test frameworks and build systems" is confusing. It's not uncommon to see people adding Pry as a "development" dependency in their gemspec.
It's so confusing, deprecation is the only thing I can think of.
Personally, I'd love to be able to do
gem install --test
and not have things like Pry+plugins and Yard installed on the CI.Technically, to match the def on RubyGems, for my needs the method should be
add_test_dependencies
and in theGemfile
I can put:And on CI do
bundle exec install --without=development
and everything works fine.The problem is naming: I need to map
dev -> test
TWICE:Gemfile
aboveBecause for me "development" and "test" are 2 different types of gems (like Pry vs Rspec).
That would be perfect, yes. I can use the
development_group
trick myself, but I'm sure few people know about it.I can make
add_development_dependency
useful for my needs. But the "know-how" needed to get this working is not something most developers can afford to spend time to get. They shouldn't need to have to learn that to get a decent setup.It's the opposite - this actually helps people make things DRYer, because often they put in a
rake
dependency in BOTH the gemspec AND the Gemfile - because for them "that's what somehow makes things start working".This is a clean separation of responsibilities - there's nothing "duplicated" to break the DRY principle.
This may just be a conflict of definitions for the word "development". For me, this is sufficient to deprecate the method.
For me, the gemspec dev deps are VERY useful if I use them to define gems for the CI builds (because I prefer them in the gemspec, while Gemfile is much better suited for "real" development deps).
The problem with things like
rake
andrspec
is: they are common to multiple groups (test and development), while e.grake-compiler
is common to e.g build and development. There are 2 workarounds to this:--without=development
)Bundler is better suited for this, because it's state is not version specific (that's what the
Gemfile.lock
- and it's why it doesn't make much sense for gems). RubyGems is version specific - and that's the point, because it defines releases.In development mode, there's no version. (and so, e.g. it doesn't make sense to build a gem in order to run unit tests on it).
I haven't seen any complains about this. Can you clarify?
I've made the mistake of adding development deps to runtime deps and vice versa.
Somehow the lines:
are just too easy to confuse. For me, the gemspec is much more important than the Gemspec because it affects users. As for the Gemspec I can experiment without breaking anything - which makes it perfect for trying out all sorts of development "conveniences".
I don't see this as compelling, it is not the only context that RubyGems must consider. (See also my response about DRY?)
Yes - and that's why I'm suggesting this context can be dropped in favor of putting focus on more practical issues.
Compared to bundler:
I'm not against adding features at all. Far from it. In fact, adding features usually means needs are changing. So if needs are changing, it makes sense that some thing are no longer relevant, necessary, or even useful.
So deprecations are not only unsurprising, they are to be expected.
And the deprecation I'm suggesting reflects a change of times - where gems have more dependencies than ever (through breaking up responsibilities into smaller gems), development tools are getting more advanced (Pry, rspec) and the productivity of the bleeding edge is prefered over stability (so pulling from git makes more sense than from rubygems), CI builds are more frequend and more demanding (so separating dev gemsets from test gemsets becomes more important).
As a sidenote I recently had a problem installing an older gem. A tiny gem installed for minutes and the
--trace
showed tar and ssl errors.Turned out the gems used the
rubyforge_home
specification option - once removed, everything worked.I wish that was deprecated and I had seen a message sooner.
By "too much" I mean visually and for editing. As a developer I care more about adding gem deps in the right place - I don't edit the metadata that often.
There's a much higher cognitive load while working with gemspec files. I usually just want to "fill it out" perfectly the first time and never work on the file ever again - that's probably also why I'd prefer to put the development deps in the Gemfile.
It's not uncommon to switch to non-rubygems sources for dependencies (
:git
, `:path). And I can tamper with the Gemfile in any way I want and it will have NO effect whatsoever on users installing them. Using the gemspec to do the same is an order of magnitude less convenient.Also, it depends on how the word "development" is defined. The default Rails distinction of
:test
vs:development
is more intuitive than RubyGems putting:test
+:build
in the term:development
. From a RubyGems perspective, what kind of dependency isPry
? Is it "development"? The RubyGems definition seems to include it, but I'm not sure. (Before the "bundler age", you could just install "pry" and use it without declaring a dependency in the gem).That's why I'm suggesting the deprecation. Rails has changed how people think about the terms since the original specification field was added.
(
gem install --test
used to be available but was thrown away - for good reasons and I'm just mentioning as an analogy).gem install --dev
isn't even expected to work anymore (which is why I probably won't find any of the complaints you mentioned - people aren't complaining that it doesn't work simply because either they don't use it, or they don't care).Yes, unfortunately the typical Gemfile with gemspec statement includes "gemspec dev deps" into Bundler's "development" group, and also typically, people expect to install on the CI with the
--without=development
option.Symptom: if you add
rake
usingadd_development_dependency
, it won't get installed on Travis (if using bundler args) and the build will fail.The workaround people resort to: adding
rake
also to the Gemfile, which ends up in bundler's 'default' group, so it gets installed. Then people try the same thing with RSpec.I don't know how to 'grep' the most popular gems, but I'm pretty sure there are lots of popular gems getting this wrong because of the confusion between the terms "development" and "development".
If
rake
doesn't get installed on CI, people aren't going to investigate further if adding it to theGemfile
solves the problem. (Which is why you won't get complaints - fixing the build takes priority, then people decide it's not worth reporting).I hope I've shed some light and shared my views clearly enough.
If not, feel free to ask further question or for more clarity.
I see 3 "solutions", but none of them are "obvious" (both are actually counter-intuitive in some way not fully documented):
assuming "gem dev deps" means "CI build deps", putting everything else (e.g. Pry) in the Gemfile in a
:dev_extras group
, usingbundler install --without=dev_extras
on CIassuming "gem dev deps" means "dev extras" (pry, etc.), which means CI-only tools would go into a
:test
group in the Gemfile , and usingbundler install --without development
on CIassuming 1), but putting everything else (e.g. Pry) in the Gemfile's
:development
group, mapping withgemspec development_group: test
and usingbundler install --without=development
You could blame it on incorrect/bad Travis docs, incorrect/bad Bundle defaults, but I think you'd agree that RubyGems was never designed to perfectly fit into this kind of workflow.
According to SemVer deprecating is a minor version number change, which SHOULD happen before a major number change so that people can gradually migrate (check semver.org references to "deprecated"). So if this was to be introduced, it would HAVE to be either in 2.x or no earlier than 3.1.
So if this would be deprecated in 2.x, it could be removed completely in 3.0
drbrain commentedon Dec 19, 2014
With respect to cucumber, etc., it seems that is a bug in RubyGems which should be fixed, not a reason to remove development dependencies. I thought the tests covered such cases but obviously they are inadequate.
For the cucumber case, diff-lcs is a regular dependency, not a development dependency and is not installed, thus the failure. I suspect the others are the same.
With respect to deployment I was unclear. Users do not want development dependencies installed along with a library they need in a deployed application. This is why multiple types of dependencies were added in the first place.
So far there has only been a handful of requests for multiple dependency types besides "runtime" and "development" which in RubyGems covers all development tasks including test-only environments and handy developer tools. RubyGems makes no distinction here.
You misunderstand the DRY principle which is “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.” (See The Pragmatic Programmer). For a library the gem specification is the single unambiguous authoritative representation of library metadata which includes dependencies.
Note that this is specific to ruby software released as gems and used as a library. For software that is used as an application there is no need for a gem specification thus the Gemfile.lock is where dependency metadata is stored. (Not in the Gemfile as a Gemfile alone does not allow consistent deployments. See http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/)
Bundler cannot be used to exclude dependencies of gems it installs. If a gem depends on ronn as a runtime dependency then the platform your are installing on must support it. RubyGems does not support optional or platform-specific dependencies in this manner.
I see in your response that we could do a better job at documentation and could make some improvements around declaring dependencies but I don't see a compelling reason to remove development dependencies.
Overall it seems you have a clear distinction between what RubyGems considers a runtime and development dependency.
That you sometimes remove a dependency from the wrong place is not something that I see removing development dependencies solving. You don't seem to mind the idea of adding additional dependency types which is also something that would prevent you from removing a dependency from the wrong set.
As a gem author it is up to you to take care to ensure that the software you release is fully correct. RubyGems has continually added checks to help authors release gems that have minimal problems, but there is a limit to what RubyGems can check. Ensuring that your dependencies are properly specified in your gem specification is one of those things that is difficult for RubyGems to check statically.
PS: I find bundler entirely inadequate as a tool for developing gems. It requires you to rebuild test suites, documentation generation, release process, correctness checking, etc. for every library you write. I use hoe which has built-in tasks or reusable plugins for all these features. For application deployment it is an acceptable tool.
e2 commentedon Dec 19, 2014
My point being: no one actually cares it's even broken.
Seriously, no one cares! You're time is EXTREMELY precious given the role you are playing in the Ruby world. Instead of working of fixing something no one cares about, I believe it would be better to show a deprecation with a url, and have a clever description protecting you of any "accusations".
And CI's are also better off also not having those installed. The problem is dev and test share more deps than runtime and dev do. Since CIs have replaced testers, CI don't require "human tools" (like PRy). And CIs have become "first class citizens" in typical development workflow. Obviously, they can't cry out and fight for their rights to "better working conditions" with "less effort for needless installs of human-only dependencies".
Neither does bundler - to the point that bundler resolves and installs EVERYTHING - except when the
--without
option is given (resolving is still done on all runtime+dev gems).But the
--without
option is still very useful. E.g. maybe I'm a contributor and I don't want to touch anything related to the gem's docs. Or I'm fixing the docs (e.g. translating), and I don't want any of the build and tools.Also, many dev tools "essential" for human being may not even be gems in the first place (external repos, tarballs). I don't see a reason why RubyGems should even care.
First, thanks for the quote - it's much better defined than what I had in mind.
And it also just solidifies my point. Simply because runtime dependencies are obviously needed by RubyGems, there's no need to force developers to "log" which tools they used in RubyGems as well (because that's what defining development dependencies is - just developers declaring what they've used for development).
In short: development dependencies and runtime dependencies together are NOT - by any stretch - "a single piece of knowledge". They are two unrealated pieces artificially "mashed" together inside a single gemspec. And test deps included with development deps don't represent a "single piece of knowledge" either. (Any reasonable dev op would confirm).
I believe it would be better to even have a separate file for development dependencies - and (if at all) just have a reference to it in the Gemfile. Because it would be rarely needed by RubyGems (if at all).
Currently, the existence of the add_development_dependencies method "inspires" people to unnecessarily fill it out - "just because it's there". Bundler has a rake task for installing dependencies (
rake spec:deps
) - maybe there should simply be a "convention" for this in Bundler's gem tasks instead. (Since it relies on Rake, which RubyGems shouldn't care about.)I am constantly aware of this - I'm actually trying hard not to let conventions from the "Rails world" cloud how I'm thinking about this subject (because the rationale there is very different).
Then again gems and apps are getting a lot more similar in terms of the complexity of the development workflow (deploying to hosting provider is not much different from deploying a release to RubyGems.
rake release
is not much different in terms of outcome fromcap deploy
).I see this as yet another reason to delegate the definitions to the Gemfile. The only reason I can think of to not put them in the Gemfile is: if Gemfiles (and Gemfile.locks) are ultimately going to be deprecated in place of gemspecs. Because why let dev dependencies be specified BOTH in the Gemfile and gemspec - especially since now RubyGems uses Gemfile.lock for dependencies
Actually, that's the first decent article I came across when I was searching for "good practices" about defining dev deps in gemspecs. And partially the reason I brought this up. Development deps for gems weren't expanded on in the article - probably because they are useless or irrelevant anyway (except when things fail when combined with
--without
in counter-intuitive ways).I was talking about development dependencies - ronn is used for generating man pages for gems, but it can't be installed on JRuby (I think it was because of rdiscont). So it's a good example of a development dependency essential for updating docs, but not e.g. for fixing bugs. So if a developer is fixing JRuby-specific bugs, it doesn't make sense for installing dev gems to fial simply because it's listed as a dev dependency.
It's unclear if dev deps in the gemspec are meant to be "optional", "suggested" or "essential". So if development deps were meant to only be in Gemfiles, I'd simply have a
:doc
group for Ronn and install usingbundle install --without=doc
on a jruby. Or even have it conditional depending on presence of JRuby - it's a development dependency, so no one will get hurt from it not being installed.If dev deps are meant to be correct, they should be checked (during
gem build
for example). If they are meant to be a manifest, then it would be more useful to explain why the dependency is needed (e.g. docs) than expect people to provide version constraints which are never checked in a practical context.I don't see a practical reason for development dependencies to exist. I mean if the field became a no-op today, what exactly would that "break" that isn't broken already?
I can't find a single use case as to why dev deps should be in the gemspec, and not a Gemfile (especially since only Bundler can reliably install dev deps right now).
Maybe informational metadata is useful - but then there's no reason to supply version constraints, because enforcing them would be more painful than not (ronn example on JRuby).
If dev deps in the gemspec were "deprecated", I wouldn't be able to put dev deps there without feedback telling me not to (I'd have to put them in the Gemfile, obviously). This would make a clear distinction: runtime stuff -> in the gemspec. dev stuff -> in the Gemfile it goes.
With dev gems allowed in the gemspec I have TWO places I can add dev deps to. That violates DRY for me. (and the development_group option is kind of proof of DRY being violated - because it means Bundler is mapping gemspec devs into it's own group).
It's an "I can live with it" thing - if metadata is so important and the gemspec provides the distinctions tools need, then I don't have to bother with a Gemfile anymore. Because I actually need multiple types defined (
:dev
,:test
::build
,:doc
), and I can do all that cleanly in a Gemfile right now. If gemspecs supported groups and it was recommended over putting them in the Gemfile, that would work for me too.(It would still be "strange" to define "gem-version agnostic" dev deps in the gemspec, but if that would be recommended over putting them in the Gemfile for gems, I'd live with it).
But having "just" a "development group" (aside from
:runtime
) means I have to decide whether:development
in the gemspec means::dev
or:build
or:test
or all of them (including:doc
) - and other gem authors are going to make different decisions.So if the dev deps aren't clearly defined what they should have (as opposed to the
homepage
field, where I don't put e.g. my phone number), that's why I'm suggesting to deprecate that in favor of something recommended, but maybe a bit more specific and practical.I'm open to hearing a "right way to do things" or being pointed to docs showing the "the right way" - and Yehuda's article is the best thing out there. And from what I understand, the idea of putting dev deps into the gemspec is wrong.
Just to switch to apps/Rails for a second: the distinctions there are MUCH clearer and more practical. The only issues is Bundler resolves all dependencies anyway (just skips the installs if told to), and many people just put all their gems in both the
development
andtest
andproduction
groups until they want to optimize deployment or CI builds. Other times people put things in development, they deploy, they see things broken and then they separate things into groups (which is good, because they can change their decision about deps at any time - and even delay the decision until it's necessary).Rubygems doesn't allow me to optimize those, because I have no idea what
development
should semantically mean in the RubyGems. If it's just metadata no one cares about, I can just as well put my dev deps in the Gemfile.The thing is: most people learn through "herd mentality" - if something is done in a gemspec in one project, they mindlessly "copy" it thinking it's a good practice. So deprecations seems to be the only way to communicate the intention behind the gemspec dev deps.
To me it seems
add_development_dependency
it's an ancient artifact from a pre-bundler and pre-TDD world. If so, deprecation is the obvious solution. Now it makes as much sense assetup.rb
.After all the investigation and research about how to use "gemspec dev deps" correctly, I still don't understand how specifying them is even useful from even a metadata perspective. It either increases bandwidth unnecessarily, or, if it's optimized not to be served - that just proves it's not important for anything.
Yes, that's why I have no objection to defining runtime dependencies in the gemspec (or install time dependencies). I expect installing a gem to work. And I do not expect development dependencies to have ANY impact on installing a gem. If they do, it's a bug or I've screwed up the runtime dependencies.
For some gems I may need Rails for development and testing (not unusual). But who really cares if I do? Only the person who is working on the area of the gem that actually needs it. Maybe it's only good for debugging. Maybe it's used for generating docs (not a strange idea).
And that's why I'd leave dealing with development dependencies to Bundler. It's because I don't want RubyGems to be checking stuff - especially if I'm using other gems in the form of potentially checked out master branches. I don't care about the version number at all, because on a master branch of a gem can have the same version number - but a totally different behavior (which I'm actually relying on). Or if the gem is no longer maintained, I may need to switch to a fork (which obviously isn't released as a gem).
I don't see a reason for RubyGems to have anything to do with my workflow in regards to development, testing, generating docs, etc. It's only once I've finished development and I want to release - when I even care about version constraints or metadata.
I only expect RubyGems to come into play when the user types in
gem install
, or when they need help or support (basically contact info/website). That alone is awesome. If RubyGems can tell me duringgem build
that stuff in a gem I'm contributing to is outdated (like the rubyforge_home metadata), I'd be overjoyed.But otherwise RubyGems only makes development harder if I have specify the deps manually and then manually install them (or add a special rake task to do that). With Bundler I can just put in
gem 'rspec'
and I'm done. If someone has outdated versions, I don't want to care about them ifbundle update
can resolve the issue.I would have agreed not that long ago. Nowadays I find that if a gem needs that kind of "management", it probably should be split up into smaller gems. It's as if when then documentation doesn't fit in the README - or the code isn't sufficiently simple and descriptive to replace API documentation - then I'm overdoing things.
Ironically, RubyGems makes it so easy to deal with "tiny" gems ... and Bundler takes care of dependencies so well ... releases can happen faster with more confidence, APIs of components can mature faster and independently and each component can have it's own branches, so migration becomes fluid and unobtrusive - yet still any change can be backported without forcing people into major versions they otherwise don't care about, etc.
I've used hoe in the past and it somehow overall discouraged me from creating gems. I was based on great ideas (extracting the specification data), and the plugins were nice - it "seemed" to be more optimized for "managing" gems intended for users who are developers. (The coolest thing about hoe (that I miss) is that I actually can avoid seeing the gemspec altogether.)
And I wish I could "hoeize" existing gems in a less intrusive way that existing gem authors would accept in their project.
To me "adding" stuff to a gem "from the outside" (as opposed to editing existing files as configuration) just seems better - probably because I prefer to add functionality through "revertible commits" and not "config".
I also do things very differently nowadays, which probably explains why I don't see the point of involving RubyGems in development:
hub clone user/project
bundle exec rspec --init
(plus add Rakefile task - which is a bit bothersome I admit)bundle exec rake
.rspec
version.rb
&&bundle exec rake release
and sometimes publish release notes automatically using Octokitgit push -u origin branch_name && hub pull-request
(triggers Travis build on multiple Rubies, etc. - I can monitor withhub ci-status
)for spec in spec/**/*_spec.rb; do bundle exec rspec $spec || break; done
And those commands are pretty project-agnostic, so it would be hard to justify migrating projects to use
hoe
.On the otherhand, if
hoe
could "lint" projects and "autocorrect" them (without becoming a dependency) - it would be one of the most marvelous tools out there.Also, since docs usually represent the API for SemVer, if docs change without a major release it's more likely a sign that compatibility was broken.
I don't see a reason for even Hoe to actually bother with the development dependencies. The hoe
newb
command could just as well use aGemfile
or evenGemfile.lock
for installing dependencies. And strangely in hoe, I can't runrake deps:list
on an unreleased project.To summarize, here are options I'd be happy with:
a) dev deps in a separate file (Gemfile would be fine) so I can define them for installing after cloning a project (but not for inclusion in the metadata)
b) separate metadata/groups for dev and test (and probably build/doc - e.g. rake-compiler, ronn), so that
bundler install --without=development
would work as expectedc) deprecating the method, so a warning is shown during
gem build
I believe I understand this issue well enough to explain to other developers how to correctly get RubyGems, Bundler and Travis working as expected. And given the amount of people out there - I can't pass on this knowledge by contacting every person on the planet. I'm not really interested in sending thousands of PRs to fix other people's dev deps (I'd die under email notifications). And it's annoying for me to break away from the PR/fix I'm working on to create a separate PR or commit just fixing the deps.
So it would be really nice if other people just had a way of knowing how to define dev/test/build/doc dependencies to get the results they want both for them and other contributors to their projects.
As a contributor, I'm a "nobody", so people will be suspicious if I submit a PR fixing "something" in their gemspec.
But if RubyGems points them to an article with explanations - I can just point them to that official doc even in the commit. And the patch is more likely to be accepted.
If I have trouble convincing even one of the biggest Ruby heroes out there that this is confusing...
... there's a problem either in my understanding or my communication.
I'm not asking to touch anything that would affect users installing gems (that would obviously be a nightmare for the Rubygems team). This is no "worse" than the "gem dependency defined multiple times" Bundler warning when people have e.g. rake in both the Gemfile and gemspec. In fact the deprecation would help guide people towards correctly avoiding the warning.
According to SemVer it would be perfectly normal to include this in the next minor release - so it will only affect developers who've upgraded RubyGems.
Let me know if I can do anything else to help resolve this (or if there is anything this deprecation could actually "brake").
drbrain commentedon Dec 19, 2014
I've fixed at least two bugs in development dependency installation (as fallout from the switch to the dependency resolver) in the past six months. There is one or two more that isn't explicitly mentioned in History, so I know the feature gets used.
With respect to time invested, maintaining the feature is something I will continue to have to do until the feature can be removed. Since there is no timeline on removal and, relative to the rest of my workload, the maintenance burden is small I do not find this compelling. Just because I am removing a feature doesn't mean I'm allowed to stop fixing bugs in it as that is equivalent to removing the feature.
There are many different types of users of RubyGems, not just "dev ops" people. I can't make a decision based on the ideals of one group of users unless it is absolutely overwhelming.
There are many gems that are written and released without using Bundler in any way. Forcing users to edit two files to build their gems does not follow the DRY principle regardless of how you wish to define dependencies. You still force users to put one type of gem metadata in one place and another type in a separate place.
It is almost always more work for me to remove a feature from RubyGems than it is to maintain it. Removing a feature involves many carefully orchestrated, documented and announced steps over many months and years. In this particular case it involves writing new code to deal with existing gems with existing development dependencies in a sane manner, something I will need to do for the rest of RubyGems history.
I've read and skimmed about a fifth of what you've written, I don't have the time or energy to read it all, can you summarize down to a three or four paragraphs?
e2 commentedon Dec 19, 2014
Sorry if I suggested that. To me the feature feels "abandoned" by users (given it doesn't work for popular gems).
Yes, and those users are likely manually using
gem install
instead ofgem install --dev
on whatever is missing.BUILD GEMS??? So that's what I was confused about!!!
It's not "development dependencies", it's "BUILD dependencies" the gemspec defines!
So I can assume:
$ gem install --dev
is REALLY something like:
$ gem install --build-deps
This makes LOTS of sense to me now!
I wish there was at least an alias to
add_development_dependency
such as:add_build_dependencies
.An in Gemfiles I could use e.g.:
I'm sorry I was so confused about this and it took me this long to make the distinction I needed.
My replies:
I understand completely.
My goal here is to reduce confusion without creating extra work for you.
Would aliasing 'add_build_dependency
to
add_development_dependency` be too much to ask for?I understand those - that's why I though deprecation would be a good way of "testing the waters" without risk or maintenance costs.
Thank you sincerly for your time and insights!
Thanks and have a warm, happy Christmas!
chulkilee commentedon Feb 10, 2015
+1 to mark
add_development_dependency
deprecated and remove in the next major release (v2)Agree that it might be useful in pre-bundler era. However things changed and any ruby projects should use bundler - including gem development. I don't know what is the merit of having dev dependency in gem spec itself and in gem source.
If it's extraneous feature, then it would be better to deprecate and then remove it to focus on other key features.
I appreciate contributors' efforts on this feature, but I don't think we need to consider such sunken cost. Of course we have to get more feedback and then take phaseout process when decided.
I get your point but I'd argue that using a gem and developing a gem is different use case. In bundler era all ruby developers are expected to use bundler in their environment. dev dependencies can be in such environment by Gemfile. It doesn't need to be in gemspec.
I think all *users use just runtime dependency. dev dependency is for developer of each gem.
I think the main point is how seriously we consider bundler as a "standard tool"...
evanphx commentedon Feb 10, 2015
I don't see any reason to deprecate it. Having development dependencies is still super useful and there is really no reason to remove them.
[-]Please deprecate `add_development_dependency`[/-][+]Please deprecate (and/or rename) `add_development_dependency`[/+]e2 commentedon Feb 10, 2015
@evanphx - how so? I've always believed that, but now I can't find a single "good" reason where it isn't using the "wrong tool for a given job".
The point I 100% agree with is that given the "age" and scale of RubyGems, deprecation is simply not worth the actual costs.
e2 commentedon Feb 11, 2015
TL;DR - I agree deprecation is costly, so I think a "new" tool (alongside Bundler and RubyGems) would be a better place for "new features" and "deprecations".
@chulkilee - thanks, you totally understand me!
And without Bundler, there's no guarantee you're actually using the right versions of build tools to build the gem anyway (while
rake install
can use Bundler implicitly).I understand that just keeping RubyGems working as it is requires a ton of effort few people are even capable of appreciating. Also, there are at least thousands of people "relying" on the feature (even for strange purposes), so any "feedback" will be enormous in quantity (i.e. tons of "complaints").
I think a better option would be a "new" tool that would integrate Bundler and RubyGems functionality in an "integrated" and "recommended" way. Package managers are a tough problem - especially when it comes to backward compatibilty. While it would be confusion to have another tool (given there's Bundler and RubyGems already), but it does give millions of people a choice (backward compatibility vs new features).
Also, a lot of gems out there are "ancient" - adding a deprecation to them would only cause maintainers and users to waste focus.
I can't agree more.
Gems are for "use", while repositories are for development. (In a perfect world, we'd all be using the master branches of everything in production).
Not to mention - almost every development dependency can be optional (contributing/developing does not necessarily imply you have to be able to build a gem yourself on your own workstation). And so if dev/build deps are optional - why should they be expected to be part of a "released specification" (gemspec)?
I'd disagree that it's even about bundler - you can have a Rakefile so that e.g.
rake install:deps
usesgem install -v <version> <gem>
and other system commands to make sure the dev environment is properly set up (including checking out other repositories, submodules, toolchains, OS packages, IDE plugins, etc).A gemspec is just too restrictive for dev deps - you can't lock to a specific version of Golang for example (e.g.
zeus
gem requires golang properly set up to build it) and you can't use checked out repositories. Even plain Rake is is more flexible and also more "standard" for this.evanphx commentedon Feb 11, 2015
@e2 What features are you thinking a new tool needs? We've been adding new things people need to Rubygems fine.
Because dev_deps is unable to articulate the universe of development needs doesn't mean it's a bad option. It's not uncommon when working on a gem for the first time to install the development dependencies and the best way they're articulated is via dev_deps.
e2 commentedon Feb 11, 2015
TL;DR - Bundler Rakefile shows that gemspec devdeps don't help with a real-world problems
@evanphx - a new tool would "absorb" paradigm changes faster than the heavily relied on RubyGems which already has a massive legacy or expensive (backward-compatibility) features.
As for dev_deps in a gemspec - I believe the fact that no extremely knowledgeable and experienced Ruby "guru" (e.g. you or @drbrain) can articulate a single "good" use case for it is ... case in point.
It's not a bad option technically (I'd just say it's plain inferior given other options) - it's just "cargo cult". It's like a configuration option that no one uses, but everyone specifies because "other people do it".
Rake and Bundler are much better for setting up the dev environment - whether those deps are in the gemspec or not is just an implementation detail. There's just no advantage to having those in a gemspec - only disadvantages.
This is especially true for platform-specific dev tools (Rubinius, JRuby, etc.).
It's pretty much the same story as with specifying test files in a gemspec. People do it but ... what's the point?
Not true - the best way to install deps is by what's recommended in the docs. E.g. even Bundler (which can't use itself to install deps) has custom code (for
rake spec:deps
):https://github.com/bundler/bundler/blob/f2c712b6f5a16ea8f81b432537d2852bbea2df67/Rakefile#L53-L75
And because it uses the gemspec (a mistake in my opinion), it breaks the DRY principle (that @drbrain talked about), because it has to remove
ronn
andrdiscount
on JRuby. This is a clear example of why the gemspec is a bad place to store gem deps - except maybe for gems that don't have documentation to generate, etc..16 remaining items