Contributions to ruby/spec

As a maintainer of ruby/spec I sometimes wonder how much each Ruby implementation contributes to ruby/spec. Without further ado, here it is:

Interpretation

Rubinius started ruby/spec (called RubySpec back then) and was the main contributor until 2014. In 2015, Rubinius development slowed down significantly and so did their work on adding specs (more on that in this 10 years old post).

In 2014, TruffleRuby was started and quickly became the main contributor to ruby/spec, contributing more than half of all commits since then.

CRuby has contributed consistently to ruby/spec, around 20% of all commits each year. This is great to see as CRuby is in the unique position of implementing new methods first and deciding the expected behavior.

JRuby contributed significantly until 2012 but since then has made few contributions.

Natalie, a Ruby implementation started in 2019 made significant contributions since 2022, big thanks to @herwinw which contributed most of that.

External contributors have always been an important part of ruby/spec. This chart might unfortunately have slightly lower numbers for external contributors due to the GitHub API only providing contributor statistics for the top 100 contributors (see below for more details on that).

You might notice the huge spike for external contributors in 2009. This is because @runpaint contributed 1145 commits that year, while working on migrating specs from Ruby 1.8 to Ruby 1.9 and adding many new specs.

Before 2008

The first specs were actually added directly in the Rubinius repository, in 2006. Commits between 2006 and 9 May 2008 are recorded in a changelog file but are not part of the ruby/spec repository history. Basically, before May 2008 everything was contributed through the Rubinius repository. Also bear in mind that the data for the chart above is incomplete for the first and last years.

Methodology

If one want to see contributions by individual author instead, that is available directly on GitHub.

The data is computed by this script and the chart is generated by this script. You can find all computed data files in this directory.

The GitHub API only provides contributor statistics for the top 100 contributors, so everything is based on that data. The top 100 contributors to ruby/spec amounts to 13069 commits, out of 15045 commits total.

To classify authors by Ruby implementation, I used the top 100 contributors of each Ruby implementation. If the author contributed at least 10 commit to a Ruby implementation and is in the top 100 contributors the script considers them as contributing to ruby/spec through that Ruby implementation. If the author contributed to more than one Ruby implementation, I just pick the one they contributed the most commits to. This is not perfect but it seems reasonably accurate.

As always, commits are not a perfect measure of how much was contributed, but seems the best estimate we have readily available.

More Charts

For curiosity, I also redid the chart with a minimum of 100 commits to classify authors as contributing through a Ruby implementation (to only count key contributors to a Ruby implementation as such and classify the rest as external contributors). The result looks similar, though with more external contributors as expected:

I also did a chart with the absolute numbers of commits instead of relative to the number of commits per year. The commits for 2025 are fewer because we are only in May. This chart uses the 10 commits limit as the original chart to classify per Ruby implementation.

Conclusion

ruby/spec is key to implementing Ruby correctly and efficiently. It provides extensive coverage of Ruby’s behavior, far more precise than the documentation. ruby/spec has always relied on external contributions, from individuals and from contributors to alternative Ruby implementations. Contributing to ruby/spec is contributing to making Ruby better specified, better tested and helping Ruby implementations to be more compatible.

The CRuby test suite, while extensive and valuable, is unfortunately in my experience not enough to implement Ruby correctly based on it alone. A clear indication of that is the hundreds of CRuby issues (inconsistencies, segfaults, incorrect behaviors, etc) reported and discovered while writing ruby/spec specs.

My intuition there is the structure of ruby/spec specs (nested describes, etc) encourages to test all representative behaviors such as e.g. the various combinations of passing different argument types to some parameters together with passing a block or not, etc. OTOH unit tests as done in the CRuby test suite typically consist of a few assertions within a single test_foo method. I think having everything in a single test method discourages to write many assertions (to avoid too long methods) and often means there are only a few edge cases or basic cases being tested. In addition, there is no descriptive text explaining what is being tested so one needs to reverse engineer what an assertion is supposed to test every time one looks at such a test. I am thinking to potentially pick 10 random methods and compare their specs & tests to figure out how much this holds in a future blog post.