On the Edge of Ruby

Contributions to ruby/spec

2025-05-03T00:00:00+02:00

As a maintainer of ruby/spec I sometimes wonder how much each Ruby implementation contributes to ruby/spec. Without further ado, here it is:

Interpretation

Rubinius started ruby/spec (called RubySpec back then) and was the main contributor until 2014. In 2015, Rubinius development slowed down significantly and so did their work on adding specs (more on that in this 10 years old post).

In 2014, TruffleRuby was started and quickly became the main contributor to ruby/spec, contributing more than half of all commits since then.

CRuby has contributed consistently to ruby/spec, around 20% of all commits each year. This is great to see as CRuby is in the unique position of implementing new methods first and deciding the expected behavior.

JRuby contributed significantly until 2012 but since then has made few contributions.

Natalie, a Ruby implementation started in 2019 made significant contributions since 2022, big thanks to @herwinw which contributed most of that.

External contributors have always been an important part of ruby/spec. This chart might unfortunately have slightly lower numbers for external contributors due to the GitHub API only providing contributor statistics for the top 100 contributors (see below for more details on that).

You might notice the huge spike for external contributors in 2009. This is because @runpaint contributed 1145 commits that year, while working on migrating specs from Ruby 1.8 to Ruby 1.9 and adding many new specs.

Before 2008

The first specs were actually added directly in the Rubinius repository, in 2006. Commits between 2006 and 9 May 2008 are recorded in a changelog file but are not part of the ruby/spec repository history. Basically, before May 2008 everything was contributed through the Rubinius repository. Also bear in mind that the data for the chart above is incomplete for the first and last years.

Methodology

If one want to see contributions by individual author instead, that is available directly on GitHub.

The data is computed by this script and the chart is generated by this script. You can find all computed data files in this directory.

The GitHub API only provides contributor statistics for the top 100 contributors, so everything is based on that data. The top 100 contributors to ruby/spec amounts to 13069 commits, out of 15045 commits total.

To classify authors by Ruby implementation, I used the top 100 contributors of each Ruby implementation. If the author contributed at least 10 commit to a Ruby implementation and is in the top 100 contributors the script considers them as contributing to ruby/spec through that Ruby implementation. If the author contributed to more than one Ruby implementation, I just pick the one they contributed the most commits to. This is not perfect but it seems reasonably accurate.

As always, commits are not a perfect measure of how much was contributed, but seems the best estimate we have readily available.

More Charts

For curiosity, I also redid the chart with a minimum of 100 commits to classify authors as contributing through a Ruby implementation (to only count key contributors to a Ruby implementation as such and classify the rest as external contributors). The result looks similar, though with more external contributors as expected:

I also did a chart with the absolute numbers of commits instead of relative to the number of commits per year. The commits for 2025 are fewer because we are only in May. This chart uses the 10 commits limit as the original chart to classify per Ruby implementation.

Conclusion

ruby/spec is key to implementing Ruby correctly and efficiently. It provides extensive coverage of Ruby’s behavior, far more precise than the documentation. ruby/spec has always relied on external contributions, from individuals and from contributors to alternative Ruby implementations. Contributing to ruby/spec is contributing to making Ruby better specified, better tested and helping Ruby implementations to be more compatible.

The CRuby test suite, while extensive and valuable, is unfortunately in my experience not enough to implement Ruby correctly based on it alone. A clear indication of that is the hundreds of CRuby issues (inconsistencies, segfaults, incorrect behaviors, etc) reported and discovered while writing ruby/spec specs.

My intuition there is the structure of ruby/spec specs (nested describes, etc) encourages to test all representative behaviors such as e.g. the various combinations of passing different argument types to some parameters together with passing a block or not, etc. OTOH unit tests as done in the CRuby test suite typically consist of a few assertions within a single test_foo method. I think having everything in a single test method discourages to write many assertions (to avoid too long methods) and often means there are only a few edge cases or basic cases being tested. In addition, there is no descriptive text explaining what is being tested so one needs to reverse engineer what an assertion is supposed to test every time one looks at such a test. I am thinking to potentially pick 10 random methods and compare their specs & tests to figure out how much this holds in a future blog post.

Matching Regexps 200 Times Faster

2025-03-14T00:00:00+01:00

You might have seen @byroot’s excellent blog post series on optimizing the json gem. From the first blog post it’s clear most of the time for generating JSON is spent in generate_json_string() and specifically in convert_UTF8_to_JSON(), i.e., in converting Ruby Strings to JSON Strings.

Because this is such a hot part of JSON generation, it has received a lot of attention. In part 3 @byroot shows a lookup table approach to implement convert_UTF8_to_JSON(), which is now used in the json gem.

There has been a couple attempts since then to optimize it further using SIMD instructions. This is however quite difficult and messy to do in C as it’s similar to writing inline assembly (N times, with N the number of SIMD variants), and it’s not portable so it needs both compile-time and run-time detection to pick the correct SIMD variant.

In this blog post we compare 3 contenders for the job of converting Ruby Strings to JSON Strings.

The C extension code used in ruby/json
The C extension code + AVX2 SIMD part of this SIMD PR
A pure-Ruby version

The pure-Ruby version is so simple it fits in a few lines (and a big Hash literal):

ESCAPE_MAP = {
  "\x0" => '\u0000', "\x1" => '\u0001', "\x2" => '\u0002', "\x3" => '\u0003',
  "\x4" => '\u0004', "\x5" => '\u0005', "\x6" => '\u0006', "\x7" => '\u0007',
  "\b" => '\b', "\t" => '\t', "\n" => '\n', "\xb" => '\u000b',
  "\f" => '\f', "\r" => '\r', "\xe" => '\u000e', "\xf" => '\u000f',
  "\x10" => '\u0010', "\x11" => '\u0011', "\x12" => '\u0012', "\x13" => '\u0013',
  "\x14" => '\u0014', "\x15" => '\u0015', "\x16" => '\u0016', "\x17" => '\u0017',
  "\x18" => '\u0018', "\x19" => '\u0019', "\x1a" => '\u001a', "\x1b" => '\u001b',
  "\x1c" => '\u001c', "\x1d" => '\u001d', "\x1e" => '\u001e', "\x1f" => '\u001f',
  '"' => '\"', '\\' => '\\\\',
}

def utf8_to_json(string)
  '"' + if /["\\\x0-\x1f]/.match?(string)
    string.gsub(/["\\\x0-\x1f]/, ESCAPE_MAP)
  else
    string
  end + '"'
end

The ESCAPE_MAP is a bit verbose but it’s a simple way to exhaustively store how to escape each character which needs to be escaped. The match? is not strictly necessary but it avoids the allocation of a new String for the common case of no character to escape.

We use ruby/json’s string encoding benchmarks with a small diff to run only those:

diff --git a/benchmark/encoder.rb b/benchmark/encoder.rb
index f0a05db..efbac2b 100644
--- a/benchmark/encoder.rb
+++ b/benchmark/encoder.rb
@@ -33,6 +33,8 @@ def implementations(ruby_obj)
 end
 
 def benchmark_encoding(benchmark_name, ruby_obj, check_expected: true, except: [])
+  return unless ["mixed utf8", "mostly utf8"].include?(benchmark_name)
   json_output = JSON.dump(ruby_obj)
   puts "== Encoding #{benchmark_name} (#{json_output.bytesize} bytes)"
 
@@ -40,6 +42,7 @@ def benchmark_encoding(benchmark_name, ruby_obj, check_expected: true, except: [
   except.each { |i| impls.delete(i) }
 
   Benchmark.ips do |x|
+    x.warmup = 5
     expected = ::JSON.dump(ruby_obj) if check_expected
     impls.values.each do |name, block|
       begin

We select the benchmarks we want with an early return and we increase warmup to increase the stability of the benchmark results. We use ONLY=json because the SIMD PR doesn’t have the json_coder variant (anyway both of these give very similar results for the selected benchmarks).

JSON String Escaping Benchmark

The benchmarks are straightforward:

# mixed utf8
JSON.generate([("a" * 5000) + "€" + ("a" * 5000)] * 500)
# mostly utf8
JSON.generate([("€" * 3333)] * 500)

They both dump to JSON an Array of 500 strings and end up generating a JSON result of 5MB.

Here are the results on my machine (AMD Ryzen 7 3700X 8-Core Processor), with frequency scaling disabled and the performance CPU governor:

C extension with CRuby 3.4.2 with YJIT (on ruby/json master):

$ ONLY=json ruby --yjit -Ilib:ext benchmark/encoder.rb
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +YJIT +PRISM [x86_64-linux]
mixed utf8    397.953 (± 4.5%) i/s    (2.51 ms/i)
mostly utf8   402.388 (± 0.5%) i/s    (2.49 ms/i)

We will use this as the baseline, specifically the mixed utf8 benchmark to keep things simple.

C extension + SIMD with CRuby 3.4.2 with YJIT (on the SIMD PR):

$ ONLY=json ruby --yjit -Ilib:ext benchmark/encoder.rb
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +YJIT +PRISM [x86_64-linux]
mixed utf8      1.498k (± 4.0%) i/s  (667.68 μs/i)
mostly utf8     1.474k (± 1.6%) i/s  (678.55 μs/i)

3.76 times faster, nice!

Pure-Ruby generator with TruffleRuby 24.1.1 JVM (on ruby/json master):

$ ONLY=json ruby -Ilib:ext benchmark/encoder.rb
truffleruby 24.1.1, like ruby 3.2.4, Oracle GraalVM JVM [x86_64-linux]
mixed utf8      8.104k (± 0.6%) i/s  (123.40 μs/i)
mostly utf8     8.070k (± 1.1%) i/s  (123.91 μs/i)

20 times faster than the baseline and 5.4 times faster than SIMD!
And, of course much simpler, this is just a few lines of Ruby code and a Regexp.

If you guessed earlier that the C extension code + SIMD would be fastest, you may be surprised to learn the pure-Ruby version is considerably faster (on TruffleRuby)! That’s the benefit of an advanced JIT compiler which understands Ruby semantics like TruffleRuby’s JIT compiler (called Graal): you get to write nice, idiomatic Ruby and the JIT compiler does the hard work of optimizing it for your target platform.

For comparison, here’s the pure-Ruby generator with CRuby 3.4.2 with YJIT (on ruby/json master):

$ ONLY=json ruby --yjit -Ilib:ext benchmark/encoder.rb
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +YJIT +PRISM [x86_64-linux]
mixed utf8     39.551 (± 0.0%) i/s   (25.28 ms/i)
mostly utf8    28.119 (± 0.0%) i/s   (35.56 ms/i)

Unfortunately, it’s 10 times slower than the baseline, demonstrating the Regexp approach is much slower on CRuby. But, we ran with YJIT enabled, so why is it so much slower?

As a disclaimer, TruffleRuby (which uses the pure-Ruby generator) is currently slower than CRuby using the C extension on JSON.generate macro benchmarks (except canada.json where it is faster). Other aspects of JSON generation need to be better optimized on TruffleRuby.

Regexp#match? Benchmark

Let’s simplify the benchmark to just Regexp#match? to understand better:

require 'benchmark/ips'

mixed_utf8 = ("a" * 5000) + "€" + ("a" * 5000)

Benchmark.ips do |x|
  x.warmup = 5

  x.report '/["\\\x0-\x1f]/.match?(mixed_utf8)' do
    raise if /["\\\x0-\x1f]/.match?(mixed_utf8)
  end
end

$ ruby --yjit bench_regexp_match.rb
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +YJIT +PRISM [x86_64-linux]
20.700k (± 2.4%) i/s   (48.31 μs/i)

$ ruby bench_regexp_match.rb
truffleruby 24.1.1, like ruby 3.2.4, Oracle GraalVM JVM [x86_64-linux]
 4.699M (± 0.3%) i/s  (212.83 ns/i)

TruffleRuby is 227 times faster than CRuby at Regexp matching on this benchmark! TruffleRuby matches 10003 bytes in 213 nanoseconds, that is 47 bytes per nanosecond, which is greater than the number of CPU cycles we have per nanosecond (this processor has a max frequency of 4.4 GHz), so what’s going on here?

How can Regexp matching be so fast on TruffleRuby?

There are multiple reasons:

Regexps on TruffleRuby use TRegex, a state-of-the-art Regular Expression Engine:
TRegex compiles regexps to finite-state automata, meaning it never goes back in the input string and will read each character at most once. Therefore, it always runs in linear time. CRuby uses backtracking which may go over the input string characters many times, which is much slower.
Backtracking is prone to ReDoS when backtracking too much while finite-state automata is immune to ReDoS. CRuby 3.2+ has a protection for most regexps against ReDoS by using some form of caching. While that caching avoids many ReDoS, it also slows down the regexp execution by needing to check or fill this cache every time it might backtrack.
TRegex JIT-compiles regexps to native code. CRuby compiles the regexp to regexp bytecode and then uses an interpreter to run that bytecode. That means it has dispatch overhead, does not optimize across bytecodes, etc. YJIT does not know anything about that regexp interpreter, it sees it as a blackbox because it is all written in C.
TruffleRuby uses splitting so that each call site for Regexp#match? and other Regexp-related methods has its own copy of the logic. With that, TruffleRuby is able to inline the JIT-compiled code for that specific Regexp in the Ruby method calling match?.
TRegex automatically uses SIMD, and specifically the best SIMD variant available on the running processor. It is able to do so because of the JIT compiler, which queries the CPU for which SIMD variants are available. TRegex does not need to dispatch between different SIMD variants like C would since the JIT compiler knows the best SIMD variant available. So we get to use SIMD instructions, but we don’t have to complicate a Ruby gem to benefit.

Specifically for this Regexp TRegex internally uses a ByteIndexOfCodePointSetNode, which automatically creates a lookup table for the character range in that regexp (["\\\x0-\x1f]) and also automatically uses SIMD. If you would like to know more about TRegex, the creator of TRegex and I did a talk about it at RubyKaigi 2021. Kevin Menard also gave a talk about ReDoS and solutions at RailsConf 2022.

The JSON String Escaping example above is not an isolated case where TruffleRuby coincidentally is faster. Having a fast regular expression engine allows for more idiomatic solutions and allows Ruby code running on TruffleRuby to be faster than C code in many cases. Here are two more real world examples where TruffleRuby uses Regexps and it’s faster than C code!

Time.new(String)

Time.new since Ruby 3.2 accepts a String argument.

CRuby implements it by parsing strings with char pointers in about 80 lines of C. TruffleRuby implements it in a straightforward way with a Regexp.

require 'benchmark/ips'

NOW_STRING = Time.now.to_s

Benchmark.ips do |x|
  x.report("Time.new(String)") do
    Time.new(NOW_STRING)
  end
end

$ ruby --yjit bench_time_new_string.rb
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +YJIT +PRISM [x86_64-linux]
Time.new(String)      1.595M (± 0.5%) i/s  (626.84 ns/i)

$ ruby bench_time_new_string.rb
truffleruby 24.1.1, like ruby 3.2.4, Oracle GraalVM JVM [x86_64-linux]
Time.new(String)      3.923M (± 5.4%) i/s  (254.90 ns/i)

TruffleRuby is 2.5 times faster here, while having a much simpler and cleaner implementation.

Let’s benchmark just the Regexp, although we use match here since we need the capture groups:

require 'benchmark/ips'

NOW_STRING = Time.now.to_s

TIME_REGEXP = /\A (?<year>\d{4,5})
(?:
  - (?<month>\d{2})
  - (?<mday> \d{2})
  [ T] (?<hour> \d{2})
     : (?<min>  \d{2})
     : (?<sec>  \d{2})
     (?:\. (?<usec> \d+) )?
 (?:\s* (?<offset>\S+))?
)?\z/x

Benchmark.ips do |x|
  x.report("TIME_REGEXP.match(NOW_STRING)") do
    raise unless TIME_REGEXP.match(NOW_STRING)
  end
end

$ ruby --yjit bench_time_new_string_regexp.rb
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +YJIT +PRISM [x86_64-linux]
TIME_REGEXP.match(NOW_STRING)      1.234M (± 0.8%) i/s  (810.47 ns/i)

$ ruby bench_time_new_string_regexp.rb
truffleruby 24.1.1, like ruby 3.2.4, Oracle GraalVM JVM [x86_64-linux]
TIME_REGEXP.match(NOW_STRING)     33.448M (± 9.9%) i/s   (29.90 ns/i)

TruffleRuby is 27 times faster for this Regexp. So the Time.new(String) benchmark is actually spending a lot of time in creating and initializing the Time object.

StringScanner#scan_integer

StringScanner#scan_integer is a method recently added to StringScanner.

CRuby implements it in 114 lines of C. TruffleRuby implements it in 18 lines of Ruby using Regexps. We use TruffleRuby master here since the method was introduced after the last TruffleRuby release.

require 'benchmark/ips'
require 'strscan'

SCANNER = StringScanner.new("123456789")

Benchmark.ips do |x|
  x.report("StringScanner#scan_integer") do
    SCANNER.reset
    raise unless 123456789 == SCANNER.scan_integer
  end
end

$ ruby --yjit bench_scan_integer.rb
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +YJIT +PRISM [x86_64-linux]
StringScanner#scan_integer     10.530M (± 0.6%) i/s   (94.97 ns/i)

$ ruby bench_scan_integer.rb
truffleruby 25.0.0-dev-48a0a626, like ruby 3.3.5, Oracle GraalVM JVM [x86_64-linux]
StringScanner#scan_integer     30.230M (± 3.0%) i/s   (33.08 ns/i)

TruffleRuby is about 3 times faster. BTW, TruffleRuby implements StringScanner entirely in Ruby code. Ruby code is not only shorter and more expressive, it’s also more correct. To that point, while implementing new StringScanner methods we found 6 issues with the C extension implementation of StringScanner: 2 segfaults and 4 inconsistent/incorrect behaviors.

Let’s also benchmark just the Regexp for completeness:

require 'benchmark/ips'

INPUT = "123456789"

Benchmark.ips do |x|
  x.report("/\A[+-]?\d+/.match(INPUT)") do
    raise unless /\A[+-]?\d+/.match(INPUT)
  end
end

$ ruby --yjit bench_scan_integer_regexp.rb
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +YJIT +PRISM [x86_64-linux]
/A[+-]?d+/.match(INPUT)      1.972M (± 0.8%) i/s  (507.00 ns/i)

$ ruby bench_scan_integer_regexp.rb
truffleruby 25.0.0-dev-48a0a626, like ruby 3.3.5, Oracle GraalVM JVM [x86_64-linux]
/A[+-]?d+/.match(INPUT)     61.086M (± 6.9%) i/s   (16.37 ns/i)

TruffleRuby is 31 times faster for this Regexp.

Conclusion

Sometimes, such as in the cases shown in this blog post, the idiomatic pure-Ruby solution is also the fastest and even faster than C code or inline assembly. This is not new on TruffleRuby, pure-Ruby code is often faster than everything else, including C extensions on CRuby. Ruby code is more expressive and higher-level which makes some optimizations possible that otherwise wouldn’t be if written in a lower-level language.

If you want to run idiomatic Ruby code fast, give TruffleRuby a try.

Benchmarking Ruby Parsers

2024-10-27T00:00:00+02:00

The new Prism parser has become the default in Ruby 3.4.0 preview 2.

Let’s benchmark Ruby parsers and find out how fast they are.

We run benchmarks on Ruby 3.4.0 preview 2 with YJIT (ruby 3.4.0preview2 (2024-10-07 master 32c733f57b) +YJIT +PRISM [x86_64-linux]), on an AMD Ryzen 7 3700X 8-Core Processor and a NVMe M.2 SSD, on Linux, with frequency scaling disabled and the performance CPU governor.

We will compare:

prism 1.2.0
parser 3.3.5.0
ruby_parser 3.21.1
Ripper from ruby 3.4.0preview2
RubyVM::AbstractSyntaxTree from ruby 3.4.0preview2

Which are the latest releases at the time of writing.

Our corpus will be all .rb files in railties 7.2.1.2, that is 151 files, consisting of 14625 lines and 455673 bytes in total.

Let me start with: benchmarking parsers fairly is hard.

All these parsers return different ASTs. Some have more information than others and might be more or less practical to use, but they are all ASTs representing Ruby code.

One major concern performance-wise is that some parsers might allocate Ruby objects lazily and instead only keep some C structs around. This is the case of RubyVM::AbstractSyntaxTree, as can be seen by looking at allocations:

Prism::Node allocations for Prism.parse
Parser::AST::Node allocations for Parser gem
Sexp allocations for RubyParser
230854 Array allocations for Ripper.sexp
RubyVM::AbstractSyntaxTree::Node allocations for RubyVM::AbstractSyntaxTree

36835 nodes for Prism is consistent with the 37762 nodes for the Parser gem. 79749 for RubyParser seems a bit much but still the same order of magnitude. 230854 for Ripper looks to be an outlier, but one inaccuracy here is Ripper uses Arrays not only for nodes but also for positions like [line, column] and Arrays might be used for other things too. RubyVM::AbstractSyntaxTree only allocates 151 nodes (one per file), which proves it creates Ruby node objects lazily.

One way to avoid that laziness is to walk all nodes in the AST. Doing that, we get:

Prism::Node allocations for Prism.parse
Prism::Node allocations for Prism.parse + walk
Parser::AST::Node allocations for Parser gem
Parser::AST::Node allocations for Parser gem + walk
Sexp allocations for RubyParser
Sexp allocations for RubyParser + walk
230854 Array allocations for Ripper.sexp
230854 Array allocations for Ripper.sexp + walk
RubyVM::AbstractSyntaxTree::Node allocations for RubyVM::AbstractSyntaxTree
RubyVM::AbstractSyntaxTree::Node allocations for RubyVM::AbstractSyntaxTree + walk

So RubyVM::AbstractSyntaxTree needs special care to be benchmarked fairly.

While we are at it, here are the total number of Ruby-level allocations:

  75610 total allocations for Prism.parse
 112445 total allocations for Prism.parse + walk
  diff: +36835 Arrays

 823320 total allocations for Parser gem
 823320 total allocations for Parser gem + walk

1028578 total allocations for RubyParser
1028578 total allocations for RubyParser + walk

 341687 total allocations for Ripper.sexp
 341687 total allocations for Ripper.sexp + walk

  16044 total allocations for RubyVM::AbstractSyntaxTree
  92495 total allocations for RubyVM::AbstractSyntaxTree + walk
  diff: +38568 Arrays, +36111 RubyVM::AbstractSyntaxTree::Node, +1689 Strings, +83 Regexps

We can see walking the AST:

causes extra Array allocations for Prism because each call to child_nodes allocates an Array.
does not allocate for the Parser gem because it keeps child nodes in an Array internally and #children just returns that.
does not allocate for RubyParser and Ripper because their representation is just Arrays (or Sexp, a subclass of Array).
allocates Arrays and nodes for RubyVM::AbstractSyntaxTree.

Parsing and Walking

So to try to benchmark RubyVM::AbstractSyntaxTree fairly we can walk the AST for all parsers:

Calculating -------------------------------------
               Prism.parse + walk     27.087 (± 0.0%) i/s   (36.92 ms/i) -    136.000 in   5.021906s
                Parser gem + walk      2.246 (± 0.0%) i/s  (445.33 ms/i) -     12.000 in   5.343971s
                RubyParser + walk      1.632 (± 0.0%) i/s  (612.90 ms/i) -      9.000 in   5.516460s
               Ripper.sexp + walk     11.265 (± 0.0%) i/s   (88.77 ms/i) -     57.000 in   5.065812s
RubyVM::AbstractSyntaxTree + walk     24.272 (± 4.1%) i/s   (41.20 ms/i) -    122.000 in   5.031333s

Comparison:
               Prism.parse + walk:       27.1 i/s
RubyVM::AbstractSyntaxTree + walk:       24.3 i/s - 1.12x  slower
               Ripper.sexp + walk:       11.3 i/s - 2.40x  slower
                Parser gem + walk:        2.2 i/s - 12.06x  slower
                RubyParser + walk:        1.6 i/s - 16.60x  slower

However, this gives an advantage to Ripper, the Parser gem, and RubyParser as they don’t need extra Array allocations to walk the AST. Despite that advantage, all three are slower than Prism.

Prism and RubyVM::AbstractSyntaxTree are more than 10 times as fast as the Parser gem and RubyParser in this benchmark. That’s quite the speedup.

Parsing and not Walking

Or we can exclude RubyVM::AbstractSyntaxTree and not walk the AST:

Calculating -------------------------------------
         Prism.parse     32.590 (± 0.0%) i/s   (30.68 ms/i) -    165.000 in   5.063030s
          Parser gem      2.256 (± 0.0%) i/s  (443.30 ms/i) -     12.000 in   5.319708s
          RubyParser      1.638 (± 0.0%) i/s  (610.58 ms/i) -      9.000 in   5.495618s
         Ripper.sexp     11.834 (± 0.0%) i/s   (84.51 ms/i) -     60.000 in   5.077234s

Comparison:
Prism.parse:       32.6 i/s
Ripper.sexp:       11.8 i/s - 2.75x  slower
 Parser gem:        2.3 i/s - 14.45x  slower
 RubyParser:        1.6 i/s - 19.90x  slower

Just Parsing

We can also find a way to measure just the parsing itself and the creation of C data structures to hold the parse result. In this configuration the parser will allocate as few Ruby objects as possible. This approach is feasible for Prism and for RubyVM::AbstractSyntaxTree. Prism.profile parses to C structs and does nothing else. RubyVM::AbstractSyntaxTree.parse uses parse.y, the LALR-based parser used in CRuby 3.3 and before. RubyVM::AbstractSyntaxTree.parse does very little extra work besides parsing to C structs, which I double-checked by measuring the time around yyparse() (the entry point for parse.y) for a large file. About 99.8% of the time is spent inside yyparse(), so RubyVM::AbstractSyntaxTree.parse is a good approximation for yyparse()/parse.y.

151 files, 14625 lines, 455673 bytes

0 total allocations for Prism.profile

16095 total allocations for RubyVM::AbstractSyntaxTree:
  15559 Strings, 302 Datas, 151 RubyVM::AbstractSyntaxTree::Node, 83 Regexps

Calculating -------------------------------------
             Prism.profile    100.447 (± 2.0%) i/s    (9.96 ms/i) -    510.000 in   5.079433s
RubyVM::AbstractSyntaxTree     39.186 (± 2.6%) i/s   (25.52 ms/i) -    198.000 in   5.056720s

Comparison:
             Prism.profile:      100.4 i/s
RubyVM::AbstractSyntaxTree:       39.2 i/s - 2.56x  slower

Prism parses this corpus of 455673 bytes in 9.96 ms, which is 43.6 MB/s, compared to 17.0 MB/s for parse.y. In other words, Prism is 2.56 times as fast as parse.y in parsing this corpus of Ruby code!

There are 16095 allocations for RubyVM::AbstractSyntaxTree, most of which are Strings. Of those Strings, most are the one String per line (14625 lines) that the lexer used by yyparse() allocates (in rb_parser_lex_get_str()). We see 151 RubyVM::AbstractSyntaxTree::Node: one per file, as expected.

From Source To CRuby Bytecode

We have one more benchmark to measure compilation from source code to CRuby bytecode using RubyVM::InstructionSequence.compile_prism and RubyVM::InstructionSequence.compile_parsey (for parse.y). In this benchmark the input and output are exactly the same for both variants so it is as fair as it gets.

151 files, 14625 lines, 455673 bytes

10477 total allocations for RubyVM::InstructionSequence.compile_prism:
  3753 Strings, 3389 Arrays, 2717 Imemos, 378 Hashs, 151 RubyVM::InstructionSequence, 83 Regexps, 6 Structs

28782 total allocations for RubyVM::InstructionSequence.compile_parsey, including 151 RubyVM::InstructionSequence:
  18476 Strings, 4237 Arrays, 2868 Imemos, 2501 Datas, 377 Hashs, 166 Regexps, 151 RubyVM::InstructionSequence, 6 Structs

Calculating -------------------------------------
 RubyVM::InstructionSequence.compile_prism     36.499 (± 0.0%) i/s   (27.40 ms/i) -    183.000 in   5.014395s
RubyVM::InstructionSequence.compile_parsey     25.050 (± 0.0%) i/s   (39.92 ms/i) -    126.000 in   5.030555s

Comparison:
 RubyVM::InstructionSequence.compile_prism:       36.5 i/s
RubyVM::InstructionSequence.compile_parsey:       25.1 i/s - 1.46x  slower

That’s a nice 1.46x speedup for Prism here. This improvement makes Ruby programs and applications boot faster on CRuby, by reducing the amount of time spent parsing and compiling to bytecode. Prism is also used in other Ruby implementations and has sped up parsing and boot times there as well.

Script

Here is the script I used to benchmark and get allocation counts.

Conclusion

Benchmarking Ruby parsers is trickier than one might expect, but overall the result is clear: Prism is the fastest Ruby parser.

When parsing and walking, Prism is 12% faster than RubyVM::AbstractSyntaxTree and 12x as fast as the Parser gem!
When parsing to C structs, Prism is 2.56x as fast as parse.y. Who thought parsing Ruby in C could be made more than twice as fast?
When compiling to bytecode, Prism is 1.46x as fast as parse.y, speeding up boot times.

Beyond raw performance, Prism also provides a great and well documented API, which makes it much easier to build fast tooling for and in Ruby.

Benchmarking CRuby, MJIT, YJIT, JRuby and TruffleRuby

2022-01-06T00:00:00+01:00

In this blog post we benchmark many Ruby versions and the latest Ruby Just-in-Time compilers (JITs) on the newest Ruby benchmark suite, yjit-bench. As a teaser, the geometric mean speedups compared to CRuby 3.1 on these 14 benchmarks are: MJIT 1.26x, YJIT 1.39x, JRuby 1.86x and TruffleRuby 6.23x. Read on to find more about the benchmarks and gain insights on these speedups. This blog post is also available on Medium.

Approach

Rubies

We benchmark the following Ruby versions:

CRuby 2.0: ruby 2.0.0p648 (2015-12-16 revision 53162) [x86_64-linux]
CRuby 2.7: ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
CRuby 3.0: ruby 3.0.3p157 (2021-11-24 revision 3fb7d2cadc) [x86_64-linux]
CRuby 3.1: ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) [x86_64-linux]
CRuby 3.1+MJIT: ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) +MJIT [x86_64-linux]
CRuby 3.1+YJIT: ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) +YJIT [x86_64-linux]
JRuby+invokedynamic, on the system JDK 11: jruby 9.3.2.0 (2.6.8) 2021-12-01 0b8223f905 OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]
TruffleRuby JVM EE (JDK 11): truffleruby 22.0.0-dev-9bdb2dfc, like ruby 3.0.2, GraalVM EE JVM [x86_64-linux]

We ran all the benchmarks on 4 January 2022 on an AMD Ryzen 7 3700X 8-Core Processor, with 32GB of memory and a NVMe M.2 SSD, on Linux.

We chose CRuby 3.1 as the baseline (= 1.0x) in order to easily compare how much MJIT and YJIT gain over CRuby without JIT (the default).

CRuby 2.0 can only run a small subset of the benchmarks. CRuby 3.1 includes both MJIT and YJIT which makes it convenient to measure them. For TruffleRuby we picked a single variant (out of Native/JVM CE/EE) to keep the charts readable as there are already many Ruby implementations. We used TruffleRuby from the release branch of the upcoming GraalVM 22.0.

yjit-bench

We run all 14 macro benchmarks from the yjit-bench benchmark suite at commit 1751916ceb, except for the jekyll benchmark because it is very noisy and does not work on several Ruby versions. We do not report the micro benchmarks (example), since those seem to be there mostly to tweak JITs rather than to compare general Ruby performance. Of course micro benchmarks do not represent anything close to a real workload. As a fun fact, TruffleRuby optimizes away (>1000x faster) half of those micro benchmarks.

The yjit-bench benchmark suite contains a variety of benchmarks, and is maintained principally by Noah Gibbs (who also worked on benchmarks for Ruby 3x3) and Maxime Chevalier-Boisvert (leading YJIT) at Shopify. One significant advantage of this benchmark suite is all benchmarks are really easy to run, any setup (including bundle install and db:migrate) is done automatically and running a benchmark is just ruby -I./harness benchmarks/some/benchmark.rb. There are no subprocesses, no magic, and so it is really easy to run them and profile them.

Warmup

We measure peak performance, i.e., performance after enough warmup, that is performance after the relevant methods are compiled. This is what is typically used to compare JITs, and it is also what this benchmark suite aims to measure. All benchmarks were run with enough warmup. The warmup is automatically detected by using a custom harness which tracks the median absolute deviation over all iterations so far and keeps running more iterations until the median absolute deviation threshold is reached (more details here).

In general, more advanced JITs need more warmup. The TruffleRuby and GraalVM teams are working on this issue by using multi-tier compilation, persisting the JITed code and optimizing interpreter performance, among other ideas.

The plots report the median as the bar heights and the median absolute deviation as error bars, for the second half of all iterations run. The first half is considered warmup and typically more noisy. Those estimators are chosen because they are robust (which means they are not affected by outliers like a GC in one iteration), unlike the usual mean and standard deviation. As one can see, the error bars are very small for almost all benchmarks, which is a further indication of enough warmup and having reached stable performance.

The median is the middle value when we sort all values. The median absolute deviation is the median of all absolute deviations, which are the absolute value of the difference between a given value and the median. In other words, the median represents the middle value (typically the closest value to most values), and the median absolute deviation shows us how far other values are from the median.

Benchmarks

The activerecord benchmark uses Rails’ ActiveRecord to query the database and return the title of the first Post. The sqlite3 in-memory database is used for convenience (sqlite3 is a C extension gem). On JRuby, activerecord-jdbcsqlite3-adapter is used instead.

One might think this benchmark is dominated by database performance but it seems clearly not the case, with YJIT being 1.42x the speed of CRuby 3.1 and TruffleRuby showing a 4.54x speedup. There is quite a bit of Ruby logic in ActiveRecord and some in the sqlite3 gem, which Ruby JITs can potentially optimize. TruffleRuby might also run the C extension faster since TruffleRuby JIT-compiles C extensions together with Ruby code.

The binarytrees benchmark (from the Computer Language Benchmarks Game, abbreviated CLBG later) creates binary trees using lots of recursive calls. It also stresses the GC due to many allocation and keeps one tree alive for a while. MJIT is doing well here with a 1.79x speedup, JRuby 2.06x and TruffleRuby 2.73x.

The erubi benchmark renders an ERB template (gem server’s index with 322 gems) using the erubi gem, which is the default template engine in Rails. The rendered template has a length of 166563 bytes. TruffleRuby performs great here due to its very efficient String concatenation and achieves a 5.26x speedup. YJIT is currently not optimizing this benchmark well due to not supporting a bytecode instruction used in this benchmark.

The erubi_rails benchmark is a small Rails application which renders an ERB view coming from Discourse. The rendered template has a length of 9369 bytes. The benchmark renders a topic’s page 100 times per iteration, in process (no network involved). No database is involved in this benchmark as it focuses on erubi in the context of Rails. YJIT gets a speedup of 1.31x, JRuby of 1.82x and TruffleRuby of 3.3x.

The fannkuchredux benchmark (from CLBG) creates and manipulates Arrays extensively. MJIT and YJIT seem to not achieve any speedup here, probably because the benchmark is heavy on Array operations and Array operations are defined in C in CRuby and so MJIT/YJIT cannot optimize them (except for simple a[i] and a[i] = v but they do not seem to optimize it better than the interpreter in this benchmark). JRuby and TruffleRuby OTOH can optimize and JIT compile Array operations, and TruffleRuby uses storage strategies for Arrays. JRuby gets a speedup of 5.53x, TruffleRuby gets a speedup of 33.88x.

The hexapdf benchmark uses the hexapdf gem, wraps a long text (The Odyssey) to a line width of 50 characters and renders it to a PDF file, saved to /tmp. hexapdf is written entirely in Ruby (no C extension). Wrapping text is heavy on arithmetic operations and JITs often do a good job on those. YJIT gets a speedup of 1.4x, TruffleRuby of 4.38x (with some variability). CRuby 3.1 MJIT fails this benchmark with buffer error (Zlib::BufError).

The lee benchmark is from Chris Seaton (see his blog post) and some of his work during his PhD. The benchmark lays out the wires on a circuit board in a manner which avoids crossing wires. This is the sequential version of the benchmark. It seems worth noting the approach itself is from scientific literature and Chris wrote this in straightforward pure Ruby, not particularly optimizing it for any Ruby implementation. In his blog post he found that TruffleRuby was over 10x faster than CRuby 2.7. On latest TruffleRuby, we see a speedup of 22x!

The liquid-render benchmark renders all the performance test Liquid templates which are parsed before starting the benchmark. As you may know, rendering Liquid templates quickly is important for Shopify. Liquid is not a typical template engine, it does not generate Ruby code and eval it and then call the generated method (like ERB engines). Instead, Liquid interprets an AST representing the template for security reasons and avoiding eval. YJIT gets a speedup of 1.61x and TruffleRuby gets a speedup of 4.86x, some of it due to TruffleRuby’s advanced String representation (Ropes) and the rest due to JIT compiling the Liquid interpreter better.

The mail benchmark parses an email with the mail gem (more context). Some of the parsing is done by parsers generated by Ragel. YJIT and TruffleRuby show nice speedups.

The nbody benchmark (from CLBG) models the orbits of Jupiter, Saturn, Uranus and Neptune around the Sun. The benchmark consists mainly of arithmetic computations and accessing instance variables heavily (planets are represented as objects). JRuby reaches a good speedup of 4.43x. TruffleRuby is about 50x the speed of CRuby 3.1, which is simply incredible.

The optcarrot benchmark is one of the famous benchmarks for Ruby 3x3 (by @mame, my blog post about it). It is a NES emulator and this benchmark renders 200 frames of the Lan Master game per iteration. MJIT is known to be about 3x faster than CRuby 2.0 on this benchmark. In this run, 3.1 MJIT is exactly 3.0x as fast as CRuby 2.0. TruffleRuby is known to be good at it since the benchmark came out, and it reaches a 10.4x speedup over CRuby 2.0.

The psych-load benchmark parses a set of YAML files using Psych.load (same as YAML.load). Psych 2.2.4 is used on CRuby 2.0 instead of Psych 4.0.1, since it is the latest Psych version compatible with 2.0. Although a significant part of the benchmark is likely spent inside the psych C extension (which uses the libyaml YAML parser), we see the time is not only spent in C extensions (where CRuby JITs cannot help) but also in Ruby code as MJIT and YJIT show some speedups here.

The railsbench benchmark is inspired from k0kubun’s railsbench, which is based on headius’s pgrailsbench. It is a small blog-style Rails app scaffolded with a Post model. This variant uses sqlite3 as the database for convenience (stored on disk), and runs everything in a single process (much simpler to run). The benchmark repeatedly visits the index of the posts (both as HTML and JSON) and each of the 100 posts in a pseudorandom order with a fixed seed. It uses the default ERB templates generated by Rails. MJIT gets a speedup of 1.09x (it took a while for MJIT to speedup Rails), YJIT of 1.33x, JRuby of 1.13x and TruffleRuby of 3.27x. In other words, on this small Rails benchmark, TruffleRuby response times are more than 3 times faster than CRuby 3.1!

The rubykon benchmark (by @PragTob) is well known from the blog posts by its author. Rubykon is an AI for the Go board game. YJIT reaches a 2.15x speedup and JRuby 2.59x. TruffleRuby is ahead with a 9.54x speedup.

Summary

The benchmarks we just analyzed are of various size and have many different characteristics, notably some are more realistic and some less. I think a geometric mean over their results alone is not really a good summary given the variety. But yet, it is nice to have a one-number summary for these benchmarks. Note that CRuby 2.0 can only run 7 benchmarks out of 14, and MJIT runs 13 out of 14 (not hexapdf). The others Ruby versions run all 14 benchmarks. Error bars are not meaningful on this chart.

The chart satisfyingly goes higher from left to right. This means, for the geomean on those benchmarks, that newer CRuby versions are faster than older versions, that MJIT and YJIT are faster than without them, that JRuby’s geomean is faster than MJIT and YJIT, and that TruffleRuby is simply in a different league than other Ruby implementations with an overall 6.23x geomean speedup.

On the 7 benchmarks that CRuby 2.0 can run, CRuby 3.1 MJIT is 1.74x as fast as CRuby 2.0’s geomean (binarytrees 1.9x, erubi 1.25x, fannkuchredux 1.33x, nbody 2.17x, optcarrot 3.0x, psych-load 1.12x, rubykon 2.08x). That’s closer to 2x than 3x (i.e., Ruby 3x3) but it is a significant improvement since CRuby 2.0.

Conclusion

I think the numbers speak for themselves.

My take on these results (as the TruffleRuby lead), is that TruffleRuby is able to understand and optimize Ruby code significantly better than any other implementation of Ruby. It seems unlikely other Ruby implementations would ever reach TruffleRuby’s level of performance. TruffleRuby uses the GraalVM JIT compiler, one of the most advanced JIT compilers available today. TruffleRuby, through the Truffle framework, is in direct communication with the JIT (and so are the TruffleRuby and GraalVM Compiler teams) and can easily tell the JIT what is worth optimizing/profiling/inlining and what is not, and when it should deoptimize to the interpreter to reprofile or recompile differently. The Truffle framework makes it much easier to optimize many parts of Ruby. As an example, adding an inline cache is just 2 lines of code (and so TruffleRuby has >100 inline caches), while in other VMs few inline caches are used because of the large complexity involved to add a single one. Finally, the GraalVM architecture makes it possible to inline and just-in-time compile together Ruby code, Java code, C extensions, Ruby regular expressions and any other language implemented on GraalVM, which brings this awesome performance.

Review: Ruby Installers and Ruby Switchers

2021-06-04T00:00:00+02:00

In this post I review the most popular Ruby installers (making it easier to install a Ruby) and Ruby switchers (to switch between different Rubies conveniently).

I contributed to all 3 Ruby installers when adding support for TruffleRuby, and so I have experience both with their codebase and their usage.

I am not reviewing Ruby installers or switchers for Windows as I have no experience with them.

TLDR: feel free to jump to the Conclusion and Recommendation.

Ruby Installers

RVM

RVM is probably the oldest and certainly the Ruby installer with the most features. That’s actually a doubled-edged sword, as it means RVM has a much larger codebase than all the others (25000 SLOC of Bash), and that makes it hard to maintain and to keep correct (note: I’m not counting test SLOC). Some features like gemsets seem much less needed nowadays as there is Bundler.

To give an idea, when I added support for TruffleRuby in RVM, I had many uncertainties, and the large codebase plus a lot of global variables made it really tricky. As a result, there were multiple bugs and a lot more effort than for other Ruby installers. Also, RVM actually changes standard Ruby executables like rake, which I really dislike as a Ruby implementer because it tends to break things in subtle ways (this behavior was later disabled when installing TruffleRuby).

To be fair, RVM served me well at the time other Ruby installers/switchers did not exist.

RVM supports a lot of platforms, as well as ancient Rubies. More platforms can be nice for beginners, but platform-specific code eventually always breaks due to incompatible changes in operating systems (e.g., package names).

Pros:

Most features
Automatically installs system packages for many platforms
Support the most Rubies, even very old ones (some with patches)
Binary builds of some CRuby versions on some platforms
Supports building -head versions

Cons:

The built Ruby can differ quite a bit from a standard build, which can cause extra issues
Huge (25000 SLOC of Bash), hard to maintain, and due to that many bugs

ruby-build

ruby-build is the best-maintained Ruby installer. Active maintainers include @hsbt from the CRuby core team, others and myself.

It offers a wide choice of Ruby versions. It also offers -dev versions, which is incredibly useful to let users check Ruby implementation bug fixes, or simply to try the latest/fastest/best build of a Ruby implementation.

The design is really straightforward (one definition per Ruby version), and as a result it’s much more reliable. Users update ruby-build regularly as that is the way to get new versions. This is great to also ship bug fixes quickly. The release process is entirely automated (kudos to @mislav).

ruby-build does not install system packages for you, but documents them on their wiki, which I find is much more future-proof.

Pros:

Actively maintained
Small, 1390 SLOC of Bash
Supports building -dev versions

Cons:

Does not install system packages on its own (but that is more future-proof, and building Rubies tend to fail clearly when there is missing dependency)

ruby-install

ruby-install is the simplest Ruby installer.

It has an interesting design where installing new versions does not require to update ruby-install. Instead, ruby-install --latest is able to fetch the latest versions from ruby-versions and install them. This is another double-edged sword. On one hand it’s very elegant, and super easy to update. On the other hand, if any other change is needed (e.g., different system packages, the download URL changed, etc), that actually requires ruby-install changes, and users seem less used to update ruby-install. ruby-versions only contains Ruby versions and checksums, not URLs or packages, which IMHO feels too limited.

As an example, even though I made a PR to address a changed dependency the day it was reported, it took 8 months to have a ruby-install release with that (to be fair, there was an easy workaround).

ruby-install does not support development builds by design, which is IMHO a big limitation.

Pros:

Tiny, 866 SLOC of Bash
Elegant design
Super easy to install the latest Ruby releases, if they don’t require ruby-install changes

Cons:

Does not support -dev builds
Infrequent releases
Users are not used to update ruby-install
Cuts a few corners in the name of simplicity

Ruby Switchers

RVM

RVM is both a Ruby installer and Ruby switcher so it appears here too. The points above really apply to both so I will not repeat them here.

The gemset feature tends to IMHO add more complexity than it helps, by making switching a 2-dimensional operation (ruby, gemset) instead of just 1 dimension (ruby). Also gemsets have symlimks in their path (not expanded, that’s how RVM knows which gemset is active), and this can occasionally cause issues.

Pros:

Most features

Cons:

Overrides cd
Loads thousands of lines of Bash in your shell
Many features which are of little use nowadays

rbenv

rbenv is a simple Ruby switcher. It uses small shims (executables) to always redirect to the currently-selected Ruby. There is an overhead of that, in the order of 11 ms on my machine (pretty small, considering that ruby -e 0 takes 38 ms on my machine for ruby-3.0.1). OTOH, it guarantees it will never pick the wrong Ruby which is obviously important.

For me, the killer feature of rbenv is it does not mess with RubyGems. Specifically, it does not set GEM_HOME or GEM_PATH (unlike the other Ruby switchers), and so the gems corresponding to the selected Ruby are always correctly used. That also means it expects that Rubies are installed in user-writable locations (or one uses gem install --user-install or set Bundler path), but I think that’s a reasonable assumption for a Ruby switcher.

asdf is a multi-language switcher with a very similar design to rbenv and it uses ruby-build.

Pros:

Just works
Also works for non-Bash shells like fish
Reliable, does not mess with GEM_HOME
Well maintained, least number of open issues
Small, 1500 SLOC of Bash

Cons:

Small overhead on startup (~11 ms) due to shims
which ruby-related-executable is not enough to show the actual executable path, one needs rbenv which instead

chruby

chruby is a minimal Ruby switcher. In fact, it’s so small it fits in a 100 lines of Bash. After all, switching Rubies only requires changing PATH and that’s it.

Unfortunately, chruby does one additional thing which is to set GEM_HOME. That can be convenient to install gems when a Ruby is installed in a non user-writable location. Unfortunately the way it’s done is incorrect for non-CRuby. I made a PR to fix this but 18 months after it was merged there is still no release. So at this point I feel very disappointed about the maintenance of chruby.

I still like chruby though, and I use a branch which simply doesn’t set GEM_HOME. That’s just 75 SLOC of Bash, one cannot get much simpler for switching Rubies. This is probably the most future-proof Ruby switcher out there.

FWIW, I made another PR to chruby to use a similar approach, but still set GEM_HOME if the default Gem home is not writable. After lots of efforts, and even getting consensus on the related RubyGems issue for the general approach, the PR is still not merged so I have not much hope for that (maybe it is time to hard fork chruby …).

Pros:

Very simple and minimal, 149 SLOC of Bash

Cons:

Sets GEM_HOME incorrectly on non-CRuby
Requires to be loaded in the shell (there is chruby-fish though)
Too rare releases, not addressing bugs promptly

Conclusion and Recommendation

In summary:

I recommend to use ruby-build over RVM or ruby-install as IMHO it is a better Ruby installer in about every way.
I recommend to use rbenv over RVM or chruby as it is the only Ruby switcher not messing with gem paths.

The rbenv/ruby-build combo is fairly natural and clearly the best maintained.

Alternatively, you can try ruby-build + my branch of chruby which does not set GEM_HOME. This is what I use and it works perfectly from my experience.

I would recommend against RVM because of the sheer complexity and the huge codebase which is basically impossible to keep bug-free.

Note that while I might criticize various softwares in this blog post, I have a lot of respect for their authors, and I know it is a hard job to maintain such software (written in Bash, one of the worst programming languages).

Correct Delegation with Ruby 2.6, 2.7 and 3.0

2021-02-13T00:00:00+01:00

While looking at how a few gems handle delegation on Ruby 2.7, I noticed that many of them are unfortunately incorrect. The official blog post about keyword arguments changes in Ruby 2.7 and Ruby 3.0 is rather long and might be unclear. So I will keep this one really short and to the point.

I am a CRuby committer which discussed and designed the keyword arguments changes along with Jeremy Evans and Yusuke Endoh.

The Only Correct Way

There is actually just a single way to do delegation that works in Ruby <= 2.6, in Ruby 2.7 and in Ruby >= 3:

def foo(*args, &block)
  target(*args, &block)
end
ruby2_keywords :foo if respond_to?(:ruby2_keywords, true)

It’s simple: ruby2_keywords is the only way to achieve correct delegation on Ruby 2.7. The approach above is used in Rails, etc.

You can also add the ruby2_keywords gem to remove the respond_to? check and be able to use:

ruby2_keywords def foo(*args, &block)
  target(*args, &block)
end

Some Frequent Incorrect Ways

# Broken on 2.6 and 2.7
def foo(*args, **kwargs, &block)
  target(*args, **kwargs, &block)
end

is incorrect on Ruby 2.6 and on Ruby 2.7.

# Broken on 2.7 and 3.0
def foo(*args, &block)
  target(*args, &block)
end

is incorrect on Ruby 2.7 and on Ruby 3.0.

Some try a combination of the above:

# Broken on 2.7
if RUBY_VERSION < "2.7"
  def foo(*args, &block)
    target(*args, &block)
  end
else
  def foo(*args, **kwargs, &block)
    target(*args, **kwargs, &block)
  end
end

But that’s still incorrect on Ruby 2.7 (like the first example, foo({}) passes no arguments at all).

Conclusion

Use ruby2_keywords for delegation, that’s the only correct way, as shown above.

If you maintain some Ruby code or a gem, search for methods delegating (= receiving + passing) keyword arguments with **, like the first and third incorrect patterns. Delegation with ** does not work in Ruby 2.6 and 2.7, use ruby2_keywords + *args instead.

All methods delegating with *args should use ruby2_keywords for Ruby >= 2.7, as long as the target method might have keyword arguments.

Post Scriptum

Needing to use ruby2_keywords explicitly for delegation is unfortunate, I wish there would be a more natural way to express delegation in Ruby 2.7. Unfortunately there is not.

I once proposed to enable ruby2_keywords by default to preserve compatibility, but this was rejected. It would also have provided better warnings for delegation. We could have postponed the delegation-related changes in order to migrate delegation code once (*args -> *args, **kwargs) and not twice (*args -> ruby2_keywords -> *args, **kwargs / (...)), but matz wanted to make all keyword arguments deprecations at once.

At least now we have Ruby 3.0, so one can more easily find problematic delegation code as it just breaks on Ruby 3.0 if not done correctly. Note that keyword argument warnings are disabled by default in Ruby 2.7.2. Use RUBYOPT=-W:deprecated command or Warning[:deprecated] = true in Ruby code to see them.

There is the def foo(...); target(...); end syntax but it fails to parse on Ruby <= 2.6, and only works with no other arguments at all in Ruby 2.7.0 - 2.7.2, therefore it has extremely limited applicability these days and I just ignore it. Note that (arg, ...) might finally get backported to 2.7.3. It might make sense to consider ... when dropping Ruby 2.6 support.

When dropping Ruby 2 support entirely, one can switch to (*args, **kwargs)-style delegation, as that works correctly on Ruby >= 3.

Another way would be to explicitly not support Ruby 2.7, then the last example with the version check would work. It is probably impractical for most gems to do that though.

The Ruby Spec Suite Compatibility Report

2020-06-27T00:00:00+02:00

Introduction

For a long time I wanted to show visually the compatibility of Ruby implementations with ruby/spec. This is not as easy as it sounds. First, we would like the same version of ruby/spec for all tested Ruby implementations, but in general each Ruby implementation uses a different version (except when synchronized once a month). We would like the same Ruby compatibility version (e.g., 2.6), but some Ruby implementations do not target 2.6 currently. And finally the number of specs (that is, of it examples) on a given platform should be fairly constant, which required a few fixes in ruby/spec.

As a quick recap, ruby/spec and CRuby tests are the two major test suites for Ruby compatibility. ruby/spec has the advantage that each spec example has a description of a specific behavior it tests. CRuby tests on the other hand often require to reverse-engineer a test to understand what behavior(s) it intends to test, and they are more coarse-grained. In this post I focus on ruby/spec.

Approach

In this first report I ran the specs on CRuby 2.6.6, TruffleRuby, JRuby, Opal and Artichoke on Linux. I used development versions because I want to use the same specs for all implementations when possible, along with the corresponding tags – that is specs marked as failing – which are only available from the corresponding repository’s development branch.

The specific development versions and command lines used are shown in this gist. We use the number of examples that CRuby runs, which is always equal or slightly higher than the number of examples when running on other implementations.

The setup for running specs on Opal and on Artichoke is more complex – due to typically not having access to the real filesystem – so I used the version of ruby/spec that they import, which is older and therefore has less specs. Opal currently targets Ruby 2.5 and Artichoke is still an early Ruby implementation. They both use an include list of specs to run, and so might pass more than tested in CI. So please consider numbers for those Ruby implementations with a grain of salt.

Below you can see two totals, because some implementations do not support the Ruby C-API:

All specs except C-API specs.
All specs.

Without further ado, here is the data.

Results

Group	CRuby	TruffleRuby	JRuby	Opal	Artichoke
Command-line 141 specs
Language 2367 specs
Core Library 20841 specs
Standard Library 6891 specs
Security 40 specs
TOTAL without C-API specs 30280 specs	30280 passing	29382 passing	27875 passing	11101 passing	1694 passing
C-API 1325 specs
TOTAL 31605 specs	31605 passing	30672 passing	27875 passing	11101 passing	1694 passing

Interpretation

The interpretation that follows is mine, and I tried to keep it as neutral and factual as possible. I apologize if there is any inaccuracy or subjective opinion: let me know, I am happy to update the blog post based on justified comments.

As expected, CRuby passes all specs. This is by design, ruby/spec only includes specs that pass on CRuby since it is the standard implementation. There is a rare exception for specs specifying the correct behavior for known bugs in CRuby (ruby_bug in MSpec) which we can ignore here (there are only 3 ruby_bug specs currently in ruby/spec).

We see that TruffleRuby passes 97% of all specs which is quite impressive. To be fair, TruffleRuby focuses more on ruby/spec than on CRuby tests for the advantages mentioned above.

Next, we have JRuby passing 88% or 92% of all specs, depending on whether you count C-API specs. That’s impressive as well, notably with high scores on language and core specs. It is worth noting JRuby is currently working on finishing Ruby 2.6 support.

Then we have Opal, which passes a sizable number of language, core and library specs. Opal is compiling to JavaScript, so unlike JRuby and TruffleRuby it does not aim to implement lower-level system functionality. Opal is also typically run in browsers and not on the command line, so Opal does not run command-line specs.

Finally, Artichoke, a fairly new Ruby implementation, passes some core and library specs. Based on MRuby, which implements a subset of Ruby, it still has a long way to achieve compatibility with CRuby. Update (February 2021): Artichoke now passes an estimated 22% of core library specs when trying to run all core specs with this PR, however those are not run in CI and might regress.

Note that specs from ruby/spec are not necessarily representative of general compatibility with the entire Ruby ecosystem, even though they are obviously related. For instance, there might be far more spec examples on a given method than usages in Ruby gems, or the opposite. There are Ruby semantics that are not yet covered in ruby/spec: it is an ongoing effort to improve coverage by the way of contributions to ruby/spec from Ruby implementations and from external contributors. In general, when an alternative implementation has a bug report, they either ensure there is already a spec or add one so over time the ruby/spec coverage improves.

I took a look at CRuby tests for more context and there are in total 20587 CRuby tests when run on CRuby 2.6.6 locally. TruffleRuby currently runs and passes 10467 CRuby tests in CI. JRuby currently runs and passes 8133 CRuby tests in CI. Opal and Artichoke do not seem to run CRuby tests currently. So it seems clear alternative Ruby implementations focus more on ruby/spec, but also that there are still a lot more CRuby tests they could run. Some CRuby tests are however CRuby-specific and not applicable for any other Ruby implementation.

Conclusion

I am looking forward to Ruby implementations improving their compatibility and passing a higher percentage of ruby/spec. I plan to make such a report again when that happens.

Other Ruby Implementations

I would like to report on other Ruby implementations but currently it seems difficult.

Rubinius 5.0 fails to compile on my machine. I have noticed similar issues on various Ubuntu versions too. I used ruby-build and installed the clang llvm-devel packages.
MRuby 2.1.1 (as installed by ruby-install) does not support $: and require and so cannot run the regular MSpec harness. Also, mruby/mruby does not seem to run ruby/spec as part of its CI.

Any help to run ruby/spec on these implementations with the stats formatter or a compatible output is welcome. The stats formatter has a very simple format, it’s a Hash of spec files to number of passing examples as a YAML file:

---
language/some_spec.rb:
  :examples: 10
  :errors: 1
  :failures: 2
  :tagged: 3

Acknowledgments

I used the technique from this blog post to draw the percentage charts.

A Migration Path to Bundler 2+

2020-01-13T00:00:00+01:00

Bundler 2 did not arrive quietly. It was noticed by almost every CI build failing when running bundle install. As a result, it seems many still avoid Bundler 2 and just use Bundler 1. In this post, I present some ideas on how to get more people to use Bundler 2, and no longer need Bundler 1 which will not be maintained forever.

The RubyGems Requirement of Bundler 2

The original release of Bundler 2, which is version 2.0.0, required such a recent RubyGems version that none of the released Ruby versions had a recent enough RubyGems version shipped with them. Bundler 2 also requires Ruby 2.3+, and finally drops support for Ruby 1.8.

This RubyGems version requirement was quickly found as problematic, and as a result Bundler 2.0.1 was released to lower the RubyGems version requirement to RubyGems 2.5. All versions of Ruby supported by Bundler 2 (that is, Ruby 2.3+) ship with RubyGems 2.5 or newer, but the blog post did not make this clear.

This resulted in a lot of confusion. Most repositories ended up explicitly updating RubyGems in CI to make Bundler 2.0.0 work, as mentioned in the original blog post.

To summarize, it is not needed to update RubyGems to use Bundler 2.0.1+.

In fact, I would even argue against updating RubyGems in CI because:

It makes the CI slower.
It makes the CI less reliable by frequently changing the RubyGems version used.
It uses the latest RubyGems instead of the well known version bundled with that Ruby version.

Bundler Version Autoswitching

Bundler 2 came with another feature, which to say the least is very confusing. This feature is actually implemented directly in RubyGems and it unfortunately included a bug, which led to even more confusion.

Supposing you have both Bundler 1.17.2 and Bundler 2.0.2 installed, what should this print?

$ bundle --version

For every other gem, the answer would be the latest version of the gem installed, so in this case 2.0.2.

The actual answer though, due to Bundler version autoswitching is “it depends on many things”. Specifically, if the current directory or any parent directory has a Gemfile.lock file with a BUNDLED WITH section, then that exact version, or the closest version with the same major version (depending on the RubyGems version), will be used. If such a version is not available, it fails with Could not find 'bundler' (<VERSION>) (e.g., when only Bundler 2 is installed with a Gemfile.lock BUNDLED WITH 1.17.2).

Otherwise, if there is no Gemfile.lock in the current directory or any parent directory, bundle behaves like a normal gem and uses the latest version.

As an example, this feature means that as long as there is a Gemfile.lock BUNDLED WITH 1.x, even if Bundler 2 is installed, Bundler 1 would be used, or the error above would be shown.

I think this “feature” is so confusing that it should be considered a bug. It’s counter-intuitive and prevents using Bundler 2 on Gemfile.lock BUNDLED WITH 1.x.

The Migration Problem

The problem I see with all this is I think very few people use Bundler 2, and rather they just keep using Bundler 1. I tried to estimate how often Bundler 2 is used by various means:

I made a poll on Twitter but that got rather few responses, with rather surprising results.
Some usage statistics from RubyGems.org show that Bundler 2 is only used in about 2% of requests to RubyGems.org.
TravisCI seems to currently use Bundler 1 for every Ruby version.
Most Ruby projects I have seen with a Gemfile.lock are BUNDLED WITH 1.x, showing Bundler 1 is used.

However, Bundler 1 seems no longer maintained (which is fair enough, who would want to maintain Ruby 1.8 compatibility?) and likely will no longer receive bug fixes and security patches.

So it is important that Rubyists migrate to Bundler 2, but I think the current situation made it so inconvenient that very few did. To address those concerns, I propose some ideas below to make migration to Bundler 2 easier.

A Better Migration Path for Bundler 2+

Here are my suggestions to make migration to Bundler 2 easier. I think it would be good to integrate them in future Bundler releases, including 2.x releases.

Do not require a RubyGems version newer than what Ruby versions supported by Bundler ship. Requiring to update RubyGems causes many troubles as seen wth Bundler 2.0.0.
Drop the version autoswitching mechanism in RubyGems for Bundler entirely. For older RubyGems versions with autoswitching, do whatever is necessary in Bundler to avoid that behavior, or at least warn.
Every future version of Bundler should support previous versions of Gemfile and Gemfile.lock.
Do not modify the Gemfile.lock if there would be no other change than the BUNDLED WITH version (e.g., for bundle install with an existing up-to-date Gemfile.lock).

This third point about backward compatibility might require extra maintenance work in Bundler. However, I think almost every software of the scale of Bundler needs to do this kind of things, and I would expect keeping a few extra parsers for older formats of Gemfile.lock is not so much work. For instance, even in Bundler 10, I expect installing gems from an already-resolved Gemfile.lock is basically just installing every listed gem with the listed version. Why Gemfile and not just Gemfile.lock? Because Bundler evaluates the Gemfile in all cases in its current design.

I think supporting older Gemfile and Gemfile.lock is the key for more people to adopt newer versions of Bundler. And importantly, this is the only way people can use newer versions of Bundler and stop using unmaintained and old Bundler versions for existing Gemfile.lock files with BUNDLED WITH 1.x (not all Gemfile.lock will be migrated immediately to Bundler 2).

Conclusion

I shared these suggestions with the Bundler team in October 2019 and they generally agreed and already discussed these subjects at RubyKaigi 2019.

Some of the suggestions listed above have already been addressed. For instance, the RubyGems version requirement (#1) was fixed in 2.0.1, one day after the 2.0.0 release.

The latest RubyGems and Bundler 2 can actually already install a Gemfile.lock BUNDLED WITH 1.x (#3), but it unfortunately changes the BUNDLED WITH section for seemingly no reason (#4 not yet). This can be seen on 2.7.0-preview2:

$ chruby 2.7.0-preview2
$ bundle _2.1.0.pre.2_ install
...
Warning: the lockfile is being updated to Bundler 2,
after which you will be unable to return to Bundler 1.
Bundle complete! 2 Gemfile dependencies, 7 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
$ git diff
--- Gemfile.lock
+++ Gemfile.lock
 BUNDLED WITH
-   1.17.2
+   2.1.0.pre.2

That changes is annoying. Suppose I’m contributing to Rails and I’m just doing bundle install (without modifying any Gemfile). If this changes the Gemfile.lock I will have to constantly undo that change, or be careful to not commit it and keep a dirty working tree. If it is committed and merged, it would force every single contributor of Rails (and the CI) to use Bundler 2 once they pull that change. And every such contributor, when doing bundle install on their own projects would also see the BUNDLED WITH version updated, making this a cascading effect. That sounds to me like a recipe for a really not smooth migration forcing everyone to update in a really short time, so that’s why I think Bundler should not touch the Gemfile.lock if there would be no other change than the BUNDLED WITH version (#4).

On the other hand, it seems fine to use BUNDLED WITH 2.x if there is no Gemfile.lock or the dependencies need to be re-resolved due to, e.g., adding a gem, since Bundler 2 might resolve gems differently than Bundler 1.

So #1 and #3 seem mostly done for the latest 2.x release, but #2 and #4 are not yet addressed.

Does that sound like a good migration plan to you?

Would you be willing to help making these changes in Bundler and RubyGems?

The Delegation Challenge of Ruby 2.7

2019-11-10T00:00:00+01:00

Ruby 3.0 will introduce the separation between positional and keyword arguments. The upcoming Ruby 2.7 release aims to introduce warnings for every argument behavior that will change in Ruby 3.0 to ease migration. However, delegation that works for Ruby 2.6, 2.7 and 3 seems a hard problem to solve.

See the migration guide on ruby-lang.org if you want to migrate your code to work on 2.7+, and see Correct Delegation with Ruby 2.6, 2.7 and 3.0 regarding delegation compatible with Ruby 2.6, 2.7 and 3.0.

This blog post is mostly discussing various possibilities to fix delegation in Ruby 2.7. By now it is mostly history, only ruby2_keywords is available in Ruby 2.7.

What will change for arguments in Ruby 3?

From Ruby 2.0 until Ruby 2.6, keyword arguments in method definitions such as def m(kwreq:, kwopt: 42, **kwrest) are basically just syntactic sugar for extracting values from a Hash passed in last position.

This however leads to some issues when keyword arguments are mixed with optional = default or *rest arguments. For instance, should def m(opt = 42, **kwargs); end; m({ kw: 1 }) pass the Hash as keyword arguments (**kwargs) or as the value for the optional argument (opt)? In Ruby 2, the answer is it passes the Hash as keyword arguments. In Ruby 3 however, such a call would pass a Hash as positional, i.e., assign it to opt.

There is no clear answer as to which is better, without a general rule, and the separation between positional and keyword arguments is that rule.

Actually, there already is a similar separation in Ruby for the block argument. A block argument is always passed with &block or as a literal block (meth(*args) { ... }) and is never mixed with other kinds of arguments.

Here are some basics on what passes keyword arguments and what passes positional arguments:

def m(*args, **kwargs)
end

h = { a: 1 }

m(a: 1) # passes kwargs in Ruby 2 and 3
m(**h)  # passes kwargs in Ruby 2 and 3

m({a: 1}) # passes kwargs in Ruby 2 and positional in Ruby 3
m(h)      # passes kwargs in Ruby 2 and positional in Ruby 3

Methods not taking keyword arguments can still be called with the keyword arguments syntax. The reason here is to avoid breaking compatibility too much. If that was not done, such a call would raise an ArgumentError.

def nokwargs(*args)
  args
end

nokwargs(a: 1) # => [{a: 1}] in both Ruby 2 and 3

One important change in Ruby 2.7, is that **empty_hash actually passes nothing:

empty_hash = {}
nokwargs(**empty_hash) # => [] in Ruby 2.7+, [{}] in Ruby 2.6

Confusingly enough, **{} is treated specially by the parser and automatically removed in Ruby 2.6, which means:

nokwargs(**{}) # => [] in Ruby 2 and 3

The rationale here is m(**kwargs), much like m(*args) must pass nothing if it’s empty, that is if there are no arguments to pass. It was essentially a mistake of Ruby 2.6 and before to pass an empty Hash in such a case, but that’s how it is and we cannot change the past. Maybe it was somehow more compatible for some cases.

These changes have been the work of Jeremy Evans (of Sequel fame), Yusuke Endoh (@mame) and others. I have been mostly watching and commenting on those changes.

Now that we’ve got the basics we can look at the Delegation Challenge!

I will not explain in this blog post how to make non-delegation code work on Ruby 3 and how to fix the warnings of Ruby 2.7, that’s the goal of the (upcoming at the time of writing) migration guide on ruby-lang.org, which is mentioned in the Ruby 2.7 release notes.

The Delegation Challenge

So now that we have a proper separation of keyword arguments, how should delegation look like?

Ruby 2-style Delegation

Up until Ruby 2.6, delegation had always been rather simple:

def delegate(*args, &block)
  target(*args, &block)
end

And this was enough for perfect forwarding, i.e., pass whatever arguments are passed to delegate to target, as if target had been called directly.

However this doesn’t quite work in Ruby 3:

def target(*args, **kwargs)
  [args, kwargs]
end

target(1, b: 2) # => [[1], {b: 2}] in Ruby 2 & 3

delegate(1, b: 2) # => [[1], {b: 2}] in Ruby 2, [[1, {b: 2}], {}] in Ruby 3
# Ruby 2.7:
# warning: The last argument is used as the keyword parameter
# warning: for `target' defined here
# => [[1], {b: 2}]

Because delegate does not take keyword arguments, b: 2 is passed as positional. In Ruby 3, it remains as positional and therefore results in different behavior than calling the method directly. In Ruby 2.7, it warns because the behavior changes in Ruby 3.

Ruby 3-style Delegation

So maybe we should use this to delegate instead?

def delegate(*args, **kwargs, &block)
  target(*args, **kwargs, &block)
end

delegate(1, b: 2) # => [[1], {b: 2}] in Ruby 2 & 3
delegate(1)       # => [[1], {}]     in Ruby 2 & 3
delegate(b: 2)    # => [[], {b: 2}]  in Ruby 2 & 3

Alright, this seems to work fine, but what if target does not take keyword arguments?

def target(*args)
  args
end

target()  # => []   in Ruby 2 & 3
target(1) # => [1]  in Ruby 2 & 3

delegate()  # => [{}]    in Ruby 2.6, []  in Ruby 2.7+
delegate(1) # => [1, {}] in Ruby 2.6, [1] in Ruby 2.7+

Now Ruby 2.6 starts to pass an extra positional Hash to target when there are no keyword arguments! That’s because **empty_hash passes {} in Ruby 2.6, and nothing in Ruby 2.7+.

So neither of those delegate methods works on both Ruby 2 and 3, that is very unfortunate.

And to make matters worse Ruby 2.7 has its own behavior different from Ruby 2.6 and Ruby 3:

delegate(b: 2)     # => {b: 2} in Ruby 2 & 3
delegate({ b: 2 }) # => {b: 2} in Ruby 2 & 3
# However in 2.7 emits a warning (which is unjustified, passing a positional Hash to #target is fine):
# warning: The last argument is used as the keyword parameter
# warning: for `delegate' defined here

delegate({}) # => [{}] in Ruby 2.6 & 3, but [] in Ruby 2.7! (with a warning)

For compatibility with 2.6, Ruby 2.7 passes {} as a keyword argument here, but then delegate receives args=[], kwargs={} and call target with **kwargs, which passes nothing in 2.7 and therefore drops the given Hash!

We can actually convince Ruby 2.7 to pass a positional Hash with , **{}:

delegate({}, **{}) # => [{}] in Ruby 2 and 3

So for Ruby 2.7, neither of these approaches even works for all cases (without changing call sites to use , **{}).

So that is the Delegation Challenge: How can we achieve perfect forwarding in Ruby 2.6, Ruby 2.7 and Ruby 3+ and avoid warnings from Ruby 2.7?

Possible Solutions

There are multiple possibilities:

Version check with Ruby 2 + Ruby 3-style delegation

We could use a version check like:

if RUBY_VERSION < "3"
  def delegate(*args, &block)
    target(*args, &block)
  end
else
  def delegate(*args, **kwargs, &block)
    target(*args, **kwargs, &block)
  end
end

This would work fine in Ruby 2.0-2.6 and Ruby 3+. Unfortunately it does not work in Ruby 2.7 which has behavior “in between” Ruby 2.6 and Ruby 3 (**empty_hash passes nothing but positional Hash are still converted to keyword arguments like in 2.6). We’d probably still want to be able to run the code on Ruby 2.7 to get the migration warnings to help migrating to Ruby 3. For that, we would need to address delegation on Ruby 2.7, otherwise there would be a lot of false-positive warnings.

Maybe using Ruby 3-style delegation and adding , **{} at call sites which need it for Ruby 2.7 would be acceptable, I’m not sure.

The … operator

The new ... operator of Ruby 2.7 passes all arguments (positional, keywords, block):

def delegate(...)
  target(...)
end

This works in Ruby 2.7 and 3+, but it’s a SyntaxError in 2.6! So we’d need something like this if we still want to run on 2.6:

all_args = RUBY_VERSION < "2.7" ? "*args, &block" : "..."

class_eval <<RUBY
def delegate(#{all_args})
  target(#{all_args})
end
RUBY

... currently does not allow any sibling argument (e.g., def method_missing(meth, ...)), which makes it unusable in those situations. Probably we should make it work in those situations too, otherwise it’s quite limited. That is something that seems worth arguing for on this ticket.

pass_keywords

pass_keywords is a simple mechanism that basically simulates ... in a syntax-compatible way:

def pass_keywords(*); end unless respond_to?(:pass_keywords, true)

if RUBY_VERSION < "3"
  pass_keywords def delegate(*args, &block)
    target(*args, &block)
  end
else
  def delegate(*args, **kwargs, &block)
    target(*args, **kwargs, &block)
  end
end

It works by forwarding keyword arguments passed to delegate as keyword arguments to target. That only applies for call sites using *args in pass_keywords methods, so it’s simple to understand.

We could also easily make it work for blocks inside that method (the same goes for ...). That’s actually more useful than it sounds, and allows “saving” a delegated call and trigger it later:

pass_keywords def initialize(*args, &block)
  @call_later = -> { target(*args, &block) }
end

def call
  @call_later.call
end

pass_keywords is enough to make the delegation in ActionDispatch::MiddlewareStack::Middleware (a rather complicated case) work and requires rather few changes.

We could also keep pass_keywords in Ruby 3+, although I think it would be preferable to use more idiomatic ways for delegation in Ruby 3 and later (either def delegate(*args, **kwargs, &block) or def delegate(...)).

ruby2_keywords

ruby2_keywords is a method introduced in Ruby 2.7 that alters how a method taking a *rest parameter behaves. Specifically, if keyword arguments are passed to such a method, they are remembered by flagging the Hash. When a flagged Hash is passed at any call site using a *rest argument (and no keyword arguments), then that flagged Hash is converted back to keyword arguments.

def ruby2_keywords(*); end unless respond_to?(:ruby2_keywords, true)
# Or use the 'ruby2_keywords' gem:
require 'ruby2_keywords'

ruby2_keywords def delegate(*args, &block)
  target(*args, &block)
end

delegate({}) # => [{}]

Here is how ActionDispatch::MiddlewareStack::Middleware looks like with ruby2_keywords.

This works fine on Ruby 2.6 (ruby2_keywords is defined as no-op) and 2.7. However, what should happen for Ruby 3? Should we even have a method called ruby2_keywords in Ruby 3?

The current idea seems to keep ruby2_keywords until at least Ruby 2 is end-of-life, let’s say in Ruby 3.X. That means however code using ruby2_keywords will break at that date, unless the code also includes a version check and uses the Ruby 3-style delegation:

require 'ruby2_keywords'

if RUBY_VERSION < "3"
  ruby2_keywords def delegate(*args, &block)
    target(*args, &block)
  end
else
  def delegate(*args, **kwargs, &block)
    target(*args, **kwargs, &block)
  end
end

This is becoming not so pretty, but I think is a good path forward. Using this, code will not break in the foreseeable future, and the ruby2_keywords workaround is only used on Ruby 2.7, which is the only version needing such a complicated workaround. Also, we use the Ruby 3-style delegation in Ruby 3 which is rather natural, and not some workaround with complicated semantics. The Ruby 2 branch can be removed as soon as Ruby 2 support is dropped for that code.

However, it seems most MRI committers think it’s fine to migrate your code to the version just using ruby2_keywords and then ask you again when Ruby 3.X comes out to change the code again to use Ruby 3-style delegation, or .... I’m not comfortable with the idea of telling people to change their code, knowing it will break in Ruby 3.X, and this is part of the reason why I am writing this blog post.

I don’t think keeping ruby2_keywords forever is any good either, I think it doesn’t make sense to have a method named ruby2_keywords in e.g., Ruby 4. We could maybe rename it, but I have other concerns.

I dislike ruby2_keywords because it introduces hidden state (a hidden flag on a Hash). Do you know which other construct uses hidden state in Ruby? The flip-flop operator is an example. Hidden state is almost always a sign of bad design for programming languages, it always comes back and bites you by having surprising semantics and non-trivial performance implications. Practically, it means that ruby2_keywords might flag a Hash and when it’s used in a different file at a call site very far away it could magically be passed as keyword arguments even though it’s syntactically passed as a positional argument (e.g., foo(*args)). Have fun debugging that.

Also, it slows down every single call site using a *rest argument, not just the one in the delegate method. Because of that I consider it a hack, which seems acceptable if only used in Ruby 2.7, but seems like we’d shoot ourselves in the foot to keep it any longer, both for making migration more difficult when removing it, and for suffering lower performance for all foo(*args) calls until then. This issue has more details about performance implications of ruby2_keywords and Ruby 3 keyword arguments.

Finally, ruby2_keywords introduces magic conversion of positional arguments to keyword arguments. Is it not ironic that we work on separating positional and keyword argument but then introduce ruby2_keywords which brings back automatic conversation, i.e., potentially breaks the separation?

So let’s keep thinking.

ruby2_keywords + send_keyword_hash

This is a variant of ruby2_keywords above but it makes it explicit which call site can convert a flagged Hash to keyword arguments. send_keyword_hash is then just send with the additional conversion of a flagged Hash to keyword arguments.

require 'ruby2_keywords'
# Could be defined by the ruby2_keywords gem too
alias_method :send_keyword_hash, :__send__ unless respond_to?(:send_keyword_hash)

if RUBY_VERSION < "3"
  ruby2_keywords def delegate(*args, &block)
    send_keyword_hash(:target, *args, &block)
  end
else
  def delegate(*args, **kwargs, &block)
    target(*args, **kwargs, &block)
  end
end

This still has hidden state by flagging a keyword Hash passed to a ruby2_keywords method, but at least it’s explicit where the conversion can occur and where the flag has any effect. By making it explicit where such a conversion can happen with send_keyword_hash, it’s both clearer for semantics (easier to debug) and it removes the performance concern by letting other call sites using a *rest argument behave unchanged. Also, with send_keyword_hash, there is no more magic conversion of positional to keyword arguments, so the separation remains clean with no backdoor.

Here is how ActionDispatch::MiddlewareStack::Middleware looks like with ruby2_keywords + send_keyword_hash.

What Should We Do?

Here are a few questions for the Ruby community and for the reader:

Do you like any of these approaches?
Is there a better approach that would work but wasn’t considered?
Do you think it’s OK to break code/having to change code again in Ruby 3.X by just recommending to use ruby2_keywords with no version check? It seems to be what MRI committers are thinking currently.
Until when should we keep ruby2_keywords (if at all)?
What do you think of the hidden state, magic conversion at any foo(*args) call site and hard-to-debug concerns about ruby2_keywords?
What delegation style would you use once Ruby 2 support is dropped? def delegate(*args, **kwargs, &block), def delegate(...), pass_keywords?

A New Expectation Syntax for ruby/spec

2019-10-07T00:00:00+02:00

ruby/spec is a test suite for the behavior of the Ruby programming language. The utility to run the test suite is called MSpec and is very similar to RSpec 2.

The reason to have its own runner and not simply using RSpec is that MSpec is significantly simpler than RSpec 2. For example, it does not require any standard library. Early-stage Ruby implementations typically do not support the entire standard library, but based on the simplicity of MSpec they are still able to run language and core library specs with minimal efforts.

MSpec also provides a few features that RSpec does or did not have such as automatically tagging/untagging failing specs, various guards useful for ruby/spec, etc.

Existing Syntax in RSpec 2 and RSpec 3

MSpec uses a syntax very similar to RSpec 2:

describe "String#start_with?" do
  it "returns true only if beginning match" do
    "hello".start_with?('hel').should == true
  end
end

This contrasts to the RSpec 3 style:

RSpec.describe "String#start_with?" do
  it "returns true only if beginning match" do
    expect("hello".start_with?('hel')).to eq true
  end
end

I don’t particularly like the RSpec 3 style:

It’s more verbose and needs two method calls instead of one (not counting the matcher)
It needs extra parentheses which add visual clutter
It’s harder to switch between regular code and specs because some sort of manual code translation is always required (i.e., it’s not just adding .should)

These reasons are why I am mostly happy with the existing RSpec 2 style in ruby/spec.

Using expect has of course advantages too, notably when using delegation, as explained in the original blog post about expect.

The Various Equality Methods

One aspect discussed in that blog post is that should == generates Ruby warnings. This used to indeed be annoying. However in Ruby 2.4+ it’s easy to automatically filter out these warnings by overriding Warning.warn, which MSpec does.

When I compare actual.should eq(expected) to actual.should == expected, I strongly prefer the latter, because it is much closer to normal Ruby code which would just test actual == expected. There are also various equality methods in Ruby: ==, eql? and equal?. I’d like to not have to remember any kind of mapping between regular code and specs; I’d like to use the exact same method names in specs. Continuing on that idea, I’m not such a huge fan of having so many matchers in MSpec to do very basic tests I could do more simply in Ruby code. Here are a few matchers from MSpec: ==, eql, equal, =~, <=, include, be_true, be_false, be_empty, be_nan, be_an_instance_of, be_kind_of, be_ancestor_of, respond_to, have_instance_method, …

So many already, and there are many more! And of course, there will always be some missing matchers, for instance there is no matcher for String#start_with?. Contrary to RSpec, there are no magic be_.../have_... matchers in MSpec.

The New Simple and Consistent Syntax

Could we do without having to remember all these mappings and just use normal predicates in Ruby code? That’s what I wanted to try with this new syntax in MSpec:

describe "String#start_with?" do
  it "returns true only if beginning match" do
    "hello".should.start_with?('hel')
  end
end

The idea is: add .should just before the predicate you expect to be truthy, and done! And this works for any method, because we can implement this through #method_missing. This syntax is not new, it was actually already used for should ==, but now it can be used for any predicate:

(1 + 2).should == 3
(1 + 2).should_not == 4
(1 + 2).should != 4
1.should < 2
"abc".should =~ /b/
4.should.eql? 4.0
self.should.equal?(self)
"end".should.end_with?("d")
[].should.empty?
(1..3).should.include?(2)
File.should.exist?("README.txt")
(1..5).step(2).should.all? { |e| e.odd? }

I think some of these look really nice, some do not read as nicely, but overall they are all consistent and extremely simple to learn.

The simplicity of removing .should to go from specs to regular code is very useful for debugging purposes. Having a minimal amount of self-contained code to reproduce an issue is often key to understand and fix a bug.

A significant advantage of this approach is that all matchers have a similar output for errors, and provide a lot of information if they fail.

With the old syntax (s.start_with?('b').should == true), the error wouldn’t be very helpful and one would need to look at the test code to get an idea of what went wrong:

String#start_with? returns true only if beginning match FAILED
Expected false to equal true

With the new syntax (s.should.start_with?('b')), we get to see the receiver and arguments, no matter what predicate is used, and it’s immediately clear which expectation failed:

String#start_with? returns true only if beginning match FAILED
Expected "a".start_with? "b"
to be truthy but was false

And it’s all consistent, so once we learn the formatting of just one error, we know it for all other errors.

Expected 1 == 2
to be truthy but was false

Expected [1, 3].include? 2
to be truthy but was false

Previously, the message and formatting would differ for every matcher:

Expected 1 to equal 2

Expected [1, 3] to include 2

I no longer need to reverse map equal to == (how confusing!) and include to include?, it just shows me the method which was called and returned something unexpected.

Expected Actual to Equal Expected

Moreover actual and expected are often unclear in many test framework outputs, such as Expected ACTUAL to equal EXPECTED above. Some test-unit frameworks even disagree on which should be first for assert_equal. We no longer need a notion of actual and expected values in the new syntax, we always expect a whole predicate to be truthy or falsy, and that’s it. There is no question of order, it’s plain Ruby code and we can see == was called on the left receiver value, with the right value(s) as the argument(s).

Of course, this new syntax is not the answer to everything. For instance, it does not address the fact we need a different syntax for exceptions (and mocks). We could extend the new syntax to deal with exceptions as going through method_missing gives a lot of control, but I did not find a nice syntax for that yet.

Conclusion

The existing RSpec 2-like syntax will still remain available in MSpec, for compatibility and because migrating ruby/spec would be a very large amount of work. I also don’t want to force anyone to use the new syntax. I started using the new syntax in a few specs where I found it to be a clear gain, such as for start_with? specs. The most frequent matcher, should ==, already uses the new error output, because that matcher uses the same syntax in the new syntax and therefore the new implementation is used. As a fun fact, the implementation of the new syntax actually removes more lines than it adds.

In this blog post I presented a new expectation syntax for ruby/spec. The main advantages are:

Consistent and simple (“add .should before the predicate”) and therefore easy to learn
Very similar to regular code, so going from plain Ruby code to specs or back is trivial
No extra mapping between predicates and matchers, matchers are just Ruby predicates
Clear output when an expectation fails, showing the relevant values that caused the error

What do you think? Is this new syntax a good or a bad idea?