Written by Benoit Daloze @eregontp
on October 27, 2024

Benchmarking Ruby Parsers

The new Prism parser has become the default in Ruby 3.4.0 preview 2.

Let’s benchmark Ruby parsers and find out how fast they are.

We run benchmarks on Ruby 3.4.0 preview 2 with YJIT (ruby 3.4.0preview2 (2024-10-07 master 32c733f57b) +YJIT +PRISM [x86_64-linux]), on an AMD Ryzen 7 3700X 8-Core Processor and a NVMe M.2 SSD, on Linux, with frequency scaling disabled and the performance CPU governor.

We will compare:

prism 1.2.0
parser 3.3.5.0
ruby_parser 3.21.1
Ripper from ruby 3.4.0preview2
RubyVM::AbstractSyntaxTree from ruby 3.4.0preview2

Which are the latest releases at the time of writing.

Our corpus will be all .rb files in railties 7.2.1.2, that is 151 files, consisting of 14625 lines and 455673 bytes in total.

Let me start with: benchmarking parsers fairly is hard.

All these parsers return different ASTs. Some have more information than others and might be more or less practical to use, but they are all ASTs representing Ruby code.

One major concern performance-wise is that some parsers might allocate Ruby objects lazily and instead only keep some C structs around. This is the case of RubyVM::AbstractSyntaxTree, as can be seen by looking at allocations:

Prism::Node allocations for Prism.parse
Parser::AST::Node allocations for Parser gem
Sexp allocations for RubyParser
230854 Array allocations for Ripper.sexp
RubyVM::AbstractSyntaxTree::Node allocations for RubyVM::AbstractSyntaxTree

36835 nodes for Prism is consistent with the 37762 nodes for the Parser gem. 79749 for RubyParser seems a bit much but still the same order of magnitude. 230854 for Ripper looks to be an outlier, but one inaccuracy here is Ripper uses Arrays not only for nodes but also for positions like [line, column] and Arrays might be used for other things too. RubyVM::AbstractSyntaxTree only allocates 151 nodes (one per file), which proves it creates Ruby node objects lazily.

One way to avoid that laziness is to walk all nodes in the AST. Doing that, we get:

Prism::Node allocations for Prism.parse
Prism::Node allocations for Prism.parse + walk
Parser::AST::Node allocations for Parser gem
Parser::AST::Node allocations for Parser gem + walk
Sexp allocations for RubyParser
Sexp allocations for RubyParser + walk
230854 Array allocations for Ripper.sexp
230854 Array allocations for Ripper.sexp + walk
RubyVM::AbstractSyntaxTree::Node allocations for RubyVM::AbstractSyntaxTree
RubyVM::AbstractSyntaxTree::Node allocations for RubyVM::AbstractSyntaxTree + walk

So RubyVM::AbstractSyntaxTree needs special care to be benchmarked fairly.

While we are at it, here are the total number of Ruby-level allocations:

  75610 total allocations for Prism.parse
 112445 total allocations for Prism.parse + walk
  diff: +36835 Arrays

 823320 total allocations for Parser gem
 823320 total allocations for Parser gem + walk

1028578 total allocations for RubyParser
1028578 total allocations for RubyParser + walk

 341687 total allocations for Ripper.sexp
 341687 total allocations for Ripper.sexp + walk

  16044 total allocations for RubyVM::AbstractSyntaxTree
  92495 total allocations for RubyVM::AbstractSyntaxTree + walk
  diff: +38568 Arrays, +36111 RubyVM::AbstractSyntaxTree::Node, +1689 Strings, +83 Regexps

We can see walking the AST:

causes extra Array allocations for Prism because each call to child_nodes allocates an Array.
does not allocate for the Parser gem because it keeps child nodes in an Array internally and #children just returns that.
does not allocate for RubyParser and Ripper because their representation is just Arrays (or Sexp, a subclass of Array).
allocates Arrays and nodes for RubyVM::AbstractSyntaxTree.

Parsing and Walking

So to try to benchmark RubyVM::AbstractSyntaxTree fairly we can walk the AST for all parsers:

Calculating -------------------------------------
               Prism.parse + walk     27.087 (± 0.0%) i/s   (36.92 ms/i) -    136.000 in   5.021906s
                Parser gem + walk      2.246 (± 0.0%) i/s  (445.33 ms/i) -     12.000 in   5.343971s
                RubyParser + walk      1.632 (± 0.0%) i/s  (612.90 ms/i) -      9.000 in   5.516460s
               Ripper.sexp + walk     11.265 (± 0.0%) i/s   (88.77 ms/i) -     57.000 in   5.065812s
RubyVM::AbstractSyntaxTree + walk     24.272 (± 4.1%) i/s   (41.20 ms/i) -    122.000 in   5.031333s

Comparison:
               Prism.parse + walk:       27.1 i/s
RubyVM::AbstractSyntaxTree + walk:       24.3 i/s - 1.12x  slower
               Ripper.sexp + walk:       11.3 i/s - 2.40x  slower
                Parser gem + walk:        2.2 i/s - 12.06x  slower
                RubyParser + walk:        1.6 i/s - 16.60x  slower

However, this gives an advantage to Ripper, the Parser gem, and RubyParser as they don’t need extra Array allocations to walk the AST. Despite that advantage, all three are slower than Prism.

Prism and RubyVM::AbstractSyntaxTree are more than 10 times as fast as the Parser gem and RubyParser in this benchmark. That’s quite the speedup.

Parsing and not Walking

Or we can exclude RubyVM::AbstractSyntaxTree and not walk the AST:

Calculating -------------------------------------
         Prism.parse     32.590 (± 0.0%) i/s   (30.68 ms/i) -    165.000 in   5.063030s
          Parser gem      2.256 (± 0.0%) i/s  (443.30 ms/i) -     12.000 in   5.319708s
          RubyParser      1.638 (± 0.0%) i/s  (610.58 ms/i) -      9.000 in   5.495618s
         Ripper.sexp     11.834 (± 0.0%) i/s   (84.51 ms/i) -     60.000 in   5.077234s

Comparison:
Prism.parse:       32.6 i/s
Ripper.sexp:       11.8 i/s - 2.75x  slower
 Parser gem:        2.3 i/s - 14.45x  slower
 RubyParser:        1.6 i/s - 19.90x  slower

Just Parsing

We can also find a way to measure just the parsing itself and the creation of C data structures to hold the parse result. In this configuration the parser will allocate as few Ruby objects as possible. This approach is feasible for Prism and for RubyVM::AbstractSyntaxTree. Prism.profile parses to C structs and does nothing else. RubyVM::AbstractSyntaxTree.parse uses parse.y, the LALR-based parser used in CRuby 3.3 and before. RubyVM::AbstractSyntaxTree.parse does very little extra work besides parsing to C structs, which I double-checked by measuring the time around yyparse() (the entry point for parse.y) for a large file. About 99.8% of the time is spent inside yyparse(), so RubyVM::AbstractSyntaxTree.parse is a good approximation for yyparse()/parse.y.

151 files, 14625 lines, 455673 bytes

0 total allocations for Prism.profile

16095 total allocations for RubyVM::AbstractSyntaxTree:
  15559 Strings, 302 Datas, 151 RubyVM::AbstractSyntaxTree::Node, 83 Regexps

Calculating -------------------------------------
             Prism.profile    100.447 (± 2.0%) i/s    (9.96 ms/i) -    510.000 in   5.079433s
RubyVM::AbstractSyntaxTree     39.186 (± 2.6%) i/s   (25.52 ms/i) -    198.000 in   5.056720s

Comparison:
             Prism.profile:      100.4 i/s
RubyVM::AbstractSyntaxTree:       39.2 i/s - 2.56x  slower

Prism parses this corpus of 455673 bytes in 9.96 ms, which is 43.6 MB/s, compared to 17.0 MB/s for parse.y. In other words, Prism is 2.56 times as fast as parse.y in parsing this corpus of Ruby code!

There are 16095 allocations for RubyVM::AbstractSyntaxTree, most of which are Strings. Of those Strings, most are the one String per line (14625 lines) that the lexer used by yyparse() allocates (in rb_parser_lex_get_str()). We see 151 RubyVM::AbstractSyntaxTree::Node: one per file, as expected.

From Source To CRuby Bytecode

We have one more benchmark to measure compilation from source code to CRuby bytecode using RubyVM::InstructionSequence.compile_prism and RubyVM::InstructionSequence.compile_parsey (for parse.y). In this benchmark the input and output are exactly the same for both variants so it is as fair as it gets.

151 files, 14625 lines, 455673 bytes

10477 total allocations for RubyVM::InstructionSequence.compile_prism:
  3753 Strings, 3389 Arrays, 2717 Imemos, 378 Hashs, 151 RubyVM::InstructionSequence, 83 Regexps, 6 Structs

28782 total allocations for RubyVM::InstructionSequence.compile_parsey, including 151 RubyVM::InstructionSequence:
  18476 Strings, 4237 Arrays, 2868 Imemos, 2501 Datas, 377 Hashs, 166 Regexps, 151 RubyVM::InstructionSequence, 6 Structs

Calculating -------------------------------------
 RubyVM::InstructionSequence.compile_prism     36.499 (± 0.0%) i/s   (27.40 ms/i) -    183.000 in   5.014395s
RubyVM::InstructionSequence.compile_parsey     25.050 (± 0.0%) i/s   (39.92 ms/i) -    126.000 in   5.030555s

Comparison:
 RubyVM::InstructionSequence.compile_prism:       36.5 i/s
RubyVM::InstructionSequence.compile_parsey:       25.1 i/s - 1.46x  slower

That’s a nice 1.46x speedup for Prism here. This improvement makes Ruby programs and applications boot faster on CRuby, by reducing the amount of time spent parsing and compiling to bytecode. Prism is also used in other Ruby implementations and has sped up parsing and boot times there as well.

Script

Here is the script I used to benchmark and get allocation counts.

Conclusion

Benchmarking Ruby parsers is trickier than one might expect, but overall the result is clear: Prism is the fastest Ruby parser.

When parsing and walking, Prism is 12% faster than RubyVM::AbstractSyntaxTree and 12x as fast as the Parser gem!
When parsing to C structs, Prism is 2.56x as fast as parse.y. Who thought parsing Ruby in C could be made more than twice as fast?
When compiling to bytecode, Prism is 1.46x as fast as parse.y, speeding up boot times.

Beyond raw performance, Prism also provides a great and well documented API, which makes it much easier to build fast tooling for and in Ruby.

→ Top