Written by Benoit Daloze @eregontp
on 

The Delegation Challenge of Ruby 2.7

Ruby 3.0 will introduce the separation between positional and keyword arguments. The upcoming Ruby 2.7 release aims to introduce warnings for every argument behavior that will change in Ruby 3.0 to ease migration. However, delegation that works for Ruby 2.6, 2.7 and 3 seems a hard problem to solve.

See the migration guide on ruby-lang.org if you want to migrate your code to work on 2.7+, and see Correct Delegation with Ruby 2.6, 2.7 and 3.0 regarding delegation compatible with Ruby 2.6, 2.7 and 3.0.

This blog post is mostly discussing various possibilities to fix delegation in Ruby 2.7. By now it is mostly history, only ruby2_keywords is available in Ruby 2.7.

What will change for arguments in Ruby 3?

From Ruby 2.0 until Ruby 2.6, keyword arguments in method definitions such as def m(kwreq:, kwopt: 42, **kwrest) are basically just syntactic sugar for extracting values from a Hash passed in last position.

This however leads to some issues when keyword arguments are mixed with optional = default or *rest arguments. For instance, should def m(opt = 42, **kwargs); end; m({ kw: 1 }) pass the Hash as keyword arguments (**kwargs) or as the value for the optional argument (opt)? In Ruby 2, the answer is it passes the Hash as keyword arguments. In Ruby 3 however, such a call would pass a Hash as positional, i.e., assign it to opt.

There is no clear answer as to which is better, without a general rule, and the separation between positional and keyword arguments is that rule.

Actually, there already is a similar separation in Ruby for the block argument. A block argument is always passed with &block or as a literal block (meth(*args) { ... }) and is never mixed with other kinds of arguments.

Here are some basics on what passes keyword arguments and what passes positional arguments:

def m(*args, **kwargs)
end

h = { a: 1 }

m(a: 1) # passes kwargs in Ruby 2 and 3
m(**h)  # passes kwargs in Ruby 2 and 3

m({a: 1}) # passes kwargs in Ruby 2 and positional in Ruby 3
m(h)      # passes kwargs in Ruby 2 and positional in Ruby 3

Methods not taking keyword arguments can still be called with the keyword arguments syntax. The reason here is to avoid breaking compatibility too much. If that was not done, such a call would raise an ArgumentError.

def nokwargs(*args)
  args
end

nokwargs(a: 1) # => [{a: 1}] in both Ruby 2 and 3

One important change in Ruby 2.7, is that **empty_hash actually passes nothing:

empty_hash = {}
nokwargs(**empty_hash) # => [] in Ruby 2.7+, [{}] in Ruby 2.6

Confusingly enough, **{} is treated specially by the parser and automatically removed in Ruby 2.6, which means:

nokwargs(**{}) # => [] in Ruby 2 and 3

The rationale here is m(**kwargs), much like m(*args) must pass nothing if it’s empty, that is if there are no arguments to pass. It was essentially a mistake of Ruby 2.6 and before to pass an empty Hash in such a case, but that’s how it is and we cannot change the past. Maybe it was somehow more compatible for some cases.

These changes have been the work of Jeremy Evans (of Sequel fame), Yusuke Endoh (@mame) and others. I have been mostly watching and commenting on those changes.

Now that we’ve got the basics we can look at the Delegation Challenge!

I will not explain in this blog post how to make non-delegation code work on Ruby 3 and how to fix the warnings of Ruby 2.7, that’s the goal of the (upcoming at the time of writing) migration guide on ruby-lang.org, which is mentioned in the Ruby 2.7 release notes.

The Delegation Challenge

So now that we have a proper separation of keyword arguments, how should delegation look like?

Ruby 2-style Delegation

Up until Ruby 2.6, delegation had always been rather simple:

def delegate(*args, &block)
  target(*args, &block)
end

And this was enough for perfect forwarding, i.e., pass whatever arguments are passed to delegate to target, as if target had been called directly.

However this doesn’t quite work in Ruby 3:

def target(*args, **kwargs)
  [args, kwargs]
end

target(1, b: 2) # => [[1], {b: 2}] in Ruby 2 & 3

delegate(1, b: 2) # => [[1], {b: 2}] in Ruby 2, [[1, {b: 2}], {}] in Ruby 3
# Ruby 2.7:
# warning: The last argument is used as the keyword parameter
# warning: for `target' defined here
# => [[1], {b: 2}]

Because delegate does not take keyword arguments, b: 2 is passed as positional. In Ruby 3, it remains as positional and therefore results in different behavior than calling the method directly. In Ruby 2.7, it warns because the behavior changes in Ruby 3.

Ruby 3-style Delegation

So maybe we should use this to delegate instead?

def delegate(*args, **kwargs, &block)
  target(*args, **kwargs, &block)
end

delegate(1, b: 2) # => [[1], {b: 2}] in Ruby 2 & 3
delegate(1)       # => [[1], {}]     in Ruby 2 & 3
delegate(b: 2)    # => [[], {b: 2}]  in Ruby 2 & 3

Alright, this seems to work fine, but what if target does not take keyword arguments?

def target(*args)
  args
end

target()  # => []   in Ruby 2 & 3
target(1) # => [1]  in Ruby 2 & 3

delegate()  # => [{}]    in Ruby 2.6, []  in Ruby 2.7+
delegate(1) # => [1, {}] in Ruby 2.6, [1] in Ruby 2.7+

Now Ruby 2.6 starts to pass an extra positional Hash to target when there are no keyword arguments! That’s because **empty_hash passes {} in Ruby 2.6, and nothing in Ruby 2.7+.

So neither of those delegate methods works on both Ruby 2 and 3, that is very unfortunate.

And to make matters worse Ruby 2.7 has its own behavior different from Ruby 2.6 and Ruby 3:

delegate(b: 2)     # => {b: 2} in Ruby 2 & 3
delegate({ b: 2 }) # => {b: 2} in Ruby 2 & 3
# However in 2.7 emits a warning (which is unjustified, passing a positional Hash to #target is fine):
# warning: The last argument is used as the keyword parameter
# warning: for `delegate' defined here

delegate({}) # => [{}] in Ruby 2.6 & 3, but [] in Ruby 2.7! (with a warning)

For compatibility with 2.6, Ruby 2.7 passes {} as a keyword argument here, but then delegate receives args=[], kwargs={} and call target with **kwargs, which passes nothing in 2.7 and therefore drops the given Hash!

We can actually convince Ruby 2.7 to pass a positional Hash with , **{}:

delegate({}, **{}) # => [{}] in Ruby 2 and 3

So for Ruby 2.7, neither of these approaches even works for all cases (without changing call sites to use , **{}).

So that is the Delegation Challenge: How can we achieve perfect forwarding in Ruby 2.6, Ruby 2.7 and Ruby 3+ and avoid warnings from Ruby 2.7?

Possible Solutions

There are multiple possibilities:

Version check with Ruby 2 + Ruby 3-style delegation

We could use a version check like:

if RUBY_VERSION < "3"
  def delegate(*args, &block)
    target(*args, &block)
  end
else
  def delegate(*args, **kwargs, &block)
    target(*args, **kwargs, &block)
  end
end

This would work fine in Ruby 2.0-2.6 and Ruby 3+. Unfortunately it does not work in Ruby 2.7 which has behavior “in between” Ruby 2.6 and Ruby 3 (**empty_hash passes nothing but positional Hash are still converted to keyword arguments like in 2.6). We’d probably still want to be able to run the code on Ruby 2.7 to get the migration warnings to help migrating to Ruby 3. For that, we would need to address delegation on Ruby 2.7, otherwise there would be a lot of false-positive warnings.

Maybe using Ruby 3-style delegation and adding , **{} at call sites which need it for Ruby 2.7 would be acceptable, I’m not sure.

The … operator

The new ... operator of Ruby 2.7 passes all arguments (positional, keywords, block):

def delegate(...)
  target(...)
end

This works in Ruby 2.7 and 3+, but it’s a SyntaxError in 2.6! So we’d need something like this if we still want to run on 2.6:

all_args = RUBY_VERSION < "2.7" ? "*args, &block" : "..."

class_eval <<RUBY
def delegate(#{all_args})
  target(#{all_args})
end
RUBY

... currently does not allow any sibling argument (e.g., def method_missing(meth, ...)), which makes it unusable in those situations. Probably we should make it work in those situations too, otherwise it’s quite limited. That is something that seems worth arguing for on this ticket.

pass_keywords

pass_keywords is a simple mechanism that basically simulates ... in a syntax-compatible way:

def pass_keywords(*); end unless respond_to?(:pass_keywords, true)

if RUBY_VERSION < "3"
  pass_keywords def delegate(*args, &block)
    target(*args, &block)
  end
else
  def delegate(*args, **kwargs, &block)
    target(*args, **kwargs, &block)
  end
end

It works by forwarding keyword arguments passed to delegate as keyword arguments to target. That only applies for call sites using *args in pass_keywords methods, so it’s simple to understand.

We could also easily make it work for blocks inside that method (the same goes for ...). That’s actually more useful than it sounds, and allows “saving” a delegated call and trigger it later:

pass_keywords def initialize(*args, &block)
  @call_later = -> { target(*args, &block) }
end

def call
  @call_later.call
end

pass_keywords is enough to make the delegation in ActionDispatch::MiddlewareStack::Middleware (a rather complicated case) work and requires rather few changes.

We could also keep pass_keywords in Ruby 3+, although I think it would be preferable to use more idiomatic ways for delegation in Ruby 3 and later (either def delegate(*args, **kwargs, &block) or def delegate(...)).

ruby2_keywords

ruby2_keywords is a method introduced in Ruby 2.7 that alters how a method taking a *rest parameter behaves. Specifically, if keyword arguments are passed to such a method, they are remembered by flagging the Hash. When a flagged Hash is passed at any call site using a *rest argument (and no keyword arguments), then that flagged Hash is converted back to keyword arguments.

def ruby2_keywords(*); end unless respond_to?(:ruby2_keywords, true)
# Or use the 'ruby2_keywords' gem:
require 'ruby2_keywords'

ruby2_keywords def delegate(*args, &block)
  target(*args, &block)
end

delegate({}) # => [{}]

Here is how ActionDispatch::MiddlewareStack::Middleware looks like with ruby2_keywords.

This works fine on Ruby 2.6 (ruby2_keywords is defined as no-op) and 2.7. However, what should happen for Ruby 3? Should we even have a method called ruby2_keywords in Ruby 3?

The current idea seems to keep ruby2_keywords until at least Ruby 2 is end-of-life, let’s say in Ruby 3.X. That means however code using ruby2_keywords will break at that date, unless the code also includes a version check and uses the Ruby 3-style delegation:

require 'ruby2_keywords'

if RUBY_VERSION < "3"
  ruby2_keywords def delegate(*args, &block)
    target(*args, &block)
  end
else
  def delegate(*args, **kwargs, &block)
    target(*args, **kwargs, &block)
  end
end

This is becoming not so pretty, but I think is a good path forward. Using this, code will not break in the foreseeable future, and the ruby2_keywords workaround is only used on Ruby 2.7, which is the only version needing such a complicated workaround. Also, we use the Ruby 3-style delegation in Ruby 3 which is rather natural, and not some workaround with complicated semantics. The Ruby 2 branch can be removed as soon as Ruby 2 support is dropped for that code.

However, it seems most MRI committers think it’s fine to migrate your code to the version just using ruby2_keywords and then ask you again when Ruby 3.X comes out to change the code again to use Ruby 3-style delegation, or .... I’m not comfortable with the idea of telling people to change their code, knowing it will break in Ruby 3.X, and this is part of the reason why I am writing this blog post.

I don’t think keeping ruby2_keywords forever is any good either, I think it doesn’t make sense to have a method named ruby2_keywords in e.g., Ruby 4. We could maybe rename it, but I have other concerns.

I dislike ruby2_keywords because it introduces hidden state (a hidden flag on a Hash). Do you know which other construct uses hidden state in Ruby? The flip-flop operator is an example. Hidden state is almost always a sign of bad design for programming languages, it always comes back and bites you by having surprising semantics and non-trivial performance implications. Practically, it means that ruby2_keywords might flag a Hash and when it’s used in a different file at a call site very far away it could magically be passed as keyword arguments even though it’s syntactically passed as a positional argument (e.g., foo(*args)). Have fun debugging that.

Also, it slows down every single call site using a *rest argument, not just the one in the delegate method. Because of that I consider it a hack, which seems acceptable if only used in Ruby 2.7, but seems like we’d shoot ourselves in the foot to keep it any longer, both for making migration more difficult when removing it, and for suffering lower performance for all foo(*args) calls until then. This issue has more details about performance implications of ruby2_keywords and Ruby 3 keyword arguments.

Finally, ruby2_keywords introduces magic conversion of positional arguments to keyword arguments. Is it not ironic that we work on separating positional and keyword argument but then introduce ruby2_keywords which brings back automatic conversation, i.e., potentially breaks the separation?

So let’s keep thinking.

ruby2_keywords + send_keyword_hash

This is a variant of ruby2_keywords above but it makes it explicit which call site can convert a flagged Hash to keyword arguments. send_keyword_hash is then just send with the additional conversion of a flagged Hash to keyword arguments.

require 'ruby2_keywords'
# Could be defined by the ruby2_keywords gem too
alias_method :send_keyword_hash, :__send__ unless respond_to?(:send_keyword_hash)

if RUBY_VERSION < "3"
  ruby2_keywords def delegate(*args, &block)
    send_keyword_hash(:target, *args, &block)
  end
else
  def delegate(*args, **kwargs, &block)
    target(*args, **kwargs, &block)
  end
end

This still has hidden state by flagging a keyword Hash passed to a ruby2_keywords method, but at least it’s explicit where the conversion can occur and where the flag has any effect. By making it explicit where such a conversion can happen with send_keyword_hash, it’s both clearer for semantics (easier to debug) and it removes the performance concern by letting other call sites using a *rest argument behave unchanged. Also, with send_keyword_hash, there is no more magic conversion of positional to keyword arguments, so the separation remains clean with no backdoor.

Here is how ActionDispatch::MiddlewareStack::Middleware looks like with ruby2_keywords + send_keyword_hash.

What Should We Do?

Here are a few questions for the Ruby community and for the reader: