fix(ruby): Fix crash when calculating Message hash values on 64-bit Windows

[dazuma]

Fixes #8564

On 64-bit Windows, it is unsafe to try to create a Fixnum from a 64-bit integer, because longs can hold only 32 bits. The code change below:

• Downcasts the original unsigned 64-bit hash to a signed long (which may be either 64 bits or 32 bits depending on the platform).
• Checks the value to see if it's in the half of the range of long that corresponds to a valid Fixnum, and moves it into range if not.
• Only then does it pass the result into INT2FIX.

/attn @haberman

[dazuma]

Updated with a uniform mapping.

[haberman]

What about:

  // RUBY_FIXNUM_MAX should be one less than a power of 2.
  assert(RUBY_FIXNUM_MAX & (RUBY_FIXNUM_MAX + 1) == 0);  
  return INT2FIX(hash_value & RUBY_FIXNUM_MAX);

[dazuma]

Yeah, that works. It restricts hash values to positive, but I think 32-bit hashes are always positive for most Ruby objects anyway.

[dazuma]

Looking at the failed Ruby 2.4 test, I think we can address it by pinning the irb gem to ~> 1.1 on Ruby 2.4 (since newer versions of irb depend on reline, which requires Ruby 2.5+).

[haberman]

Sorry, this is my bad, I always forget that the precedence of & is low. It should be:

assert((RUBY_FIXNUM_MAX & (RUBY_FIXNUM_MAX + 1)) == 0);

[dazuma]

Whoops, I forgot that too. Fixed.

[haberman]

I think we can address it by pinning the irb gem to ~> 1.1 on Ruby 2.4 (since newer versions of irb depend on reline, which requires Ruby 2.5+).

SGTM.

[dazuma]

Looking at the failed Ruby 2.4 test, I think we can address it by pinning the irb gem to ~> 1.1 on Ruby 2.4 (since newer versions of irb depend on reline, which requires Ruby 2.5+).

I added this change to the Gemfile. We can try rerunning the tests to make sure it works.