feat(spec/java): add strip flag in meta string encoding spec

[chaokunyang]

What does this PR do?

add strip flag in meta string encoding spec

Related issues

#1540

Does this PR introduce any user-facing change?

• Does this PR introduce any public API change?
• Does this PR introduce any binary protocol compatibility change?

Benchmark

[chaokunyang]

CI failure seems related to actions/setup-python#850

[theweipeng]

LGTM

[qingoba]

In MetaStringEncoder.java, we use 0x08:

boolean stripLastChar = bytes.length * 8 >= totalBits + bitsPerChar;
if (stripLastChar) {
bytes[0] = (byte) (bytes[0] | 0x80);
}

In MetaString.java, we use 0b1:

if (encoding != Encoding.UTF_8) {
  Preconditions.checkArgument(bytes.length > 0);
  this.stripLastChar = (bytes[0] & 0b1) != 0;
} else {
  this.stripLastChar = false;
}

These two are not consistent.

[LiangliangSui]

In MetaStringEncoder.java, we use 0x08:

boolean stripLastChar = bytes.length * 8 >= totalBits + bitsPerChar;
if (stripLastChar) {
bytes[0] = (byte) (bytes[0] | 0x80);
}

In MetaString.java, we use 0b1:

if (encoding != Encoding.UTF_8) {
  Preconditions.checkArgument(bytes.length > 0);
  this.stripLastChar = (bytes[0] & 0b1) != 0;
} else {
  this.stripLastChar = false;
}

These two are not consistent.

Thanks for pointing out this issue, this is indeed a bug and our unit tests don't cover it.

0x80 should also be used in MetaString.

if (encoding != Encoding.UTF_8) {
  Preconditions.checkArgument(bytes.length > 0);
  this.stripLastChar = (bytes[0] & 0x80) != 0;
} else {
  this.stripLastChar = false;
}

Do you want to submit a PR to fix this issue and add unit tests? @qingoba

[qingoba]

Sure, I will do this after I finish #1566

[chaokunyang]

In MetaStringEncoder.java, we use 0x08:

boolean stripLastChar = bytes.length * 8 >= totalBits + bitsPerChar;
if (stripLastChar) {
bytes[0] = (byte) (bytes[0] | 0x80);
}

In MetaString.java, we use 0b1:

if (encoding != Encoding.UTF_8) {
  Preconditions.checkArgument(bytes.length > 0);
  this.stripLastChar = (bytes[0] & 0b1) != 0;
} else {
  this.stripLastChar = false;
}

These two are not consistent.

The MetaStringDecoder parse the flag by itself, and didn't use this flag. Currently the org.apache.fury.meta.MetaString#stripLastChar is not called by other classes. So the fury serializaiton ut still succceed.

[qingoba]

There is another question.
If input string is empty, encoded bytes is also empty:

public MetaString encode(String input, Encoding encoding) {
  Preconditions.checkArgument(
      input.length() < Short.MAX_VALUE, "Long meta string than 32767 is not allowed");
  if (input.isEmpty()) {
    return new MetaString(input, Encoding.UTF_8, specialChar1, specialChar2, new byte[0]);
  }
}

However, decode procedure doesn't check empty. It will access bytes[0] directly:

private String decodeLowerSpecial(byte[] data) {
  StringBuilder decoded = new StringBuilder();
  int totalBits = data.length * 8; // Total number of bits in the data
  boolean stripLastChar = (data[0] & 0x80) != 0; // Check the first bit of the first byte
}

So decode procedure should check empty.

[chaokunyang]

There is another question. If input string is empty, encoded bytes is also empty:

public MetaString encode(String input, Encoding encoding) {
  Preconditions.checkArgument(
      input.length() < Short.MAX_VALUE, "Long meta string than 32767 is not allowed");
  if (input.isEmpty()) {
    return new MetaString(input, Encoding.UTF_8, specialChar1, specialChar2, new byte[0]);
  }
}

However, decode procedure doesn't check empty. It will access bytes[0] directly:

private String decodeLowerSpecial(byte[] data) {
  StringBuilder decoded = new StringBuilder();
  int totalBits = data.length * 8; // Total number of bits in the data
  boolean stripLastChar = (data[0] & 0x80) != 0; // Check the first bit of the first byte
}

So decode procedure should check empty.

Hi @qingoba , do you have time to fix this today? We plan to start release fury v0.5.0 today and include this fix. If you don't have time, I can take over it and submit a fix later

[qingoba]

I can fix this now.