[Merged by Bors] - bevy_reflect: Binary formats

[MrGVSV]

Objective

Closes #5934

Currently it is not possible to de/serialize data to non-self-describing formats using reflection.

Solution

Add support for non-self-describing de/serialization using reflection.

This allows us to use binary formatters, like postcard:

#[derive(Reflect, FromReflect, Debug, PartialEq)]
struct Foo {
  data: String
}

let mut registry = TypeRegistry::new();
registry.register::<Foo>();

let input = Foo {
  data: "Hello world!".to_string()
};

// === Serialize! === //
let serializer = ReflectSerializer::new(&input, &registry);
let bytes: Vec<u8> = postcard::to_allocvec(&serializer).unwrap();

println!("{:?}", bytes); // Output: [129, 217, 61, 98, ...]

// === Deserialize! === //
let deserializer = UntypedReflectDeserializer::new(&registry);

let dynamic_output = deserializer
  .deserialize(&mut postcard::Deserializer::from_bytes(&bytes))
  .unwrap();

let output = <Foo as FromReflect>::from_reflect(dynamic_output.as_ref()).unwrap();

assert_eq!(expected, output); // OK!

Crates Tested

• rmp-serde Apparently, this is self-describing
• bincode v2.0.0-rc.1 (using this PR) This actually works for the latest release (v1.3.3) of bincode as well. You just need to be sure to use fixed-int encoding.
• postcard

Future Work

Ideally, we would refactor the serde module, but I don't think I'll do that in this PR so as to keep the diff relatively small (and to avoid any painful rebases). This should probably be done once this is merged, though.

Some areas we could improve with a refactor:

• Split deserialization logic across multiple files
• Consolidate helper functions/structs
• Make the logic more DRY

---

Changelog

• Add support for non-self-describing de/serialization using reflection.

[MrGVSV]

@cart I know we decided against this in this previous comment, but I discovered that the deserializer.deserialize_struct hint is actually used at least by bincode and postcard to determine how many fields exist in struct-like types.

If we don't want to support bincode we could just remove it.

Alternatively, we could attempt to split the logic based on deserializer.is_human_readable or something. Then if we're deserializing a non-self-describing format (such as bincode), we manually deserialize it as a tuple like deserializer.deserialize_tuple(struct_info.field_len(), StructVisitor{...}). And if it is human-readable, we just handle it as we currently do.

One concern with that is there might still be a difference between how certain formats deserialize. I actually ran into an issue where rmp-serde deserializes variant names as identifiers while bincode deserializes them as integers. This meant a simple is_human_readable couldn't actually handle both cases and a brand new VariantVisitor had to be made.

So any thoughts on this with the new context? Keep this change? Replace it with the human-readable check? Something else?

[MrGVSV]

My variant name issue might actually be related to this comment. So the split might not be as bad as originally thought.

The only other concern would be making that file a lot more complicated. But I think that can be addressed later with a good ol' refactor (unless we'd prefer to do the split and refactor at the same time).

[alice-i-cecile]

I'm fine to wait to refactor after.

[phaux]

I'm pretty sure that rmp-serde is a self-describing format because it's like CBOR and (de)serializing scene to/from CBOR already worked.

[MrGVSV]

I'm pretty sure that rmp-serde is a self-describing format because it's like CBOR and (de)serializing scene to/from CBOR already worked.

Oh didn't know that. I just assumed it wasn't self-describing because it was a binary format haha. Thanks for letting me know!

Yeah adding the deserialize_any call back in still allows rmp-serde to work. bincode and postcard both fail, though.

[irate-devil]

I don't have much insight into the implementation so just a nit :)

[zicklag]

I didn't look closely enough into the implementation to give any feedback, but I did just run into this exact issue today. I'm using reflection to power the serialization of components for sending over the network, and I'd like to be able to use postcard for it's size and speed.

For now I'll have to use a different data format, so this would definitely be appreciated!

[MrGVSV]

It was discovered on Discord that the scene deserializer does not respect non-self-describing formats. I've pinpointed the issue and will work on a fix so that this PR will actually be useful to people using scenes (and not just reflect). Blocking this in the meantime.

[MrGVSV]

DynamicScene (de)serializers should now support non-self-describing formats!

I added in tests for both postcard and bincode (since they seem to sometimes differ slightly) to ensure that we catch regressions a little quicker/easier.

[alice-i-cecile]

This LGTM now, and @inodentry was helpful to my understanding of this PR as well. I think those nits would be nice to address, but I wouldn't block on it.

[MrGVSV]

@mockersf Reverted the change. While cleaner, I forgot that there's no guarantee that we get the names back in the proper order from the HashMap.

[mockersf]

oh makes sense if they need to be ordered 👍

[cart]

Looks reasonable to me! Ultimately I think we'll want a completely different set of serializers and deserializers for binary formats, but this is a pretty easy win that unlocks these scenarios now. Great work!

[cart]

bors r+

[zicklag]

Ultimately I think we'll want a completely different set of serializers and deserializers for binary formats

I was wondering about that. In one iteration of a networking solution for a game I was working on, I ended up making a compact serializer/deserializer that would use a string cache to reduce the type names to indexes in order to reduce bandwidth usage.

I'm not sure if that's too specific to want to work into Bevy, but we could do something like that if we wanted to.

[bors]

Pull request successfully merged into main.

Build succeeded:

• build-and-install-on-iOS
• build-android
• build (macos-latest)
• build (ubuntu-latest)
• build-wasm
• build (windows-latest)
• build-without-default-features (bevy)
• build-without-default-features (bevy_ecs)
• build-without-default-features (bevy_reflect)
• check-compiles
• check-doc
• check-missing-examples-in-docs
• ci
• markdownlint
• run-examples
• run-examples-on-wasm
• run-examples-on-windows-dx12

[MrGVSV]

I ended up making a compact serializer/deserializer that would use a string cache to reduce the type names to indexes in order to reduce bandwidth usage.

I actually have a branch where I implemented that haha. But yeah talking with Cart about it, it's likely something we want to create a specialized mechanism for to be as efficient as possible, and reduce inadvertently placing limitations on the human-readable (de)serializers or vice-versa.