[Merged by Bors] - bevy_reflect: Improve serialization format even more #5723

MrGVSV · 2022-08-17T07:10:50Z

Note: This is rebased off #4561 and can be viewed as a competitor to that PR. See Comparison with #4561 section for details.

Objective

The current serialization format used by bevy_reflect is both verbose and error-prone. Taking the following structs¹ for example:

// -- src/inventory.rs

#[derive(Reflect)]
struct Inventory {
  id: String,
  max_storage: usize,
  items: Vec<Item>
}

#[derive(Reflect)]
struct Item {
  name: String
}

Given an inventory of a single item, this would serialize to something like:

// -- assets/inventory.ron

{
  "type": "my_game::inventory::Inventory",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "inv001",
    },
    "max_storage": {
      "type": "usize",
      "value": 10
    },
    "items": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "my_game::inventory::Item",
          "struct": {
            "name": {
              "type": "alloc::string::String",
              "value": "Pickaxe"
            },
          },
        },
      ],
    },
  },
}

Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need "struct". For lists, "list". For tuple structs, "tuple_struct". And so on.

It also requires that the "type" entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data.

This makes it very prone to errors and annoyances.

Solution

Using #4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like:

// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        name: "Pickaxe"
      ),
    ],
  ),
}

This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names).

Additionally, it is a lot more familiar to users as it follows conventional serde mechanics. For example, the struct is represented with (...) when serialized to RON.

Custom Serialization

Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example¹:

// -- src/inventory.rs

#[derive(Reflect, Serialize)]
#[reflect(Serialize)]
struct Item {
  #[serde(alias = "id")]
  name: String
}

// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        id: "Pickaxe"
      ),
    ],
  ),
},

By allowing users to define their own serialization methods, we do two things:

We give more control over how data is serialized/deserialized to the end user
We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a #[reflect(alias)] attribute).

Improved Formats

One of the improvements this PR provides is the ability to represent data in ways that are more conventional and/or familiar to users. Many users are familiar with RON so here are some of the ways we can now represent data in RON:

Structs

{
  "my_crate::Foo": (
    bar: 123
  )
}
// OR
{
  "my_crate::Foo": Foo(
    bar: 123
  )
}

Old Format

{
  "type": "my_crate::Foo",
  "struct": {
    "bar": {
      "type": "usize",
      "value": 123
    }
  }
}

Tuples

{
  "(f32, f32)": (1.0, 2.0)
}

Old Format

{
  "type": "(f32, f32)",
  "tuple": [
    {
      "type": "f32",
      "value": 1.0
    },
    {
      "type": "f32",
      "value": 2.0
    }
  ]
}

Tuple Structs

{
  "my_crate::Bar": ("Hello World!")
}
// OR
{
  "my_crate::Bar": Bar("Hello World!")
}

Old Format

{
  "type": "my_crate::Bar",
  "tuple_struct": [
    {
      "type": "alloc::string::String",
      "value": "Hello World!"
    }
  ]
}

Arrays

It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially are tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by serde and by bevy_reflect).

{
  "[i32; 3]": (1, 2, 3)
}

Old Format

{
  "type": "[i32; 3]",
  "array": [
    {
      "type": "i32",
      "value": 1
    },
    {
      "type": "i32",
      "value": 2
    },
    {
      "type": "i32",
      "value": 3
    }
  ]
}

Enums

To make things simple, I'll just put a struct variant here, but the style applies to all variant types:

{
  "my_crate::ItemType": Consumable(
    name: "Healing potion"
  )
}

Old Format

{
  "type": "my_crate::ItemType",
  "enum": {
    "variant": "Consumable",
    "struct": {
      "name": {
        "type": "alloc::string::String",
        "value": "Healing potion"
      }
    }
  }
}

Comparison with #4561

This PR is a rebased version of #4561. The reason for the split between the two is because this PR creates a very different scene format. You may notice that the PR descriptions for either PR are pretty similar. This was done to better convey the changes depending on which (if any) gets merged first. If #4561 makes it in first, I will update this PR description accordingly.

Changelog

Re-worked serialization/deserialization for reflected types
Added TypedReflectDeserializer for deserializing data with known TypeInfo
Renamed ReflectDeserializer to UntypedReflectDeserializer
~~Replaced usages of deserialize_any with deserialize_map for non-self-describing formats~~ Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like bincode can throw an error when attempting to deserialize non-self-describing formats (bincode results in DeserializeAnyNotSupported)
Structs, tuples, tuple structs, arrays, and enums are now all de/serialized using conventional serde methods

Migration Guide

This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly:

// Old format
{
  "type": "my_game::item::Item",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "bevycraft:stone",
    },
    "tags": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "alloc::string::String",
          "value": "material"
        },
      ],
    },
}

// New format
{
  "my_game::item::Item": (
    id: "bevycraft:stone",
    tags: ["material"]
  )
}

Some derives omitted for brevity. ↩ ↩²

This reverts commit 1ddf870.

Alainx277 · 2022-08-31T20:21:00Z

crates/bevy_reflect/src/serde/de.rs

    {
-        Ok(Box::new(v))
+        deserializer.deserialize_any(UntypedReflectDeserializerVisitor {


Could this be deserialize_map? UntypedReflectDeserializerVisitor doesn't seem to handle anything except a map.

deserialize_any doesn't work with formats that aren't self-describing like ex. bincode.

I mentioned this in #4561 (Changelog section), but I'm avoiding this change because I don't think it works even with deserialize_map. There might be other problems preventing it, but im not sure what those are.

For now, I’m leaving it like this so crates like bincode can at least recognize it and throw a helpful error message. Making this format binary safe should probably be its own PR.

cart

Love the new look and the implementation looks good for the most part! I think this is probably the direction we should take.

cart · 2022-08-31T20:32:14Z

crates/bevy_reflect/bevy_reflect_derive/src/impls/enums.rs

    let typed_impl = impl_typed(
        enum_name,
        reflect_enum.meta().generics(),
        quote! {
            let variants = [#(#variant_info),*];
-            let info = #bevy_reflect_path::EnumInfo::new::<Self>(&variants);
+            let info = #bevy_reflect_path::EnumInfo::new::<Self>(#string_name, &variants);


Storing the static "short" type name in EnumInfo/StructInfo might be (and maybe should be) redundant with efforts like #5805. We might want to split that pr's impl into:

impl TypePath { fn short_type_name() -> &'static str; fn module() -> &'static str; fn type_path() -> &'static str; }

That being said, this seems fine for now.

Ooh that's a good thought for #5805 actually. But yeah when that lands we can probably make these constructors take T: Reflect + TypePath instead.

crates/bevy_reflect/src/struct_trait.rs

crates/bevy_reflect/src/fields.rs

cart · 2022-08-31T20:56:50Z

crates/bevy_reflect/src/serde/ser.rs

        };

-        state.serialize_entry(type_fields::VARIANT, variant_name)?;
+        let enum_name = enum_info.name();
+        let variant_info = enum_info


These enum_info.variant(self.enum_value.variant_name()) and enum_info.index_of(self.enum_value.variant_name()) calls seem like "unnecessary" expenses and maybe point to holes in our Enum reflection impl?

Getting the variant info and the variant index of an enum that we have access to at runtime should be a simple match on the enum value, not two string hashes.

I think we need Enum::variant_index?

Yeah Enum::variant_index could work to reduce the hash lookups. I was going to suggest Enum::variant_info, but that would also be inefficient as we already have access to info here.

Yeah I think Enum::variant_index() combined with EnumInfo::variant_at is sufficient.

I updated the code. One note is that adding Enum::variant_index meant that we had to update DynamicEnum to account for this. I added DynamicEnum::new_with_index and DynamicEnum::set_variant_with_index.

However, this change means that we now rely on the index being properly set. If a user tries to serialize DynamicEnum, they need to make sure they actually set the index.

We could just force the user to define this whenever they set the variant or create an instance of DynamicEnum, but it makes the API a bit clunkier (and you need to know/remember the variant order).

Side note: For others' reference, I don't think we need to be concerned with the stability of variant order here. This change only affects serialization. So, except in very specific circumstances, the index should always be valid (at least for derived enums).

Hm, I'm wondering if I should just revert back to using the previous solution. I'm a little concerned about the potential pitfalls this introduces for users trying to serialize incomplete DynamicEnums (i.e. ones missing a valid variant index).

Oh, this also made me realize that this we really rely on proper ordering for things that shouldn't really care (at least in a Dynamic context), such as with structs. Not sure if we want to address that and opt for the name lookups instead (would involve editing Struct::iter_fields probably).

That could also (and probably should) just be addressed in a future PR.

Hm, I'm wondering if I should just revert back to using the previous solution. I'm a little concerned about the potential pitfalls this introduces for users trying to serialize incomplete DynamicEnums (i.e. ones missing a valid variant index).

Yeah this problem is the same for DynamicStructs and serialization (which also rely on the dynamic struct's field_at aligning with the actual TypeInfo's field_at).

Seems like we might want a "fast path" for "real structs/enums" and a "always correct path" (using field names) for dynamic structs/enums, rather than treating both as the same.

I don't want to revert the fast path (the common case) in the interest of correctness for the (very) uncommon case of using Dynamic types in user code (and then trying to serialize them).

Short term, I'm cool with the current impl as it aligns with the struct pattern, but post-merge we can follow up with an issue / discuss fixes for these cases.

crates/bevy_reflect/src/serde/de.rs

crates/bevy_reflect/bevy_reflect_derive/src/utility.rs

crates/bevy_reflect/src/serde/ser.rs

crates/bevy_reflect/src/serde/de.rs

cart · 2022-09-20T19:28:56Z

bors r+

> Note: This is rebased off #4561 and can be viewed as a competitor to that PR. See `Comparison with #4561` section for details. # Objective The current serialization format used by `bevy_reflect` is both verbose and error-prone. Taking the following structs[^1] for example: ```rust // -- src/inventory.rs #[derive(Reflect)] struct Inventory { id: String, max_storage: usize, items: Vec<Item> } #[derive(Reflect)] struct Item { name: String } ``` Given an inventory of a single item, this would serialize to something like: ```rust // -- assets/inventory.ron { "type": "my_game::inventory::Inventory", "struct": { "id": { "type": "alloc::string::String", "value": "inv001", }, "max_storage": { "type": "usize", "value": 10 }, "items": { "type": "alloc::vec::Vec<alloc::string::String>", "list": [ { "type": "my_game::inventory::Item", "struct": { "name": { "type": "alloc::string::String", "value": "Pickaxe" }, }, }, ], }, }, } ``` Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need `"struct"`. For lists, `"list"`. For tuple structs, `"tuple_struct"`. And so on. It also ***requires*** that the `"type"` entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data. This makes it very prone to errors and annoyances. ## Solution Using #4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like: ```rust // -- assets/inventory.ron { "my_game::inventory::Inventory": ( id: "inv001", max_storage: 10, items: [ ( name: "Pickaxe" ), ], ), } ``` This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names). Additionally, it is a lot more familiar to users as it follows conventional serde mechanics. For example, the struct is represented with `(...)` when serialized to RON. #### Custom Serialization Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example[^1]: ```rust // -- src/inventory.rs #[derive(Reflect, Serialize)] #[reflect(Serialize)] struct Item { #[serde(alias = "id")] name: String } ``` ```rust // -- assets/inventory.ron { "my_game::inventory::Inventory": ( id: "inv001", max_storage: 10, items: [ ( id: "Pickaxe" ), ], ), }, ``` By allowing users to define their own serialization methods, we do two things: 1. We give more control over how data is serialized/deserialized to the end user 2. We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a `#[reflect(alias)]` attribute). ### Improved Formats One of the improvements this PR provides is the ability to represent data in ways that are more conventional and/or familiar to users. Many users are familiar with RON so here are some of the ways we can now represent data in RON: ###### Structs ```js { "my_crate::Foo": ( bar: 123 ) } // OR { "my_crate::Foo": Foo( bar: 123 ) } ``` <details> <summary>Old Format</summary> ```js { "type": "my_crate::Foo", "struct": { "bar": { "type": "usize", "value": 123 } } } ``` </details> ###### Tuples ```js { "(f32, f32)": (1.0, 2.0) } ``` <details> <summary>Old Format</summary> ```js { "type": "(f32, f32)", "tuple": [ { "type": "f32", "value": 1.0 }, { "type": "f32", "value": 2.0 } ] } ``` </details> ###### Tuple Structs ```js { "my_crate::Bar": ("Hello World!") } // OR { "my_crate::Bar": Bar("Hello World!") } ``` <details> <summary>Old Format</summary> ```js { "type": "my_crate::Bar", "tuple_struct": [ { "type": "alloc::string::String", "value": "Hello World!" } ] } ``` </details> ###### Arrays It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially _are_ tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by [serde](https://docs.rs/serde/latest/serde/trait.Serialize.html#impl-Serialize-for-%5BT%3B%2032%5D) and by [bevy_reflect](https://docs.rs/bevy/latest/bevy/reflect/trait.GetTypeRegistration.html#impl-GetTypeRegistration-for-%5BT%3B%2032%5D)). ```js { "[i32; 3]": (1, 2, 3) } ``` <details> <summary>Old Format</summary> ```js { "type": "[i32; 3]", "array": [ { "type": "i32", "value": 1 }, { "type": "i32", "value": 2 }, { "type": "i32", "value": 3 } ] } ``` </details> ###### Enums To make things simple, I'll just put a struct variant here, but the style applies to all variant types: ```js { "my_crate::ItemType": Consumable( name: "Healing potion" ) } ``` <details> <summary>Old Format</summary> ```js { "type": "my_crate::ItemType", "enum": { "variant": "Consumable", "struct": { "name": { "type": "alloc::string::String", "value": "Healing potion" } } } } ``` </details> ### Comparison with #4561 This PR is a rebased version of #4561. The reason for the split between the two is because this PR creates a _very_ different scene format. You may notice that the PR descriptions for either PR are pretty similar. This was done to better convey the changes depending on which (if any) gets merged first. If #4561 makes it in first, I will update this PR description accordingly. --- ## Changelog * Re-worked serialization/deserialization for reflected types * Added `TypedReflectDeserializer` for deserializing data with known `TypeInfo` * Renamed `ReflectDeserializer` to `UntypedReflectDeserializer` * ~~Replaced usages of `deserialize_any` with `deserialize_map` for non-self-describing formats~~ Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like `bincode` can throw an error when attempting to deserialize non-self-describing formats (`bincode` results in `DeserializeAnyNotSupported`) * Structs, tuples, tuple structs, arrays, and enums are now all de/serialized using conventional serde methods ## Migration Guide * This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly: ```js // Old format { "type": "my_game::item::Item", "struct": { "id": { "type": "alloc::string::String", "value": "bevycraft:stone", }, "tags": { "type": "alloc::vec::Vec<alloc::string::String>", "list": [ { "type": "alloc::string::String", "value": "material" }, ], }, } // New format { "my_game::item::Item": ( id: "bevycraft:stone", tags: ["material"] ) } ``` [^1]: Some derives omitted for brevity.

bors · 2022-09-20T19:30:01Z

Build failed:

build (macos-latest)

cart · 2022-09-20T19:38:00Z

bors retry

> Note: This is rebased off #4561 and can be viewed as a competitor to that PR. See `Comparison with #4561` section for details. # Objective The current serialization format used by `bevy_reflect` is both verbose and error-prone. Taking the following structs[^1] for example: ```rust // -- src/inventory.rs #[derive(Reflect)] struct Inventory { id: String, max_storage: usize, items: Vec<Item> } #[derive(Reflect)] struct Item { name: String } ``` Given an inventory of a single item, this would serialize to something like: ```rust // -- assets/inventory.ron { "type": "my_game::inventory::Inventory", "struct": { "id": { "type": "alloc::string::String", "value": "inv001", }, "max_storage": { "type": "usize", "value": 10 }, "items": { "type": "alloc::vec::Vec<alloc::string::String>", "list": [ { "type": "my_game::inventory::Item", "struct": { "name": { "type": "alloc::string::String", "value": "Pickaxe" }, }, }, ], }, }, } ``` Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need `"struct"`. For lists, `"list"`. For tuple structs, `"tuple_struct"`. And so on. It also ***requires*** that the `"type"` entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data. This makes it very prone to errors and annoyances. ## Solution Using #4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like: ```rust // -- assets/inventory.ron { "my_game::inventory::Inventory": ( id: "inv001", max_storage: 10, items: [ ( name: "Pickaxe" ), ], ), } ``` This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names). Additionally, it is a lot more familiar to users as it follows conventional serde mechanics. For example, the struct is represented with `(...)` when serialized to RON. #### Custom Serialization Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example[^1]: ```rust // -- src/inventory.rs #[derive(Reflect, Serialize)] #[reflect(Serialize)] struct Item { #[serde(alias = "id")] name: String } ``` ```rust // -- assets/inventory.ron { "my_game::inventory::Inventory": ( id: "inv001", max_storage: 10, items: [ ( id: "Pickaxe" ), ], ), }, ``` By allowing users to define their own serialization methods, we do two things: 1. We give more control over how data is serialized/deserialized to the end user 2. We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a `#[reflect(alias)]` attribute). ### Improved Formats One of the improvements this PR provides is the ability to represent data in ways that are more conventional and/or familiar to users. Many users are familiar with RON so here are some of the ways we can now represent data in RON: ###### Structs ```js { "my_crate::Foo": ( bar: 123 ) } // OR { "my_crate::Foo": Foo( bar: 123 ) } ``` <details> <summary>Old Format</summary> ```js { "type": "my_crate::Foo", "struct": { "bar": { "type": "usize", "value": 123 } } } ``` </details> ###### Tuples ```js { "(f32, f32)": (1.0, 2.0) } ``` <details> <summary>Old Format</summary> ```js { "type": "(f32, f32)", "tuple": [ { "type": "f32", "value": 1.0 }, { "type": "f32", "value": 2.0 } ] } ``` </details> ###### Tuple Structs ```js { "my_crate::Bar": ("Hello World!") } // OR { "my_crate::Bar": Bar("Hello World!") } ``` <details> <summary>Old Format</summary> ```js { "type": "my_crate::Bar", "tuple_struct": [ { "type": "alloc::string::String", "value": "Hello World!" } ] } ``` </details> ###### Arrays It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially _are_ tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by [serde](https://docs.rs/serde/latest/serde/trait.Serialize.html#impl-Serialize-for-%5BT%3B%2032%5D) and by [bevy_reflect](https://docs.rs/bevy/latest/bevy/reflect/trait.GetTypeRegistration.html#impl-GetTypeRegistration-for-%5BT%3B%2032%5D)). ```js { "[i32; 3]": (1, 2, 3) } ``` <details> <summary>Old Format</summary> ```js { "type": "[i32; 3]", "array": [ { "type": "i32", "value": 1 }, { "type": "i32", "value": 2 }, { "type": "i32", "value": 3 } ] } ``` </details> ###### Enums To make things simple, I'll just put a struct variant here, but the style applies to all variant types: ```js { "my_crate::ItemType": Consumable( name: "Healing potion" ) } ``` <details> <summary>Old Format</summary> ```js { "type": "my_crate::ItemType", "enum": { "variant": "Consumable", "struct": { "name": { "type": "alloc::string::String", "value": "Healing potion" } } } } ``` </details> ### Comparison with #4561 This PR is a rebased version of #4561. The reason for the split between the two is because this PR creates a _very_ different scene format. You may notice that the PR descriptions for either PR are pretty similar. This was done to better convey the changes depending on which (if any) gets merged first. If #4561 makes it in first, I will update this PR description accordingly. --- ## Changelog * Re-worked serialization/deserialization for reflected types * Added `TypedReflectDeserializer` for deserializing data with known `TypeInfo` * Renamed `ReflectDeserializer` to `UntypedReflectDeserializer` * ~~Replaced usages of `deserialize_any` with `deserialize_map` for non-self-describing formats~~ Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like `bincode` can throw an error when attempting to deserialize non-self-describing formats (`bincode` results in `DeserializeAnyNotSupported`) * Structs, tuples, tuple structs, arrays, and enums are now all de/serialized using conventional serde methods ## Migration Guide * This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly: ```js // Old format { "type": "my_game::item::Item", "struct": { "id": { "type": "alloc::string::String", "value": "bevycraft:stone", }, "tags": { "type": "alloc::vec::Vec<alloc::string::String>", "list": [ { "type": "alloc::string::String", "value": "material" }, ], }, } // New format { "my_game::item::Item": ( id: "bevycraft:stone", tags: ["material"] ) } ``` [^1]: Some derives omitted for brevity.

bors · 2022-09-20T19:56:45Z

Pull request successfully merged into main.

Build succeeded:

> Note: This is rebased off bevyengine#4561 and can be viewed as a competitor to that PR. See `Comparison with bevyengine#4561` section for details. # Objective The current serialization format used by `bevy_reflect` is both verbose and error-prone. Taking the following structs[^1] for example: ```rust // -- src/inventory.rs #[derive(Reflect)] struct Inventory { id: String, max_storage: usize, items: Vec<Item> } #[derive(Reflect)] struct Item { name: String } ``` Given an inventory of a single item, this would serialize to something like: ```rust // -- assets/inventory.ron { "type": "my_game::inventory::Inventory", "struct": { "id": { "type": "alloc::string::String", "value": "inv001", }, "max_storage": { "type": "usize", "value": 10 }, "items": { "type": "alloc::vec::Vec<alloc::string::String>", "list": [ { "type": "my_game::inventory::Item", "struct": { "name": { "type": "alloc::string::String", "value": "Pickaxe" }, }, }, ], }, }, } ``` Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need `"struct"`. For lists, `"list"`. For tuple structs, `"tuple_struct"`. And so on. It also ***requires*** that the `"type"` entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data. This makes it very prone to errors and annoyances. ## Solution Using bevyengine#4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like: ```rust // -- assets/inventory.ron { "my_game::inventory::Inventory": ( id: "inv001", max_storage: 10, items: [ ( name: "Pickaxe" ), ], ), } ``` This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names). Additionally, it is a lot more familiar to users as it follows conventional serde mechanics. For example, the struct is represented with `(...)` when serialized to RON. #### Custom Serialization Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example[^1]: ```rust // -- src/inventory.rs #[derive(Reflect, Serialize)] #[reflect(Serialize)] struct Item { #[serde(alias = "id")] name: String } ``` ```rust // -- assets/inventory.ron { "my_game::inventory::Inventory": ( id: "inv001", max_storage: 10, items: [ ( id: "Pickaxe" ), ], ), }, ``` By allowing users to define their own serialization methods, we do two things: 1. We give more control over how data is serialized/deserialized to the end user 2. We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a `#[reflect(alias)]` attribute). ### Improved Formats One of the improvements this PR provides is the ability to represent data in ways that are more conventional and/or familiar to users. Many users are familiar with RON so here are some of the ways we can now represent data in RON: ###### Structs ```js { "my_crate::Foo": ( bar: 123 ) } // OR { "my_crate::Foo": Foo( bar: 123 ) } ``` <details> <summary>Old Format</summary> ```js { "type": "my_crate::Foo", "struct": { "bar": { "type": "usize", "value": 123 } } } ``` </details> ###### Tuples ```js { "(f32, f32)": (1.0, 2.0) } ``` <details> <summary>Old Format</summary> ```js { "type": "(f32, f32)", "tuple": [ { "type": "f32", "value": 1.0 }, { "type": "f32", "value": 2.0 } ] } ``` </details> ###### Tuple Structs ```js { "my_crate::Bar": ("Hello World!") } // OR { "my_crate::Bar": Bar("Hello World!") } ``` <details> <summary>Old Format</summary> ```js { "type": "my_crate::Bar", "tuple_struct": [ { "type": "alloc::string::String", "value": "Hello World!" } ] } ``` </details> ###### Arrays It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially _are_ tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by [serde](https://docs.rs/serde/latest/serde/trait.Serialize.html#impl-Serialize-for-%5BT%3B%2032%5D) and by [bevy_reflect](https://docs.rs/bevy/latest/bevy/reflect/trait.GetTypeRegistration.html#impl-GetTypeRegistration-for-%5BT%3B%2032%5D)). ```js { "[i32; 3]": (1, 2, 3) } ``` <details> <summary>Old Format</summary> ```js { "type": "[i32; 3]", "array": [ { "type": "i32", "value": 1 }, { "type": "i32", "value": 2 }, { "type": "i32", "value": 3 } ] } ``` </details> ###### Enums To make things simple, I'll just put a struct variant here, but the style applies to all variant types: ```js { "my_crate::ItemType": Consumable( name: "Healing potion" ) } ``` <details> <summary>Old Format</summary> ```js { "type": "my_crate::ItemType", "enum": { "variant": "Consumable", "struct": { "name": { "type": "alloc::string::String", "value": "Healing potion" } } } } ``` </details> ### Comparison with bevyengine#4561 This PR is a rebased version of bevyengine#4561. The reason for the split between the two is because this PR creates a _very_ different scene format. You may notice that the PR descriptions for either PR are pretty similar. This was done to better convey the changes depending on which (if any) gets merged first. If bevyengine#4561 makes it in first, I will update this PR description accordingly. --- ## Changelog * Re-worked serialization/deserialization for reflected types * Added `TypedReflectDeserializer` for deserializing data with known `TypeInfo` * Renamed `ReflectDeserializer` to `UntypedReflectDeserializer` * ~~Replaced usages of `deserialize_any` with `deserialize_map` for non-self-describing formats~~ Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like `bincode` can throw an error when attempting to deserialize non-self-describing formats (`bincode` results in `DeserializeAnyNotSupported`) * Structs, tuples, tuple structs, arrays, and enums are now all de/serialized using conventional serde methods ## Migration Guide * This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly: ```js // Old format { "type": "my_game::item::Item", "struct": { "id": { "type": "alloc::string::String", "value": "bevycraft:stone", }, "tags": { "type": "alloc::vec::Vec<alloc::string::String>", "list": [ { "type": "alloc::string::String", "value": "material" }, ], }, } // New format { "my_game::item::Item": ( id: "bevycraft:stone", tags: ["material"] ) } ``` [^1]: Some derives omitted for brevity.

…izer` (#12721) # Objective We have `ReflectSerializer` and `TypedReflectSerializer`. The former is the one users will most often use since the latter takes a bit more effort to deserialize. However, our deserializers are named `UntypedReflectDeserializer` and `TypedReflectDeserializer`. There is no obvious indication that `UntypedReflectDeserializer` must be used with `ReflectSerializer` since the names don't quite match up. ## Solution Rename `UntypedReflectDeserializer` back to `ReflectDeserializer` (initially changed as part of #5723). Also update the docs for both deserializers (as they were pretty out of date) and include doc examples. I also updated the docs for the serializers, too, just so that everything is consistent. --- ## Changelog - Renamed `UntypedReflectDeserializer` to `ReflectDeserializer` - Updated docs for `ReflectDeserializer`, `TypedReflectDeserializer`, `ReflectSerializer`, and `TypedReflectSerializer` ## Migration Guide `UntypedReflectDeserializer` has been renamed to `ReflectDeserializer`. Usages will need to be updated accordingly. ```diff - let reflect_deserializer = UntypedReflectDeserializer::new(&registry); + let reflect_deserializer = ReflectDeserializer::new(&registry); ```

MrGVSV added C-Usability A simple quality-of-life change that makes Bevy easier to use A-Reflection Runtime information about types X-Controversial There is active debate or serious implications around merging this PR labels Aug 17, 2022

MrGVSV marked this pull request as ready for review August 18, 2022 03:33

MrGVSV mentioned this pull request Aug 18, 2022

bevy_reflect: Improve serialization format #4561

Closed

inodentry pushed a commit to BroovyJammy/bevy that referenced this pull request Aug 19, 2022

Pr bevyengine#5723

1ddf870

MrGVSV force-pushed the reflect-improved-serialization-v2 branch from 785978b to 2bdb852 Compare August 21, 2022 07:12

inodentry added a commit to BroovyJammy/bevy that referenced this pull request Aug 21, 2022

Revert "Pr bevyengine#5723"

a02acfe

This reverts commit 1ddf870.

MrGVSV force-pushed the reflect-improved-serialization-v2 branch 2 times, most recently from 740ea5e to 63b561c Compare August 24, 2022 21:08

MrGVSV mentioned this pull request Aug 29, 2022

bevy_reflect: Type aliases #5830

Draft

3 tasks

MrGVSV force-pushed the reflect-improved-serialization-v2 branch from 63b561c to e4ecdd1 Compare August 29, 2022 20:04

cart added the A-Scenes Serialized ECS data stored on the disk label Aug 31, 2022

Alainx277 reviewed Aug 31, 2022

View reviewed changes

cart reviewed Aug 31, 2022

View reviewed changes

MrGVSV force-pushed the reflect-improved-serialization-v2 branch from e4ecdd1 to 8509719 Compare September 1, 2022 06:15

cart mentioned this pull request Sep 1, 2022

stable reflect type name #5805

Closed

5 tasks

MrGVSV force-pushed the reflect-improved-serialization-v2 branch from 5fff3bc to 86f9cab Compare September 14, 2022 07:16

MrGVSV mentioned this pull request Sep 17, 2022

bevy_reflect: Trait casting #5001

Closed

alice-i-cecile added this to the Bevy 0.9 milestone Sep 19, 2022

MrGVSV force-pushed the reflect-improved-serialization-v2 branch 2 times, most recently from 786b42c to 01a800a Compare September 19, 2022 23:01

MrGVSV commented Sep 19, 2022

View reviewed changes

crates/bevy_reflect/bevy_reflect_derive/src/utility.rs Show resolved Hide resolved

crates/bevy_reflect/src/serde/ser.rs Show resolved Hide resolved

crates/bevy_reflect/src/serde/de.rs Outdated Show resolved Hide resolved

MrGVSV force-pushed the reflect-improved-serialization-v2 branch 2 times, most recently from eb012c2 to a00f778 Compare September 20, 2022 00:14

MrGVSV added 5 commits September 19, 2022 18:27

Simplified reflection ser/de

4230acd

Cleanup comment

32a83b9

Updated example scene file

e8bdefb

Fix clippy

3dd5d0e

Fix glam tests

f794ad2

MrGVSV added 11 commits September 19, 2022 18:27

Allow Dynamic types to be serialized again

a2b6e2e

Formatting

43fb3ab

Support Option types

df99b96

Fix clippy errors

509cb9f

Fix glam tests

61a742b

Remove unnecessary Into

070e509

Fix getting registration for type twice

320d9e4

Added Enum::variant_index

1d34ea5

Remove duplicate field_names/variant_names fields

265cc30

Simplify ExpectedValues struct

6430af4

Fix doc error

fe2ee1c

MrGVSV force-pushed the reflect-improved-serialization-v2 branch from a00f778 to fe2ee1c Compare September 20, 2022 01:27

Pass in TypeRegistration to cut out redundant lookup

8f6ed32

bors bot changed the title ~~bevy_reflect: Improve serialization format even more~~ [Merged by Bors] - bevy_reflect: Improve serialization format even more Sep 20, 2022

bors bot closed this Sep 20, 2022

Gingeh mentioned this pull request Sep 20, 2022

Implement Reflect for PhantomData #5145

Closed

MrGVSV mentioned this pull request Oct 2, 2022

[Merged by Bors] - bevy_reflect: Binary formats #6140

Closed

Suficio mentioned this pull request Oct 7, 2022

bevy_reflect: Convert dynamic types to concrete instances #4597

Closed

ickk added the C-Breaking-Change A breaking change to Bevy's public API that needs to be noted in a migration guide label Oct 26, 2022

MrGVSV mentioned this pull request Mar 26, 2024

bevy_reflect: Rename UntypedReflectDeserializer to ReflectDeserializer #12721

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - bevy_reflect: Improve serialization format even more #5723

[Merged by Bors] - bevy_reflect: Improve serialization format even more #5723

MrGVSV commented Aug 17, 2022 •

edited

Alainx277 Aug 31, 2022

MrGVSV Aug 31, 2022

cart left a comment

cart Aug 31, 2022

MrGVSV Sep 1, 2022

cart Aug 31, 2022

MrGVSV Sep 1, 2022

cart Sep 1, 2022

MrGVSV Sep 1, 2022

MrGVSV Sep 14, 2022

MrGVSV Sep 19, 2022

cart Sep 20, 2022

cart commented Sep 20, 2022

bors bot commented Sep 20, 2022

cart commented Sep 20, 2022

bors bot commented Sep 20, 2022

[Merged by Bors] - bevy_reflect: Improve serialization format even more #5723

[Merged by Bors] - bevy_reflect: Improve serialization format even more #5723

Conversation

MrGVSV commented Aug 17, 2022 • edited

Objective

Solution

Custom Serialization

Improved Formats

Structs

Tuples

Tuple Structs

Arrays

Enums

Comparison with #4561

Changelog

Migration Guide

Footnotes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cart left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cart commented Sep 20, 2022

bors bot commented Sep 20, 2022

cart commented Sep 20, 2022

bors bot commented Sep 20, 2022

MrGVSV commented Aug 17, 2022 •

edited