Headless serialization with customs property name

[Metalaka]

Hello,
I want to make a headless serialization with customs properties name.

I have a DTO class with all properties of a shared schema but with custom names and different property order.
The DTO properties are bind to the schema properties with a DataMember attribute.

Because of my use case, I must use the client schema for the serialization.
In short, I have to communicate with one client in two different environment. And the order of properties is a little bit different between environments.

this is what my DTO looks like

[DataContract(Name = "...", Namespace = "...")]
public class UserDto
{
	[DataMember(Name = "Foo")]
	public string FirstName { get; set; }
	[DataMember(Name = "Bar")]
	public string LastName { get; set; }
}

what i do to serialize my payload

public void Publish(UserDto payload)
{
	string jsonSchema = GetJsonSchemaFromClient();

	byte[] bytes = AvroConvert.SerializeHeadless(payload, jsonSchema);

	// send to client
}

And for example, what i did for the read flow. It works because JsonConvert handle DataMemberAttribute

public UserDto Fetch()
{
	string jsonSchema = GetJsonSchemaFromClient();

	// var bytePayload = read from client

	string jsonPayload = AvroConvert.Avro2Json(bytePayload, jsonSchema);
	return JsonConvert.DeserializeObject<UserDto>(jsonPayload);
}

I tried to add the feature but I don't know if it's the right way: Metalaka@c29e645
Maybe you wanna use AvroContractResolver.ResolveMembers(Type) and add support for more options?

Do you have any plan to support that kind of serialization ?

[AdrianStrugala]

Hello,
Thank you for raising this topic. Yes, the custom mapping should be handled during a schema build. Let me investigate this and come back to you.
Adrian

[AdrianStrugala]

Hi @Metalaka,

I have written a test covering your case:

 public class HeadlessComponentTests2
    {
        private readonly Fixture _fixture = new();


        [Fact]
        public void Component_HeadlessSerializationUsingTwoModelsWithDifferentPropertyNames_CorrectlyDeserialized()
        {
            //Arrange
            UserDto toSerialize = _fixture.Create<UserDto>();
            string writeSchema = AvroConvert.GenerateSchema(typeof(UserDto));
            string readSchema = AvroConvert.GenerateSchema(typeof(UserDto2));

            //Act
            var result = AvroConvert.SerializeHeadless(toSerialize, writeSchema);

            var deserialized = AvroConvert.DeserializeHeadless<UserDto2>(result, readSchema);

            //Assert
            Assert.NotNull(result);
            Assert.NotNull(deserialized);
            Assert.Equal(toSerialize.Foo, deserialized.FirstName);
            Assert.Equal(toSerialize.Bar, deserialized.LastName);
        }
    }

    public class UserDto2
    {
        [DataMember(Name = "Foo")] 
        public string FirstName { get; set; }
        [DataMember(Name = "Bar")]
        public string LastName { get; set; }
    }


    public class UserDto
    {
        public string Foo { get; set; }
        public string Bar { get; set; }

    }

As you can see, I am using different schemas for serialization and deserialization. The client model uses the DataMember attribute and the functionality works correctly.
Note, that the schema used for deserialization is not the one coming from client string jsonSchema = GetJsonSchemaFromClient();, but generated for the new model string readSchema = AvroConvert.GenerateSchema(typeof(UserDto2));

Does it help in your case?

[Metalaka]

Hello,

Your test use the same data types so there is no reason to the deserialization to fail.

I added different data type and different property order. So the test will be closer to my real case.
The server schema is out of my scope, this is the reason why I have to serialize my data to that schema

        [Fact]
        public void Component_HeadlessSerializationUsingTwoModelsWithDifferentPropertyOrder_CorrectlyDeserialized()
        {
            //Arrange
            ClientUserDto toSerialize = _fixture.Create<ClientUserDto>();
            //string clientSchema = AvroConvert.GenerateSchema(typeof(ClientUserDto));
            string serverSchema = AvroConvert.GenerateSchema(typeof(ServerUserDto));

            //Act
            var result = AvroConvert.SerializeHeadless(toSerialize, serverSchema);

            var deserialized = AvroConvert.DeserializeHeadless<ServerUserDto>(result, serverSchema);

            //Assert
            Assert.NotNull(result);
            Assert.NotNull(deserialized);
            Assert.Equal(toSerialize.FirstName, deserialized.Foo);
            Assert.Equal(toSerialize.Id, deserialized.Bar);
        }

        public class ClientUserDto
        {
            [DataMember(Name = "Bar")]
            public int Id { get; set; }

            [DataMember(Name = "Foo")]
            public string FirstName { get; set; }
        }

        public class ServerUserDto
        {
            public string Foo { get; set; }
            public int Bar { get; set; }
        }

Thanks for your time

[AdrianStrugala]

To give you more insights: the property order is not a problem at all. The problem are the property names.
For the serialization, you are providing an object with two properties: Id and FirstName, and at the same time you are telling it to serialize against the schema containing a class with two properties of different names: Foo and Bar.

As I understand, you expect to take into consideration the DataMember attributes, but this is opposite to the Avro specification:

Named types and fields may have aliases. An implementation may optionally use aliases to map a writer’s schema to the reader’s. This facilitates both schema evolution as well as processing disparate datasets. Aliases function by re-writing the writer’s schema using aliases from the reader’s schema. For example, if the writer’s schema was named “Foo” and the reader’s schema is named “Bar” and has an alias of “Foo”, then the implementation would act as though “Foo” were named “Bar” when reading. Similarly, if data was written as a record with a field named “x” and is read as a record with a field named “y” with alias “x”, then the implementation would act as though “x” were named “y” when reading.

1. As a go-to solution, I would suggest not using the Headless option, but going with full Serialization and Deserialization:

           var result = AvroConvert.Serialize(toSerialize);
           var deserialized = AvroConvert.Deserialize<ServerUserDto>(result);

This is the simplest way of ensuring schema compatibility.

2. Alternatively, as a quick solution, you can rename the properties of the ClientUserDto to match the Server names.

I will try to dig deeper into this topic, but the general rule is to stick to the writer schema and use this one in the schema registry. Please let me know if I got something wrong.

[github-actions]

Closed due to inactivity.