Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support custom Json deserializer/serializer #6

Open
volyx opened this issue May 26, 2019 · 12 comments
Open

Support custom Json deserializer/serializer #6

volyx opened this issue May 26, 2019 · 12 comments
Assignees

Comments

@volyx
Copy link

volyx commented May 26, 2019

For example Gson instead Jackson.
Move json to another maven module and allow to add it as separate jar.

@voidmain
Copy link
Member

Hi @volyx. Thanks for submitting this request. The use of Jackson in the library is mostly private, meaning that Jackson classes aren't returned from the methods. Is your need for GSON so that you don't need to include Jackson in your classpath?

@volyx
Copy link
Author

volyx commented Jun 8, 2019

Yes, I already exclude all jackson libraries from my project for using only Gson, but I can't replace Jackson with Gson in fusionauth-jwt. So it would be nice to have a opportunity to somehow register custom serializer/deserializer. For example

JsonSerializer ser = new GsonSerializer()
 JWT jwt = new JWT()
                .setSerializer(ser)
                .setIssuer("www.example.com")
                .setIssuedAt(iss)
                .setSubject(subject)
                .setExpiration(exp);

@robotdan
Copy link
Member

robotdan commented Jun 8, 2019

Thanks for that additional info @volyx . In theory we could abstract the mapper interface and allow a different module / library to be bound for serializing and deserializing.

Most of that code is in Mapper.java - however - we are using some Jackson annotations in the domain which makes it a bit tough without some re-work to the library.

We are using the Jackson @AnySetter and @AnyGetter annotations from Jackson, a similar function would need to exist in Gson.

While not ideal, you could fork the repo and just rip out Jackson.

I'm open to this feature in FusionAuth JWT - we'd need to move the Jackson annotations out of the domain and into some other configuration, or perhaps we could use Jackson mixins to decorate the domain with the annotations we'd need. If that was all done, then it wouldn't be too hard to make a new interface for Guice to bind one of several implementations.

This isn't a high priority - but we can leave this open and see if we have some time to hack on it a bit. Feel free to offer some example code or PRs if you have an idea of how it would work.

Thanks!

@robotdan robotdan assigned robotdan, volyx and voidmain and unassigned volyx Jun 8, 2019
@volyx
Copy link
Author

volyx commented Aug 17, 2019

outdated

As I see there are two options:

  1. First is to create a multi-module maven project.
    So fusionauth-jwt-parent will have such structure:
fusionauth-jwt-parent
- jwt
- jacksson-serializer
- fusionauth-jwt(depends on jwt,  jacksjon-serializer)
- gson-serializer

So users can still use fusionauth-jwt library and don't notice any differences after versoin update
2) Do separation in runtime. We could check in runtime availability of JSON library:

if (ObjectMapper class exists) use jackson serserializer
if (Gson class exists) use gson serserializer

So user could exclude Jackson, and add Gson library.

First approach more modular, and allow future extension but needs more work and restructurization.
Second one needs less time, but use some kind of runtime class discovery that is not a very good pattern but still a workable solution.

@volyx
Copy link
Author

volyx commented Feb 20, 2020

PoC for removing Jackson

Problem: Jackson libraries are performant but heavy weight and clash with other Jackson libraries in a project

Solution: use lightweight JSON library or do not even use any JSON library - decode/encode manually

Library sizes:

340K jackson-core-2.10.1.jar
66K jackson-annotations-2.10.1.jar
1.3M jackson-databind-2.10.1.jar
136K fusionauth-jwt-3.1.6.jar

fusionauth-jwt library is just 136 Kilobytes.

For comparison
29K nanojson-1.4.jar
33K minimal-json-0.9.5.jar
8.1K JsonReader.java(manually written json reader)

I have added JSON serializers, deserializers which can replace Jackson. I have tested three options: minimal-json, nanojson(embeded), plain(manual)

All them can be tested for compatibility with Jackson ObjectMapper like this:
./mvnw test -Dminimal.json.decode=true use minimal-java for decoding, Jackson for decoding
./mvnw test -Dminimal.json.encode=true use minimal-java for encoding, Jackson for encoding
./mvnw test -Dminimal.json.encode=true -Dminimal.json.decode=true use minimal-java for encoding and decoding

The same is for others encoders/decoders:
./mvnw test -Dnano.json.encode=true -Dnano.json.decode=true
./mvnw test -Dplain.json.encode=true -Dplain.json.decode=true

I wrote benchmark for testing the performance of all these JSON mappers - JsonBenchmark.

Benchmark                                    (serializer)    Mode      Cnt       Score    Error   Units
JsonBenchmark.testDecode                          Jackson   thrpt        5       0,800 ±  0,329  ops/us
JsonBenchmark.testDecode                      MinimalJson   thrpt        5       0,247 ±  0,093  ops/us
JsonBenchmark.testDecode                         NanoJson   thrpt        5       0,063 ±  0,062  ops/us
JsonBenchmark.testDecode                            Plain   thrpt        5       0,538 ±  0,229  ops/us
JsonBenchmark.testDecode                             None   thrpt        5       6,863 ±  1,322  ops/us
JsonBenchmark.testEncode                          Jackson   thrpt        5       0,235 ±  0,040  ops/us
JsonBenchmark.testEncode                      MinimalJson   thrpt        5       0,097 ±  0,013  ops/us
JsonBenchmark.testEncode                         NanoJson   thrpt        5       0,088 ±  0,102  ops/us
JsonBenchmark.testEncode                            Plain   thrpt        5       0,157 ±  0,077  ops/us
JsonBenchmark.testEncode                             None   thrpt        5      24,158 ± 21,829  ops/us
JsonBenchmark.testDecode                          Jackson    avgt        5       1,189 ±  0,222   us/op
JsonBenchmark.testDecode                      MinimalJson    avgt        5       4,250 ±  2,976   us/op
JsonBenchmark.testDecode                         NanoJson    avgt        5      14,250 ±  6,014   us/op
JsonBenchmark.testDecode                            Plain    avgt        5       1,955 ±  2,875   us/op
JsonBenchmark.testDecode                             None    avgt        5       0,137 ±  0,033   us/op
JsonBenchmark.testEncode                          Jackson    avgt        5       5,854 ± 10,353   us/op
JsonBenchmark.testEncode                      MinimalJson    avgt        5      10,302 ± 10,342   us/op
JsonBenchmark.testEncode                         NanoJson    avgt        5       9,826 ±  7,726   us/op
JsonBenchmark.testEncode                            Plain    avgt        5       5,453 ±  0,905   us/op
JsonBenchmark.testEncode                             None    avgt        5       0,043 ±  0,013   us/op
JsonBenchmark.testDecode                          Jackson  sample  1201091       1,692 ±  0,097   us/op
JsonBenchmark.testDecode:testDecode·p0.00         Jackson  sample                0,957            us/op
JsonBenchmark.testDecode:testDecode·p0.50         Jackson  sample                1,024            us/op
JsonBenchmark.testDecode:testDecode·p0.90         Jackson  sample                1,880            us/op
JsonBenchmark.testDecode:testDecode·p0.95         Jackson  sample                2,184            us/op
JsonBenchmark.testDecode:testDecode·p0.99         Jackson  sample                4,200            us/op
JsonBenchmark.testDecode:testDecode·p0.999        Jackson  sample               78,336            us/op
JsonBenchmark.testDecode:testDecode·p0.9999       Jackson  sample              786,226            us/op
JsonBenchmark.testDecode:testDecode·p1.00         Jackson  sample            26869,760            us/op
JsonBenchmark.testDecode                      MinimalJson  sample  1248740       6,599 ±  0,489   us/op
JsonBenchmark.testDecode:testDecode·p0.00     MinimalJson  sample                2,860            us/op
JsonBenchmark.testDecode:testDecode·p0.50     MinimalJson  sample                3,360            us/op
JsonBenchmark.testDecode:testDecode·p0.90     MinimalJson  sample                5,472            us/op
JsonBenchmark.testDecode:testDecode·p0.95     MinimalJson  sample                5,872            us/op
JsonBenchmark.testDecode:testDecode·p0.99     MinimalJson  sample               17,280            us/op
JsonBenchmark.testDecode:testDecode·p0.999    MinimalJson  sample              410,245            us/op
JsonBenchmark.testDecode:testDecode·p0.9999   MinimalJson  sample             4738,070            us/op
JsonBenchmark.testDecode:testDecode·p1.00     MinimalJson  sample            90177,536            us/op
JsonBenchmark.testDecode                         NanoJson  sample   925259      14,588 ±  0,234   us/op
JsonBenchmark.testDecode:testDecode·p0.00        NanoJson  sample                7,344            us/op
JsonBenchmark.testDecode:testDecode·p0.50        NanoJson  sample               10,256            us/op
JsonBenchmark.testDecode:testDecode·p0.90        NanoJson  sample               15,440            us/op
JsonBenchmark.testDecode:testDecode·p0.95        NanoJson  sample               21,440            us/op
JsonBenchmark.testDecode:testDecode·p0.99        NanoJson  sample               59,802            us/op
JsonBenchmark.testDecode:testDecode·p0.999       NanoJson  sample              663,552            us/op
JsonBenchmark.testDecode:testDecode·p0.9999      NanoJson  sample             2514,731            us/op
JsonBenchmark.testDecode:testDecode·p1.00        NanoJson  sample            20054,016            us/op
JsonBenchmark.testDecode                            Plain  sample  1526500       2,457 ±  0,138   us/op
JsonBenchmark.testDecode:testDecode·p0.00           Plain  sample                1,480            us/op
JsonBenchmark.testDecode:testDecode·p0.50           Plain  sample                1,572            us/op
JsonBenchmark.testDecode:testDecode·p0.90           Plain  sample                2,788            us/op
JsonBenchmark.testDecode:testDecode·p0.95           Plain  sample                3,184            us/op
JsonBenchmark.testDecode:testDecode·p0.99           Plain  sample                6,416            us/op
JsonBenchmark.testDecode:testDecode·p0.999          Plain  sample               98,304            us/op
JsonBenchmark.testDecode:testDecode·p0.9999         Plain  sample              891,597            us/op
JsonBenchmark.testDecode:testDecode·p1.00           Plain  sample            54001,664            us/op
JsonBenchmark.testDecode                             None  sample  1371906       0,821 ±  0,246   us/op
JsonBenchmark.testDecode:testDecode·p0.00            None  sample                0,139            us/op
JsonBenchmark.testDecode:testDecode·p0.50            None  sample                0,163            us/op
JsonBenchmark.testDecode:testDecode·p0.90            None  sample                0,297            us/op
JsonBenchmark.testDecode:testDecode·p0.95            None  sample                0,311            us/op
JsonBenchmark.testDecode:testDecode·p0.99            None  sample                0,554            us/op
JsonBenchmark.testDecode:testDecode·p0.999           None  sample               19,744            us/op
JsonBenchmark.testDecode:testDecode·p0.9999          None  sample              801,597            us/op
JsonBenchmark.testDecode:testDecode·p1.00            None  sample            38010,880            us/op
JsonBenchmark.testEncode                          Jackson  sample  1411865       4,719 ±  0,049   us/op
JsonBenchmark.testEncode:testEncode·p0.00         Jackson  sample                3,468            us/op
JsonBenchmark.testEncode:testEncode·p0.50         Jackson  sample                3,584            us/op
JsonBenchmark.testEncode:testEncode·p0.90         Jackson  sample                5,936            us/op
JsonBenchmark.testEncode:testEncode·p0.95         Jackson  sample                7,056            us/op
JsonBenchmark.testEncode:testEncode·p0.99         Jackson  sample               15,568            us/op
JsonBenchmark.testEncode:testEncode·p0.999        Jackson  sample              128,913            us/op
JsonBenchmark.testEncode:testEncode·p0.9999       Jackson  sample              849,538            us/op
JsonBenchmark.testEncode:testEncode·p1.00         Jackson  sample             4415,488            us/op
JsonBenchmark.testEncode                      MinimalJson  sample  1263047      10,270 ±  0,091   us/op
JsonBenchmark.testEncode:testEncode·p0.00     MinimalJson  sample                7,064            us/op
JsonBenchmark.testEncode:testEncode·p0.50     MinimalJson  sample                7,632            us/op
JsonBenchmark.testEncode:testEncode·p0.90     MinimalJson  sample               13,536            us/op
JsonBenchmark.testEncode:testEncode·p0.95     MinimalJson  sample               15,920            us/op
JsonBenchmark.testEncode:testEncode·p0.99     MinimalJson  sample               40,448            us/op
JsonBenchmark.testEncode:testEncode·p0.999    MinimalJson  sample              265,728            us/op
JsonBenchmark.testEncode:testEncode·p0.9999   MinimalJson  sample             1106,095            us/op
JsonBenchmark.testEncode:testEncode·p1.00     MinimalJson  sample            10371,072            us/op
JsonBenchmark.testEncode                         NanoJson  sample  1378816       9,582 ±  0,330   us/op
JsonBenchmark.testEncode:testEncode·p0.00        NanoJson  sample                7,032            us/op
JsonBenchmark.testEncode:testEncode·p0.50        NanoJson  sample                7,856            us/op
JsonBenchmark.testEncode:testEncode·p0.90        NanoJson  sample                9,344            us/op
JsonBenchmark.testEncode:testEncode·p0.95        NanoJson  sample               12,928            us/op
JsonBenchmark.testEncode:testEncode·p0.99        NanoJson  sample               23,232            us/op
JsonBenchmark.testEncode:testEncode·p0.999       NanoJson  sample              164,911            us/op
JsonBenchmark.testEncode:testEncode·p0.9999      NanoJson  sample             1223,867            us/op
JsonBenchmark.testEncode:testEncode·p1.00        NanoJson  sample           116391,936            us/op
JsonBenchmark.testEncode                            Plain  sample  1085856       6,361 ±  0,711   us/op
JsonBenchmark.testEncode:testEncode·p0.00           Plain  sample                4,512            us/op
JsonBenchmark.testEncode:testEncode·p0.50           Plain  sample                4,848            us/op
JsonBenchmark.testEncode:testEncode·p0.90           Plain  sample                5,688            us/op
JsonBenchmark.testEncode:testEncode·p0.95           Plain  sample                8,192            us/op
JsonBenchmark.testEncode:testEncode·p0.99           Plain  sample               15,303            us/op
JsonBenchmark.testEncode:testEncode·p0.999          Plain  sample              122,752            us/op
JsonBenchmark.testEncode:testEncode·p0.9999         Plain  sample             1221,105            us/op
JsonBenchmark.testEncode:testEncode·p1.00           Plain  sample           228589,568            us/op
JsonBenchmark.testEncode                             None  sample  1201647       0,136 ±  0,016   us/op
JsonBenchmark.testEncode:testEncode·p0.00            None  sample                0,027            us/op
JsonBenchmark.testEncode:testEncode·p0.50            None  sample                0,073            us/op
JsonBenchmark.testEncode:testEncode·p0.90            None  sample                0,105            us/op
JsonBenchmark.testEncode:testEncode·p0.95            None  sample                0,133            us/op
JsonBenchmark.testEncode:testEncode·p0.99            None  sample                0,288            us/op
JsonBenchmark.testEncode:testEncode·p0.999           None  sample                9,904            us/op
JsonBenchmark.testEncode:testEncode·p0.9999          None  sample               61,941            us/op
JsonBenchmark.testEncode:testEncode·p1.00            None  sample             2854,912            us/op
JsonBenchmark.testDecode                          Jackson      ss           118276,270            us/op
JsonBenchmark.testDecode                      MinimalJson      ss             8362,015            us/op
JsonBenchmark.testDecode                         NanoJson      ss            21499,819            us/op
JsonBenchmark.testDecode                            Plain      ss             3517,849            us/op
JsonBenchmark.testDecode                             None      ss              100,837            us/op
JsonBenchmark.testEncode                          Jackson      ss           106437,109            us/op
JsonBenchmark.testEncode                      MinimalJson      ss            14163,948            us/op
JsonBenchmark.testEncode                         NanoJson      ss            20346,964            us/op
JsonBenchmark.testEncode                            Plain      ss             1746,693            us/op
JsonBenchmark.testEncode                             None      ss               40,539            us/op

Jackson and Plain mappers are comparable by average speed and throughput. So, can we maybe completely replace Jackson with own custom JsonReader/PlainObjectMapper implementation? WDYT? @robotdan

Code is here - https://github.com/volyx/fusionauth-jwt/tree/minimal-json

PlainObjectMapper - https://github.com/volyx/fusionauth-jwt/blob/minimal-json/src/main/java/io/fusionauth/jwt/json/PlainObjectMapper.java

NanoJsonObjectMapper - https://github.com/volyx/fusionauth-jwt/blob/minimal-json/src/main/java/io/fusionauth/jwt/json/NanoJsonObjectMapper.java

MinimalJsonObjectMapper - https://github.com/volyx/fusionauth-jwt/blob/minimal-json/src/main/java/io/fusionauth/jwt/json/MinimalJsonObjectMapper.java

@robotdan
Copy link
Member

Wow, thanks for the excellent and thorough suggestion @volyx

I'll take a look at you're proposal and follow up with comments.

@virtual-machinist
Copy link

Can this be pluggable (i.e. done via an interface)? While I do understand that not all projects use Jackson, it's de facto standard for anyone using Spring Boot (and possibly other popular web frameworks). Meaning that it wouldn't really add any extra dependencies. Pluggability may also help with writing simpler tests (possibly allow to test the validation and JSON parsing logic separately) and allow further Jackson mapper customization via the usual module/mixin/options route.

This will not be as easy as throwing away Jackson completely (you may have to do Mixins for serialization/deserialization to keep using Jackson annotations and do default ObjectMapper configuration), but avoids depending on yet-another-custom-json-parser that's only used here.

@volyx
Copy link
Author

volyx commented Mar 14, 2020

@virtual-machinist

it's de facto standard for anyone using Spring Boot (and possibly other popular web frameworks).

  • For spring projects common way of integration is separate module for example spring-fusionauth-jwt. So if we suppose that it is spring project we can manually configure JSON library.
  • For other frameworks it is also possible to write custom integration library - ${other_popular_web_framework}-fusionauth-jwt.

This will not be as easy as throwing away Jackson completely

Example is here - PlainObjectMapper. It is as fast as Jackson and uses just standard Java APIs.

I don't see any reason to support custom JSON serialization, if I can do encode/decode internally without any dependencies. PlainObjectMapper is fully compatible with ObjectMapper for all fusion-auth classes JWT, Header, Map, JSONWebKey. You can check it with:

// will encode with PlainObjectMapper, decode with ObjectMapper
./mvnw test -Dplain.json.encode=true

// will decode with PlainObjectMapper, encode with ObjectMapper
./mvnw test -Dplain.json.decode=true 

@volyx
Copy link
Author

volyx commented Mar 29, 2020

The same approach of embedding JSON serialization/deserialization code inside library is used in jose4j - https://bitbucket.org/b_c/jose4j/src/master/src/main/java/org/jose4j/json/

@virtual-machinist
Copy link

First of all I don't think speed is the main reason anyone uses Jackson (or Spring for that matter). Jackson and Gson provide a wide range of serialization/deserialization customization options, are well established, with decent documentation and have plenty integrations with other libraries.

As a developer I don't want to write glue logic just to interface with the embedded serializer of a yet-another-jwt-library with my Jackson object model when I have the option of not doing so if I use a library that natively supports Jackson, even if it's a little slower/bulkier/...
I think I may even lose any gained performance doing DTO-to-claim mapping vs configuring the serializer to write claims the way I need.

I'd like to emphasize that I'm not against having a custom-very-fast-json-library even as the default option, but please don't make it hard not using it.

@volyx
Copy link
Author

volyx commented Mar 30, 2020

@virtual-machinist I'm fully agree with you point. But what if you don't even need to think about the internal serialization/deserialization logic? PlainObjectMapper are going to be fully embedded into code, and will become an internal API. So you don't need to think about internals of JWT library.

Speed benchmarks are presented there only to show that the performance will not get worth after the replacement.

@virtual-machinist
Copy link

Sorry, @volyx , I cannot imagine a case when I'd like to add a private claim of my own object model to the token and not think about the serialization details. You're basically saying I won't have to worry about I/O because I won't have the option. 😉

I think this discussion is getting nowhere. I am for an agnostic model with first-class parser/generator access to JSON claims and header and I'm against hand-rolled JSON processing with the necessity of writing glue logic from my claims model. Especially if there's no obvious overhead of having Jackson, cause it's together with Spring.

If the model is truly agnostic, there's no problem writing a PlainObjectMapper support for it (or Gson, or json-smart, or JSR 374, or something else).

@robotdan , I'm sorry if this has already been asked - why public fields and not getters/setters?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants