[dart-next] Refactor to decouple serialization from networking #15485

ahmednfwela · 2023-05-11T23:39:14Z

Introduce a new dart-next generator.

fixes #15477
as discussed in the above issue, this PR completely decouples the networking layer from the serialization/deserialization layer.
It also minimizes the serialization/networking layer's footprint, to simplify adding new serialization/networking options (like freezed #13047, or http)

It introduces new classes

TypeInfo - similar to built_value's FullType, but adapted to all serializers
SerializationRepositoryBase - the base class to handle all serialization operations in the API layer
BuiltValueJsonRepository - built_value's implementation of SerializationRepositoryBase
JsonSerializableRepository - json_serializable's implementation of SerializationRepositoryBase
a Raw version of each API that is decoupled from serialization/deserialization, and can handle any data format (not only json)

PR checklist

Write the repository implementations
Read the contribution guidelines.
Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
Run the following to build the project and update samples:
```
./mvnw clean package 
./bin/generate-samples.sh
./bin/utils/export_docs_generators.sh
```
Commit all changed files.
This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
These must match the expectations made by your contribution.
You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
For Windows users, please run the script in Git BASH.
In case you are adding a new generator, run the following additional script :
```
./bin/utils/ensure-up-to-date.sh
```
Commit all changed files.
File the PR against the correct branch: master (6.3.0) (minor release - breaking changes with fallbacks), 7.0.x (breaking changes without fallbacks)
If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

@jaumard (2018/09) @josh-burton (2019/12) @amondnet (2019/12) @sbu-WBT (2020/12) @kuhnroyal (2020/12) @agilob (2020/12) @ahmednfwela (2021/08) @bahag-chandrana

…nto separate directories WIP serialization repo implmentation

ahmednfwela · 2023-05-13T12:40:19Z

Update:

Introduced new CLI option serializationLibrary which takes either built_value or json_serializable
Replaced library CLI option to take either dio or http
I restructured the mustache files to decouple serialization and networking files
The mustache files currently look like this:
- dart2 (for the dart generator, unchanged)
- dart (for the dart-dio generator)
  - libraries
    - dio
    - http
  - serialization
    - built_value
    - json_serializable
  - shared files between all options ...

The plan is that after this PR, we split the dart generator files to be under the libraries/http folder and serialization/native
which will be done in #14346
until then, I copied over the files from dart2 folder to libraries/http folder

…to refactor-dart-dio

ahmednfwela · 2023-06-17T02:27:45Z

@kuhnroyal I have updated the PR to introduce a new generator dart-next

kuhnroyal · 2023-06-17T16:35:43Z

Yea, looks good.
You have to set generatorMetadata.stability(Stability.EXPERIMENTAL) and regenerate docs etc. - it will automatically mark it as experimental everywhere.

ahmednfwela · 2023-06-17T17:41:08Z

@kuhnroyal updated it.

kuhnroyal · 2023-06-17T19:27:56Z

I am on vacation for the next 2 weeks, so I might be able to check some code but can't run anything.

* use much newer exec plugin * run pub via dart sub command

kuhnroyal · 2023-07-03T11:39:34Z

modules/openapi-generator/src/main/resources/dart-next/libraries/dio/api.mustache

+{{#imports}}import '{{.}}';
+{{/imports}}
+
+part '{{classFilename}}.g.dart';


This currently fails when there are no inline enums

yes, I am aware of that, and I am working on a fix rn and completing the generator changes

Nice, ill see if I have time to test during the FlutterCon days.

@ahmednfwela Do you have a fix for this?

@kuhnroyal yes, but I am a little busy for a couple of days

Can I help here in anyway?

I am pushing a fix for it today , sorry for the delay

…to refactor-dart-dio

ahmednfwela · 2023-08-02T08:31:14Z

I pushed a fix for the part files, what's currently blocking the PR is that built_value doesn't support const constructors yet
see
google/built_value.dart#1249
google/built_value.dart#1250
google/built_collection.dart#164
google/built_collection.dart#283

Also some work needs to be done on the nullability areas.

kuhnroyal · 2023-08-08T20:07:29Z

I loosely followed the built_value tickets. I assume this is regarding default values? Can we move forward without those for now?

ahmednfwela · 2023-08-08T20:47:25Z

@kuhnroyal we can just use normal dart primitives for the api definations, and only use built_value for models

NotHolst · 2023-08-29T14:22:31Z

What is the point of supporting built list and built value?
I don't see what they offer over something like freezed or simply json_serializable
This is a genuine question, i'm very curious :)

ahmednfwela · 2023-08-29T16:04:24Z

@NotHolst well, built_value was the first immutability library to come out for dart, but now that dart has grown up, it's starting to fall behind.

one of its perks though is the separation of serializers and models into different classes, which allows a plugin style for serialization.

NotHolst · 2023-08-29T17:17:43Z

@ahmednfwela okay, sounds like a niche use for most of us who just want api codegeneration that works out of the box.
I would suggest replacing the default with either json_serializable or freezed whenever they are stable. Working with these extremely verbose data classes can be confusing to anyone who don't understand or even need the benefits :)

ahmednfwela · 2023-08-29T19:03:21Z

sounds like a niche use for most of us who just want api codegeneration that works out of the box.

@NotHolst I am honestly starting to go this way too, I have been arguing with built_value maintainers over adding const constructors for some time, and they don't seem to agree.

so I will be removing built_value support for the new generator and focus on other commonly used serialization packages, like json_serializable and freezed.

kuhnroyal · 2023-09-01T17:36:58Z

modules/openapi-generator/src/main/resources/dart-next/serialization/repository_base.mustache

+    const SerializationRepositoryBase();
+
+    FutureOr<Object?> serialize<T extends Object?>(T src, TypeInfo inputTypeInfo, {Object? context,});
+    FutureOr<T> deserialize<T extends Object?>(Object? value, TypeInfo targetTypeInfo, {Object? context,});


Making these async seems overkill and increases complexity in other places.
What is your use case here?

@kuhnroyal you might want to offset the (de)serialization process to another isolate like the dart generator does

I see. Does it make sense to provide both options so that we can use the sync access especially for parameter serialization etc.?

@kuhnroyal that was actually my plan since I sometimes need the serialization logic in places where I can't use async

kuhnroyal · 2023-09-01T21:14:06Z

samples/client/echo_api/dart/dio/built_value/lib/src/api/fake_api.dart

+    required EnumQueryDoubleEnum enumQueryDouble,
+    required BuiltList<ModelEnumClass> enumQueryModelArray,
+     BuiltList<InnerEnum> enumFormStringArray = r'$',
+     EnumFormStringEnum enumFormString = r'-efg',


Need to find a way to generate these defaults correctly.
And also collection default values.

with built_value, you can't since you need const constructors, so I plan on removing support for built_value entirely and focus on json_serializable/freezed

kuhnroyal · 2023-09-01T21:15:07Z

samples/client/echo_api/dart/dio/built_value/lib/src/api/fake_api.dart

+}
+
+
+class InnerEnum extends EnumClass {


This is missing its values, not sure why.

kuhnroyal · 2023-09-01T21:16:55Z

samples/client/echo_api/dart/dio/built_value/lib/src/model/fruit_any_of_disc.dart

+// ignore_for_file: unused_element
+import 'package:openapi/src/model/fruit_type.dart';
+import 'package:openapi/src/model/apple_any_of_disc.dart';
+import 'package:openapi/src/model/banana_any_of_disc.dart';


These 2 imports do not exist, something is wrong with the anyof generation.

NotHolst · 2023-10-18T12:29:51Z

any updates to this?
is there anything we can do to help?

NotHolst · 2023-11-16T13:17:28Z

any updates to this? is there anything we can do to help?

@ahmednfwela

ahmednfwela · 2023-11-17T12:42:00Z

@NotHolst Sorry I have been extremely busy lately with my professional work, that I don't have that much time to spare on opensource projects, I will however try to finish this PR since we will have to depend on it eventually.

NotHolst · 2023-12-19T13:07:23Z

@ahmednfwela

I'm following up on this a month later to make sure it doesn't go stale.
Is there anything I or anyone else can do to expedite this process.

I'm willing to put in the work, but i'll need a briefing on current progress, goals/milestones and potential roadblocks.

oravecz-jpmc · 2024-02-27T16:33:10Z

Also, would like to know what is outstanding to move this forward?

ahmednfwela added 2 commits May 12, 2023 01:28

Add initial refactoring

1c46afb

regenerate samples

7000179

ahmednfwela mentioned this pull request May 11, 2023

[REQ] [dart-dio] Refactor api methods and classes #15477

Open

completely decouple the networking and serialization mustache files i…

13b1186

…nto separate directories WIP serialization repo implmentation

ahmednfwela mentioned this pull request May 13, 2023

[dart-dio] [dart] Unify both dart generators into just dart #14346

Closed

5 tasks

fix typo

041ec5a

ahmednfwela added 13 commits May 13, 2023 14:41

gen samples

9ca6a7e

update tests to include new option

8d4f73c

merge test schemas

b7d7e76

merge test schema

72bd9d4

WIP

7b3715f

fix pom location

9b6e6aa

fix pom locations

7893bd6

Merge branch 'master' of github.com:OpenAPITools/openapi-generator in…

5e6d2de

…to refactor-dart-dio

WIP test improvements

c4b5c2b

don't exclude tests from analysis options

9956785

remove TODOs from test projects

34760b5

gen samples

3336772

WIP blocked by OpenAPITools#10189

244ca74

ahmednfwela mentioned this pull request May 15, 2023

Introduce changes necessary for Dart 3.0.0/Flutter 3.10.0 #15516

Merged

6 tasks

Merge branch 'master' of github.com:OpenAPITools/openapi-generator in…

93386c3

…to refactor-dart-dio

ahmednfwela mentioned this pull request Jun 16, 2023

[BUG][Dart][Dio] Deserialisation of Map<String, List<String>> fails with 'Unknown type on deserialization' #15863

Open

6 tasks

introduce dart-next

26034cb

ahmednfwela changed the title ~~[dart-dio] Refactor to decouple serialization from networking~~ [dart-next] Refactor to decouple serialization from networking Jun 17, 2023

ahmednfwela added 3 commits June 17, 2023 04:14

add back original files

580ce70

Add back original files

0196789

update pom for CI

64997ea

fix tests abd paths

2676a8b

fix pom paths

46401f3

mark dart-next as experimental

13ff9b9

kuhnroyal added 3 commits July 3, 2023 13:12

Regenerate generator readme

0f58020

Improve sample execution

2f2f528

* use much newer exec plugin * run pub via dart sub command

Nit picks

8e599cb

kuhnroyal reviewed Jul 3, 2023

View reviewed changes

ahmednfwela added 3 commits August 2, 2023 03:06

fix part generation

49b4983

Merge branch 'master' of github.com:OpenAPITools/openapi-generator in…

c9fb20e

…to refactor-dart-dio

async changes

f5236ef

This was referenced Aug 22, 2023

Feat: Add freezed serialization to dart-dio generator #13047

Open

[BUG] [DART] Dart Generator does not generate Multipart properly. Sends an empty object instead of file #14472

Open

kuhnroyal reviewed Sep 1, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dart-next] Refactor to decouple serialization from networking #15485

[dart-next] Refactor to decouple serialization from networking #15485

ahmednfwela commented May 11, 2023 •

edited

ahmednfwela commented May 13, 2023 •

edited

ahmednfwela commented Jun 17, 2023

kuhnroyal commented Jun 17, 2023

ahmednfwela commented Jun 17, 2023

kuhnroyal commented Jun 17, 2023

kuhnroyal Jul 3, 2023

ahmednfwela Jul 3, 2023

kuhnroyal Jul 3, 2023

kuhnroyal Jul 17, 2023

ahmednfwela Jul 17, 2023

kuhnroyal Aug 1, 2023

ahmednfwela Aug 1, 2023

ahmednfwela commented Aug 2, 2023 •

edited

kuhnroyal commented Aug 8, 2023

ahmednfwela commented Aug 8, 2023

NotHolst commented Aug 29, 2023 •

edited

ahmednfwela commented Aug 29, 2023

NotHolst commented Aug 29, 2023

ahmednfwela commented Aug 29, 2023

kuhnroyal Sep 1, 2023

ahmednfwela Sep 1, 2023

kuhnroyal Sep 1, 2023

ahmednfwela Sep 1, 2023

kuhnroyal Sep 1, 2023

ahmednfwela Sep 1, 2023

kuhnroyal Sep 1, 2023

kuhnroyal Sep 1, 2023

NotHolst commented Oct 18, 2023

NotHolst commented Nov 16, 2023

ahmednfwela commented Nov 17, 2023

NotHolst commented Dec 19, 2023 •

edited

oravecz-jpmc commented Feb 27, 2024

[dart-next] Refactor to decouple serialization from networking #15485

Are you sure you want to change the base?

[dart-next] Refactor to decouple serialization from networking #15485

Conversation

ahmednfwela commented May 11, 2023 • edited

PR checklist

ahmednfwela commented May 13, 2023 • edited

ahmednfwela commented Jun 17, 2023

kuhnroyal commented Jun 17, 2023

ahmednfwela commented Jun 17, 2023

kuhnroyal commented Jun 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmednfwela commented Aug 2, 2023 • edited

kuhnroyal commented Aug 8, 2023

ahmednfwela commented Aug 8, 2023

NotHolst commented Aug 29, 2023 • edited

ahmednfwela commented Aug 29, 2023

NotHolst commented Aug 29, 2023

ahmednfwela commented Aug 29, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NotHolst commented Oct 18, 2023

NotHolst commented Nov 16, 2023

ahmednfwela commented Nov 17, 2023

NotHolst commented Dec 19, 2023 • edited

oravecz-jpmc commented Feb 27, 2024

ahmednfwela commented May 11, 2023 •

edited

ahmednfwela commented May 13, 2023 •

edited

ahmednfwela commented Aug 2, 2023 •

edited

NotHolst commented Aug 29, 2023 •

edited

NotHolst commented Dec 19, 2023 •

edited