Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance: rtk-query-codegen-openapi is slow when using multiple output files #4067

Open
d-mon- opened this issue Jan 12, 2024 · 0 comments · May be fixed by #4068
Open

Performance: rtk-query-codegen-openapi is slow when using multiple output files #4067

d-mon- opened this issue Jan 12, 2024 · 0 comments · May be fixed by #4068

Comments

@d-mon-
Copy link

d-mon- commented Jan 12, 2024

Issue

My team and I are currently facing some performance issue when using Multiple output files
Our swagger.json has more than 70k lines and we generate 100+ ouput files in our client with rtk-query-codegen-openapi. This result in our script taking roughly 18sec to execute.

Our config file looks a bit like this:

import type { ConfigFile } from '@rtk-query/codegen-openapi'

const config: ConfigFile = {
  schemaFile: '../server/swagger.json',
  apiFile: './src/store/api/baseApi.ts',
  apiImport: 'baseApi',
  argSuffix: 'Args',
  hooks: true,
  responseSuffix: 'Response',
  outputFiles: {
    './src/store/api/gen/agents.ts': {
      filterEndpoints: [/agentsController/],
    },
    './src/store/api/gen/tenants.ts': {
      filterEndpoints: [/tenantsController/],
    },
    // 100+ more ...
  },
}

export default config

Explanation

When running rtk-query-codegen-openapi, parseConfig will generate a list of config for each outputFile, triggering generateApi for each of them.
However, the current logic parse the spec (path of our swagger.json) at each execution of generateApi:

const v3Doc = await getV3Doc(spec);

This process takes a bit of time as you may expect, and becomes slower as our outputFiles grows

Current fix (patch)

I'm currently applying a patch on the file to cache the output of getV3Doc as follow:

diff --git a/node_modules/@rtk-query/codegen-openapi/lib/generate.js b/node_modules/@rtk-query/codegen-openapi/lib/generate.js
index bfcfed7..7e11f13 100644
--- a/node_modules/@rtk-query/codegen-openapi/lib/generate.js
+++ b/node_modules/@rtk-query/codegen-openapi/lib/generate.js
@@ -61,8 +61,11 @@ function getOverrides(operation, endpointOverrides) {
     return endpointOverrides === null || endpointOverrides === void 0 ? void 0 : endpointOverrides.find((override) => operationMatches(override.pattern)(operation));
 }
 exports.getOverrides = getOverrides;
+
+const cache = {}
 async function generateApi(spec, { apiFile, apiImport = 'api', exportName = 'enhancedApi', argSuffix = 'ApiArg', responseSuffix = 'ApiResponse', hooks = false, outputFile, isDataResponse = defaultIsDataResponse, filterEndpoints, endpointOverrides, }) {
-    const v3Doc = await (0, utils_1.getV3Doc)(spec);
+    cache[spec] = cache[spec] || await (0, utils_1.getV3Doc)(spec)
+    const v3Doc = cache[spec]
     const apiGen = new generate_1.default(v3Doc, {});
     const operationDefinitions = (0, utils_1.getOperationDefinitions)(v3Doc).filter(operationMatches(filterEndpoints));
     const resultFile = typescript_1.default.createSourceFile('someFileName.ts', '', typescript_1.default.ScriptTarget.Latest, 

This helped to cut the running time from 18 seconds to only 2 seconds without modifying our structure

Alternative approach

Single ouput file

We tried this approach already, but it creates some issues with typescript as you may expect due to the amount of information extracted in a single type.
Type instantiation is excessively deep and possibly infinite.

image

Multiple swagger files

Instead of generating a single swagger file, the other idea was to generate a swagger for each controller in our backend and run rtk-query-codegen-openapi on each one of them.
This is a more tedious task, but it should reduce it as the amount of information to parse would become small for each swagger files. However, this is not an ideal solution for us.

Action

If it is okay, I will prepare a PR to add the current fix to rtk-query-codegen-openapi

@d-mon- d-mon- linked a pull request Jan 12, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant