New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
api: output a single JSON array in REST pagination mode #7190
Conversation
When fetching N pages, avoid printing N separate JSON arrays to the output stream. Instead, massage the output so that the N pages of data are merged into a single JSON array. This is achieved by omitting the final `]` for the first page, and omitting the initial `[` for all subsequent pages.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's an elegant solution for REST, but I can't imagine how this might work for GraphQL where arrays are nested and may be numerous. Even if you kept track of scope e.g., JSON elements, where arrays are found, you still couldn't write directly to stdout.
With new APIs being added to GraphQL only (last I read), perhaps this is not an issue now if GraphQL usage is light but I imagine that will increase over time. That said, most of what users probably need to do is represented by REST anyway, and as long as those don't go away maybe this solves the 75% (or whatever) case.
Though it'd be ideal to make it just work for GraphQL, jq -s
or https://github.com/heaths/gh-merge-json can work around this (and I'm sure some other ways) can work around the problem if suddenly buffering responses for GraphQL requests is a concern.
Thanks for reviewing! I think it might be useful to solve paginated JSON outputs separately for REST (since that's relatively trivial, as showcased here) and for GraphQL, which I'm becoming more open to do your way, i.e. using buffering. |
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [aquaproj/aqua-registry](https://togithub.com/aquaproj/aqua-registry) | minor | `v4.20.0` -> `v4.21.1` | | [cli/cli](https://togithub.com/cli/cli) | minor | `v2.30.0` -> `v2.31.0` | | [weaveworks/eksctl](https://togithub.com/weaveworks/eksctl) | minor | `v0.145.0` -> `v0.146.0` | | [zellij-org/zellij](https://togithub.com/zellij-org/zellij) | patch | `v0.37.1` -> `v0.37.2` | --- ### Release Notes <details> <summary>aquaproj/aqua-registry</summary> ### [`v4.21.1`](https://togithub.com/aquaproj/aqua-registry/releases/tag/v4.21.1) [Compare Source](https://togithub.com/aquaproj/aqua-registry/compare/v4.21.0...v4.21.1) [Issues](https://togithub.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.21.1) | [Pull Requests](https://togithub.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.21.1) | aquaproj/aqua-registry@v4.21.0...v4.21.1 #### Fixes [#​13199](https://togithub.com/aquaproj/aqua-registry/issues/13199) kubernetes-sigs/kustomize: Follow up changes of kustomize v5.1.0 - [kubernetes-sigs/kustomize#5220 ### [`v4.21.0`](https://togithub.com/aquaproj/aqua-registry/releases/tag/v4.21.0) [Compare Source](https://togithub.com/aquaproj/aqua-registry/compare/v4.20.0...v4.21.0) [Issues](https://togithub.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.21.0) | [Pull Requests](https://togithub.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.21.0) | aquaproj/aqua-registry@v4.20.0...v4.21.0 #### 🎉 New Packages [#​13173](https://togithub.com/aquaproj/aqua-registry/issues/13173) [assetnote/surf](https://togithub.com/assetnote/surf): Escalate your SSRF vulnerabilities on Modern Cloud Environments. `surf` allows you to filter a list of hosts, returning a list of viable SSRF candidates [#​13174](https://togithub.com/aquaproj/aqua-registry/issues/13174) [iyear/tdl](https://togithub.com/iyear/tdl): A Telegram downloader written in Golang [#​13172](https://togithub.com/aquaproj/aqua-registry/issues/13172) [nikolaydubina/go-cover-treemap](https://togithub.com/nikolaydubina/go-cover-treemap): Go code coverage to SVG treemap [@​iwata](https://togithub.com/iwata) </details> <details> <summary>cli/cli</summary> ### [`v2.31.0`](https://togithub.com/cli/cli/releases/tag/v2.31.0): GitHub CLI 2.31.0 [Compare Source](https://togithub.com/cli/cli/compare/v2.30.0...v2.31.0) #### What's New - New suite of `project` commands for interacting with and manipulating projects. Huge shoutout 🥳 for the time and effort put into this work by [@​mntlty](https://togithub.com/mntlty) in [cli/cli#7375 [cli/cli#7578 - New `search code` command by [@​joshkraft](https://togithub.com/joshkraft) in [cli/cli#7376 - New `cs view` command by [@​dmgardiner25](https://togithub.com/dmgardiner25) in [cli/cli#7496 [cli/cli#7539 #### What's Changed - `api`: output a single JSON array in REST pagination mode by [@​mislav](https://togithub.com/mislav) in [cli/cli#7190 - `api`: support array params in GET queries by [@​mislav](https://togithub.com/mislav) in [cli/cli#7513 - `api`: force method to uppercase by [@​ffalor](https://togithub.com/ffalor) in [cli/cli#7514 - `alias`: Allow aliases to recognize extended commands by [@​srz-zumix](https://togithub.com/srz-zumix) in [cli/cli#7523 - `alias import`: Fix `--clobber` flag by [@​samcoe](https://togithub.com/samcoe) in [cli/cli#7569 - `run rerun`: Improve docs around `--job` flag by [@​williammartin](https://togithub.com/williammartin) in [cli/cli#7527 - `run view`: Support viewing logs for jobs with composite actions by [@​williammartin](https://togithub.com/williammartin) in [cli/cli#7526 - `gist edit`: Add selector option to `gist edit` command by [@​kousikmitra](https://togithub.com/kousikmitra) in [cli/cli#7537 - `repo clone`: Set upstream remote to track all branches after initial fetch by [@​samcoe](https://togithub.com/samcoe) in [cli/cli#7542 - `extension`: Speed up listing extensions by lazy-loading extension information when needed by [@​mislav](https://togithub.com/mislav) in [cli/cli#7493 - `auth`: Add timeouts to keyring operations by [@​samcoe](https://togithub.com/samcoe) in [cli/cli#7580 - `auth status`: write to stdout on success by [@​rajhawaldar](https://togithub.com/rajhawaldar) in [cli/cli#7540 - `completion`: Fix bash completions for extensions and aliases by [@​mislav](https://togithub.com/mislav) in [cli/cli#7525 - `issue/pr view`: alphabetically sort labels for `gh pr/issue view` by [@​ffalor](https://togithub.com/ffalor) in [cli/cli#7587 - Fix error handling for extension and shell alias commands by [@​samcoe](https://togithub.com/samcoe) in [cli/cli#7567 - Fix pkg imported more than once by [@​testwill](https://togithub.com/testwill) in [cli/cli#7591 - Refactor a nested if statement by [@​yanskun](https://togithub.com/yanskun) in [cli/cli#7596 - Fix a typo by [@​lerocknrolla](https://togithub.com/lerocknrolla) in [cli/cli#7557 - Fix flaky test by [@​samcoe](https://togithub.com/samcoe) in [cli/cli#7515 - Credential rotations, renames and decouplings from Mislav by [@​williammartin](https://togithub.com/williammartin) in [cli/cli#7544 - build(deps): bump github.com/cli/go-gh/v2 from 2.0.0 to 2.0.1 by [@​dependabot](https://togithub.com/dependabot) in [cli/cli#7546 - build(deps): bump github.com/AlecAivazis/survey/v2 from 2.3.6 to 2.3.7 by [@​dependabot](https://togithub.com/dependabot) in [cli/cli#7576 #### New Contributors - [@​srz-zumix](https://togithub.com/srz-zumix) made their first contribution in [cli/cli#7523 - [@​lerocknrolla](https://togithub.com/lerocknrolla) made their first contribution in [cli/cli#7557 - [@​testwill](https://togithub.com/testwill) made their first contribution in [cli/cli#7591 - [@​rajhawaldar](https://togithub.com/rajhawaldar) made their first contribution in [cli/cli#7540 **Full Changelog**: cli/cli@v2.30.0...v2.31.0 </details> <details> <summary>weaveworks/eksctl</summary> ### [`v0.146.0`](https://togithub.com/weaveworks/eksctl/releases/tag/v0.146.0): eksctl 0.146.0 (permalink) [Compare Source](https://togithub.com/weaveworks/eksctl/compare/0.145.0...0.146.0) ### Release v0.146.0 #### 🎯 Improvements - Update vpc-cni to 1.12.6 ([#​6692](https://togithub.com/weaveworks/eksctl/issues/6692)) #### 🧰 Maintenance - Bump dependencies ([#​6690](https://togithub.com/weaveworks/eksctl/issues/6690)) #### 📝 Documentation - New eksctl channel ([#​6697](https://togithub.com/weaveworks/eksctl/issues/6697)) #### Acknowledgments Weaveworks would like to sincerely thank: [@​wind0r](https://togithub.com/wind0r) </details> <details> <summary>zellij-org/zellij</summary> ### [`v0.37.2`](https://togithub.com/zellij-org/zellij/releases/tag/v0.37.2) [Compare Source](https://togithub.com/zellij-org/zellij/compare/v0.37.1...v0.37.2) This is a patch release to fix some minor issues in the previous release. #### What's Changed - hotfix: include theme files into binary by [@​jaeheonji](https://togithub.com/jaeheonji) in [zellij-org/zellij#2566 - fix(plugins): make hide_self api idempotent by [@​imsnif](https://togithub.com/imsnif) in [zellij-org/zellij#2568 **Full Changelog**: zellij-org/zellij@v0.37.1...v0.37.2 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/scottames/dots). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNS4xMzEuMCIsInVwZGF0ZWRJblZlciI6IjM1LjEzMS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
When fetching N pages, avoid printing N separate JSON arrays to the output stream. Instead, massage the output so that the N pages of data are merged into a single JSON array. This is achieved by omitting the final
]
for the first page, and omitting the initial[
for all subsequent pages.Fixes #1268 (for REST pagination)