Pagination breaks about 50k characters in due to missing comma

[amandapouget]

Describe the bug

gh version 1.4.0 (2020-12-15)

When getting multiple pages back, there is no comma between the pages, so the output is not JSON parseable, which makes it really difficult to work with.

Steps to reproduce the behavior

When executing a command like: gh api search/code?q=foobar --paginate, with example 571 code results (6 pages at max of 100 results per page), the output is not JSON parseable. It looks like this:
{"total_count:571, ....}{"total_count": ....}{"total_count": ....}{"total_count": ....}{"total_count": ....}{"total_count": ....}

Expected vs actual behavior

I expect to be able to grab the output to stdout coming from the process that runs gh api ... --paginate, and be able to run JSON.parse(stdout), but I can't, since the output is five objects concatinated into a string.

It would be better for --paginate to return a string of parseable JSON.

Logs

Unexpected token { at position 51235

[mislav]

Thanks for writing in! Merging all REST API responses in a single JSON document wouldn't be so hard, but it's harder when it comes to GraphQL responses where paginated collections are typically deeply nested. However, we still want to attempt to return only a single JSON document, and we're tracking it here #1268, so I will close this one as a duplicate.

As a workaround, you can parse the output of api --paginate using a JSON parser that accepts multiple concatenated JSON documents, for example the jq command-line tool.

[danielkza]

Why not at least split the documents with newlines, so it's a more manageable format for existing JSON-parsing tools?

[mislav]

@danielkza Existing JSON parser tools can either parse multiple JSON documents or they can not. The presence of a newline between the documents shouldn't be relevant or helpful to a parser.

[danielkza]

@mislav you can easily turn a parser that cannot parse multiple document into one that can by breaking lines, but you can't if there is no separator.

I would argue that "JSON lines" is a more easily handled format than "concatenated json documents", specially for shell and ad-hoc scripts.

[mislav]

@danielkza That's a good point about newlines. However, to better support JSON tools that can't part multiple documents I would rather put our efforts in #1268 than try to modify the current multi-document output to be easier for single-document parsers.