--paginate doesn't work well with /enterprises/{enterprise}/consumed-licenses and ?page=

[jessehouwing]

Describe the bug

Version:

gh --version
gh version 2.39.2 (2023-11-27)
https://github.com/cli/cli/releases/tag/v2.39.2

When running gh api "https://api.github.com/enterprises/{enterprise}/consumed-licenses" --paginate, it returns a broken json object:

{
   ... # page 1
}
{
   ... # page 2
}
...

So the user must loop through the pages themselves and concatenate the content like so:

$page = 1
do {
    $result =  (& gh api "https://api.github.com/enterprises/{enterprise}/consumed-licenses?per_page=100&page=$page" | ConvertFrom-Json)
    if ($result.users)
    {
        $knownEnterpriseUsers += $result.users
        $page = $page + 1
    }
} while ($result.users)

Steps to reproduce the behavior

1. Type this gh api "https://api.github.com/enterprises/{enterprise}/consumed-licenses" --paginate
2. Output:

(& gh api "https://api.github.com/enterprises/{enterprise}/consumed-licenses" --paginate | ConvertFrom-Json)
ConvertFrom-Json: Conversion from JSON failed with error: Additional text encountered after finished reading JSON content: {. Path '', line 1, position 72640.

Expected vs actual behavior

The content is correctly concatenated and a single json object is returned

Related: #6044

[jessehouwing]

Funny, if you add --jq '.users[]' it actually stiches the results together and gives me proper JSON.

[andyfeller]

@jessehouwing : I think you're experiencing a common behavior of gh api --paginate which is that we stream the output rather than aggregating all of it into a single response back. #1268 is an issue we're tracking about how we can improve that experience.

In the mean time, you would need to leverage jq -s to slurp up all of the fragments depending on your use case:

# should produce a single JSON array of all paginated result concatenated:
gh api graphql -f query='QUERY' --paginate --jq '.data.repository.branchProtectionRules.nodes[]' | jq -s

That said, passing page and per_page to the gh api call works as expected:

GH_DEBUG=api gh api /enterprises/{enterprise}/consumed-licenses --paginate -F page=2 -X GET -F per_page=10

[jessehouwing]

That's quite unexpected. But reading through it, I understand the complexity. Relying on the presence of jq is indeed not obvious, since it's not a common utility on windows. The fact that there is a --jq argument would immediately lead me to think I should be able to solve it there...

[andyfeller]

That's quite unexpected. But reading through it, I understand the complexity. Relying on the presence of jq is indeed not obvious, since it's not a common utility on windows. The fact that there is a --jq argument would immediately lead me to think I should be able to solve it there...

That is a fair assessment.

Is there anything you would suggest we do outside of the larger work in #1268 for the short term to avoid the confusion?

For example, what might you expect to see in the gh api long description about the nature of the command and this workaround to make it more apparent?

[jessehouwing]

If it could add a separator somehow it would become much easier to split the resulting text into parts so that other scripting languages can more easily work with the result sets.

But for now I'd expect this behavior to be documented here:

gh help formatting

And it mentions nothing of the sort. Many parts of gh return json objects, so the fact that some commands return malformed json by design is an interesting choice. to me I started with a small dataset, so the issue only started to appear after the dataset grew. Quite frustrating ;).

Ideally gh would register a powershell alias that would auto split the pages into pipeline objects so that it's transparent to PowerShell that this behavior exists.

[jessehouwing]

Copilot suggests doing something akin to:

$text = '{"key1": "value1"}{"key2": "value2"}'
$splitJson = $text -split '(?<=\})(?={)'

foreach($json in $splitJson){
     $parsedJson = $json | ConvertFrom-Json
    # Now you can operate on the $parsedJson object
}

which could probably be

(gh api --paginate /something) -split '(?<=\})\s*(?={)' | ConvertFrom-Json | # now there are multiple objects

[andyfeller]

return malformed json by design

I think the way to look at it is that gh api --paginate is returning a stream of valid JSON as it is processing bucket of data rather than a single aggregate response. You say malformed which isn't technically correct.

[jessehouwing]

True. Agreed. Yet there are few tools that will be able to parse the output by default. To them it's not something they understand.

[samcoe]

Having read through this I am fairly confident that this issue is the same as #1268 which is potentially being addressed in #5652. I am going to close this out as a duplicate, but please let me know if you think that is in error.