Does PowerShell support the real "Stream" response?

[chenxizhang]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

I am writing a PowerShell module for OpenAI (and Azure OpenAI or other GPT services), they are supporting the "stream" mode of chat completions (https://platform.openai.com/docs/api-reference/chat/create). In other languages, for example, Python. we can easily get the response by some code below.

# Example of an OpenAI ChatCompletion request with stream=True
# https://platform.openai.com/docs/api-reference/streaming#chat/create-stream

# a ChatCompletion request
response = client.chat.completions.create(
    model='gpt-3.5-turbo',
    messages=[
        {'role': 'user', 'content': "What's 1+1? Answer in one word."}
    ],
    temperature=0,
    stream=True  # this time, we set stream=True
)

for chunk in response:
    print(chunk)
    print(chunk.choices[0].delta.content)
    print("****************")

But I didn't find a similar way to implement this in PowerShell, instead I need to write a lot of code like below.

  $client = New-Object System.Net.Http.HttpClient
  $body = $params.Body
  Write-Verbose "body: $body"

  $request = [System.Net.Http.HttpRequestMessage]::new()
  $request.Method = "POST"
  $request.RequestUri = $params.Uri
  $request.Headers.Clear()
  $request.Content = [System.Net.Http.StringContent]::new(($body), [System.Text.Encoding]::UTF8)
  $request.Content.Headers.Clear()
  $request.Content.Headers.Add("Content-Type", "application/json;chatset=utf-8")

  foreach ($k in $header.Keys) {
      $request.Headers.Add($k, $header[$k])
  }
                  
  $task = $client.Send($request)
  $response = $task.Content.ReadAsStream()
  $reader = [System.IO.StreamReader]::new($response)
  $result = "" # message from the api
  $firstChunk = $true
  while ($true) {
      $line = $reader.ReadLine()
      if (($line -eq $null) -or ($line -eq "data: [DONE]")) { break }
      $chunk = ($line -replace "data: ", "" | ConvertFrom-Json).choices.delta.content
      if ($firstChunk) {
          $firstChunk = $false
          Write-Host "`r[$current] " -NoNewline -ForegroundColor Red
      }
      Write-Host $chunk -NoNewline -ForegroundColor Green
      $result += $chunk
      Start-Sleep -Milliseconds 5
  }

  Write-Host ""
  $reader.Close()
  $reader.Dispose()

  $messages += [PSCustomObject]@{
      role    = "assistant"
      content = $result
  }

Expected behavior

I want to read the chunk one by one.

Actual behavior

It seems like PowerShell supports the stream mode, but it will response to me until it read all the chunks.

Error details

No response

Environment data

PS C:\home> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.4.2
PSEdition                      Core
GitCommitId                    7.4.2
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

[rhubarb-geek-nz]

Hi, if the existing Invoke-WebRequest and Invoke-RestMethod don't support the response mechanisms or formats that you need and you find yourself repeating the same blocks of .NET method calls to the SDK, then I would recommend building a cmdlet in C# that does all the networking API and then streams the response to the pipeline output. Then you can have PowerShell functions or scriptblocks that are fed from the output pipeline of the cmdlet. I suggest that is the closest you might get to a stream in PowerShell.

[MatejKafka]

I don't believe Invoke-WebRequest supports streaming of the response, since it returns a complete response object, so you'll have to implement it yourself, either as a C# binary cmdlet, or the way you do now.

As an aside, it might be a worthwhile addition to add a -Stream parameter to the web cmdlets, which stream the raw response as it comes in.

[rhubarb-geek-nz]

As an aside, it might be a worthwhile addition to add a -Stream parameter to the web cmdlets, which stream the raw response as it comes in.

I suggest a proof of concept would be needed for this. Essentially you would need the HTTPResponse without the content written to the output pipeline, followed by the chunks of the response as byte []. The stage following the pipeline will need to deal with the chunks of arbitrary size. I am hard pressed to think of an example where this complexity would be appropriate with the PowerShell programming model and level of abstraction.

[MatejKafka]

As an aside, it might be a worthwhile addition to add a -Stream parameter to the web cmdlets, which stream the raw response as it comes in.

I suggest a proof of concept would be needed for this. Essentially you would need the HTTPResponse without the content written to the output pipeline, followed by the chunks of the response as byte []. The stage following the pipeline will need to deal with the chunks of arbitrary size. I am hard pressed to think of an example where this complexity would be appropriate with the PowerShell programming model and level of abstraction.

Wouldn't send both the response and the content, that would be very annoying to process. Just pass through the string segments (or byte[]) as they come, or maybe split the string line-by-line for consistency.

[rhubarb-geek-nz]

Wouldn't send both the response and the content, that would be very annoying to process. Just pass through the string segments (or byte[]) as they come, or maybe split the string line-by-line for consistency.

Absolutely, hence why a proof of concept to show how it would work. I don't think it practical for the cmdlet to be doing much processing given it has no knowledge of the format of the content other than the read chunk sizes from the response. There are many text formats which have no need for any line splitting (eg JSON and XML) and the lines may not align with the chunks read from the response, a chunk may end mid-line. It can also be ambiguous to detect whether the response is a text or binary format. There may be important information in the headers that the caller still needs, along with the status code.