Storage Emulator does not support createReadStream()

[sceee]

[REQUIRED] Environment info

firebase-tools:v9.12.1

Platform: Windows 10

[REQUIRED] Test case

Have a Cloud Function with the following code (this just accesses and downloads a file):

import * as functions from 'firebase-functions'
import * as admin from 'firebase-admin'
import * as path from 'path'
import * as os from 'os'

...

  const filepath = 'path/to/file'
  const fileName = `tmpfile_${request.fileID}`

  const bucket = storage.bucket()
  const tempFilePath = path.join(os.tmpdir(), fileName)
  const fileRef = bucket.file(filepath)

  await fileRef.download({ destination: tempFilePath })
  return Promise.resolve()

The same occurs with the following code:

  const filepath = 'path/to/file'
  const bucket = storage.bucket()
  const fileRef = bucket.file(filepath)
  const readStream = fileRef.createReadStream()

  readStream.pipe(response)

[REQUIRED] Steps to reproduce

1. Upload a file to the specified location in Storage Emulator
2. Execute the Cloud Function code from above to initiate a download from the Storage emulator
3. See the error below in the log

[REQUIRED] Expected behavior

Downloading files in Storage works with Storage emulator as it works with production Firebase service.

[REQUIRED] Actual behavior

The following error is logged

>  events.js:292
>        throw er; // Unhandled 'error' event
>        ^
> 
>  RequestError: The downloaded data did not match the data from the server. To be sure the content is the same, you should download the file again.
>      at PassThrough.onComplete (C:\proj\functions\node_modules\@google-cloud\storage\build\src\file.js:1001:43)
>      at processTicksAndRejections (internal/process/task_queues.js:93:5)
>  Emitted 'error' event on PassThrough instance at:
>      at emitErrorNT (internal/streams/destroy.js:106:8)
>      at emitErrorCloseNT (internal/streams/destroy.js:74:3)
>      at processTicksAndRejections (internal/process/task_queues.js:80:21) {

code: CONTENT_DOWNLOAD_MISMATCH

I can say that the same code works in production with the real Storage service without this error.

[abeisgoat]

I suspect we don't implement the endpoint for createReadStream() at all or we do, but we don't respect the chunked request and/or range headers. I'll have to do some research.

[samtstern]

#3486 seems to be another report of this same issue.

[adzharamrullah]

@sceee I found the same issue, and finally i tried to add optional param validation: false to the API download.

Example

...
await fileRef.download({ 
  destination: tempFilePath,
  validation: false,
})
...

it seems to work for me on emulator. but on production i remove that. I thought of doing that for a while before it was officially fixed.

[sceee]

@adzharamrullah thanks. This could be a temporary workaround for download() but for pipe(), it unfortunately does not work as there is no such option available.

[ghivert]

I had the exact same problem, and this code:

const sendPicture = async (
  res: functions.Response,
  uid: string,
  id: string
) => {
  const file = pictures.getFile(uid, id) // This is getting my file by generating correct path and connecting to bucket.
  const [metadata] = await file.getMetadata()
  res.set('Cache-Control', 'public, max-age=3600, s-maxage=3600')
  res.type(metadata.contentType)
  const validation = process.env.NODE_ENV === 'development' ? false : 'crc32c'
  file.createReadStream({ validation }).pipe(res)
}

Is working both in production and development when manually setting NODE_ENV to development. I struggled for few hours and checked many times in production to ensure everything is correct.

[dwelle]

Note that this is a recent regression (not sure how recent). It worked well in 9.1.2 (at least the .download() part).

[DKorosec]

Having same problems on firebase-tools 10.1.0