Firebase Storage Emulator crashes unexpectedly since v10.3.0

[DevSchmidtchen]

[REQUIRED] Environment info

firebase-tools: 10.4.0

Platform: macOS Monterey (12.3)

[REQUIRED] Test case

• Set up an app with storage emulator and import the existing emulator data (firebase emulators:start --import ./emulator_data)

[REQUIRED] Steps to reproduce

1. Start local emulators including storage emulator (firebase emulators:start --import ./emulator_data)2.
2. Open Emulator UI and go to storage tab

[REQUIRED] Expected behavior

The storage contents are shown and you can browse through them and upload new files. This works using version 10.2.2 of firebase emulator but since the update it doesn't work anymore.

[REQUIRED] Actual behavior

The emulator crashes.

[debug] [2022-03-18T09:40:32.246Z] Error: ENOENT: no such file or directory, open '/var/folders/pn/tjh55fz97z34qp9h8q6w72bw0000gn/T/firebase/storage/blobs/artistconnect-app.appspot.com%2Fwatermark.wav'
    at openSync (node:fs:585:3)
    at Persistence.readBytes (/Users/matti/.nvm/versions/node/v16.14.0/lib/node_modules/firebase-tools/lib/emulator/storage/persistence.js:37:36)
    at StorageLayer.getBytes (/Users/matti/.nvm/versions/node/v16.14.0/lib/node_modules/firebase-tools/lib/emulator/storage/files.js:98:38)
    at StorageLayer.handleGetObject (/Users/matti/.nvm/versions/node/v16.14.0/lib/node_modules/firebase-tools/lib/emulator/storage/files.js:83:49)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /Users/matti/.nvm/versions/node/v16.14.0/lib/node_modules/firebase-tools/lib/emulator/storage/apis/firebase.js:74:35
[error] 
[error] Error: An unexpected error has occurred.

[AlexVFornazieri]

Hey guys,

I have a similar issue here, in my case happen when clone project in a fresh new folder, in PC or in a pipeline CI/CD process. But in my case, not uploaded any files in my emulated storage eat. So the "storage_export/blob" and "storage_export/metadata" folder are empty and git don't track then. It cause a erro that don't find that directories in storage emulator startup.

To solve it for now, i create a "dont-delete-it.md" file inside that folders for git track and create in cloning, and all works fine.

[yuchenshi]

@DevSchmidtchen Anything you can share about what kind of files / workloads you're doing with the exports? It's hard to reproduce this without the specific files

@AlexVFornazieri That sounds like a separate issue and please open a new issue here. Also, feel free to send us a PR to ignore nonexistent empty folders Actually, that issue was reported as #3823 already -- thanks for confirming this and please subscribe there.

[charlesfries]

@yuchenshi I'm observing the same behavior. File type is image/png and it's imported into the emulator suite with the import flag:

firebase emulators:exec --import=./emulator-data "..."

The error only occurs when you try to request the file (either via the emulator UI or download URL).

Seeing the error on every release after 10.2.2.

Node 14.17.1
Windows 10

[charlesfries]

@yuchenshi Here is a reproduction: https://github.com/charlesfries/firebase-tools-repro

[tonyjhuang]

My guess is that this is because of #3647 which flattened the emulator directory structure and is not backwards-compatible with previously-exported emulator states prior to v10.3.0.

TBH we should have bumped the major version here to note this incompatibility but to help address current users' needs, we can build in backwards compatibility into the importer. @tohhsinpei Could you help with this issue?

[DibyodyutiMondal]

For me this happens even with a newer export...
This is what my firebase-export-metadata.json looks like

{
  "version": "10.4.2",
  "firestore": {
    "version": "1.13.1",
    "path": "firestore_export",
    "metadata_file": "firestore_export/firestore_export.overall_export_metadata"
  },
  "auth": {
    "version": "10.4.2",
    "path": "auth_export"
  },
  "storage": {
    "version": "10.4.2",
    "path": "storage_export"
  }
}

[DibyodyutiMondal]

As a temporary workaround, I manually removed the storage part of the export metadata json. Then I created a node server on localhost that, when requested on a certain endpoint, would read the all the files and metadata from the storage directory and upload them to the emulator instance. Essentially, this is now part of my local project environment startup ritual.

Interestingly, I was encountered with this error. But I have no storage triggers. I only have a single auth trigger in my functions directory. Is this connected?

!  functions: Firebase Storage function was not triggered due to emulation error. Please file a bug.
!  functions: Firebase Storage function was not triggered due to emulation error. Please file a bug.
!  functions: Firebase Storage function was not triggered due to emulation error. Please file a bug.
....

[tohhsinpei]

@DibyodyutiMondal Re: bug happening with a newer export, I'll put in an additional fix for it, thanks for pointing this out!

Edit: Actually, I believe the fix I put in in #4358 suffices. The reason the newer export also fails is because we needed to URI decode the file name, which I have done.