Storage emulator fixes and added rewriteTo support #3647

rhodgkins · 2021-08-06T09:25:22Z

Description

Downloading (using `File#download()`)

When using the emulator against @google-cloud/storage, I noticed a few of the headers were missing, this mainly caused any validation on download to fail (as the emulated endpoint was missing the x-goog-hash header).

I've included some values for the other x-goog headers.

Support for `rewriteTo` / `copyTo`

This is so File#copy() is supported.

The same endpoint/logic is used for both copyTo and rewriteTo.

For copyTo it works as is - just creates a new file based on the source files data/metadata.

The same thing is done for rewriteTo, which to the end user (calling #copy()) is fine, it doesn't handle the complexity of the rewriteToken. If the rewriteTo endpoint is called with a rewriteToken a 501 status code is returned (I can't see (at least for the moment!) any issues with that as the copying is done in one go).

Flattened object storage for buckets

GCS doesn't really have directories so in the emulator if you store a file at filename and then try to store another file at filename/child.txt it would fail. I've update the Persistence to store files for a bucket in a single directory, but URL encoded the filenames.

Fixed `Objects: list` endpoint

This didn't work properly at all (and it didn't return prefixes), so added some test cases and updated the logic to match a real life objects: list on a bucket.
Also added basic support for pageToken and maxResults.

Scenarios Tested

See updated / added unit tests.

Sample Commands

See comments - I've given allUsers the objects.list and objects.get permissions for the bookcreator-dev-firetool-tests bucket given in the examples.

Linked issues

Fixes #3469
Fixes #3751
Fixes #3778
Fixes #3695
Fixes #3764

rhodgkins · 2021-08-06T09:32:42Z

src/emulator/storage/apis/gcloud.ts

+  res.setHeader("x-goog-generation", `${md.generation}`);
+  res.setHeader("x-goog-metadatageneration", `${md.metageneration}`);
+  res.setHeader("x-goog-storage-class", md.storageClass);
+  res.setHeader("x-goog-hash", `crc32c=${crc32cToString(md.crc32c)},md5=${md.md5Hash}`);


I've matched these extra headers to a real file:

curl -I 'https://storage.googleapis.com/storage/v1/b/bookcreator-dev-firetool-tests/o/testing%2Fshoveler.svg?alt=media' HTTP/2 200 x-guploader-uploadid: ADPycdtb0wlKL1b531e7pcvB5BwHSRnLmGtHmc9rd3A2lpjNiGegBft5t_ClHASXwONq7hhh2f6n7WHs8fp4q97YweY etag: CKf07Ir4o/ICEAE= x-goog-hash: crc32c=t0Xs5w==,md5=Z0RBlgyhui3gitTlDJ/emA== content-type: application/json x-goog-generation: 1628512034961959 x-goog-metageneration: 1 x-goog-storage-class: STANDARD content-disposition: attachment date: Mon, 09 Aug 2021 15:26:16 GMT vary: Origin vary: X-Origin cache-control: no-cache, no-store, max-age=0, must-revalidate expires: Mon, 01 Jan 1990 00:00:00 GMT pragma: no-cache content-length: 5 server: UploadServer alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

rhodgkins · 2021-08-06T09:34:10Z

src/emulator/storage/metadata.ts

+
+function generateETag(bytes: Buffer): string {
+  const hash = crypto.createHash("sha1");
+  hash.update(bytes);
+  // Trim padding
+  return hash.digest("base64").slice(0, -1);
+}


Not really sure how GCS calculates these, but I guess it doesn't matter as long as its consistent for the emulator?

For now I've based it off the etag package.

GCS ensures etags are unique across resources and versions of those resources. If you simply hash the bytes, you guarantee collisions between objects with the same file content.

The generation metadata field is guaranteed to be unique across objects in GCS. It seems like that mostly holds true in the emulator today. I would recommend hashing the generation and metageneration fields here instead of the file contents.

I can use a combination of those 2 things, but as the generation is based on the current timestamp, it might not be always be unique in the emulator context - it's a small possibility but could happen.

What about a combination of name, generation, metageneration and file contents?

Ah sorry I think I misread unique across it objects in GCS as unique across GCS 😅
Combo of generation and metageneration sounds good 👍

Done - etag now based on generation/metadatageneration

scripts/storage-emulator-integration/tests.ts

rhodgkins · 2021-08-09T15:21:03Z

src/emulator/storage/apis/gcloud.ts

+function sendObjectNotFound(req: Request, res: Response): void {
+  res.status(404);
+  const message = `No such object: ${req.params.bucketId}/${req.params.objectId}`;
+  if (req.method === "GET" && req.query.alt == "media") {
+    res.send(message);
+  } else {
+    res.json({
+      error: {
+        code: 404,
+        message,
+        errors: [
+          {
+            message,
+            domain: "global",
+            reason: "notFound",
+          },
+        ],
+      },
+    });
+  }
+}


Tested with (GET with alt=media):

curl -i 'https://storage.googleapis.com/storage/v1/b/bookcreator-dev-firetool-tests/o/blah?alt=media' HTTP/2 404 x-guploader-uploadid: ADPycdst7ZSZzN7H2Yw38zD4PTx9Zu6wMutI4cgUbUWHUS366qbxzy8tklhFeSs9HeMXJj9-JWuFLhxj8cQ9X8GT88LqNALIfA content-type: text/html; charset=UTF-8 date: Mon, 09 Aug 2021 12:22:42 GMT vary: Origin vary: X-Origin expires: Mon, 09 Aug 2021 12:22:42 GMT cache-control: private, max-age=0 content-length: 51 server: UploadServer alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43" No such object: bookcreator-dev-firetool-tests/blah

Tested with (GET without alt=media):

curl -i 'https://storage.googleapis.com/storage/v1/b/bookcreator-dev-firetool-tests/o/blah' HTTP/2 404 x-guploader-uploadid: ADPycdvIQMxtmw4ACLIf5rrG3GAMTQY7ST_w_BAZtGCGGke00EPB1bca_SAuLCeTYG9UeDx_625W-N78NwaiUtKPIjtWPU_mdw content-type: application/json; charset=UTF-8 date: Mon, 09 Aug 2021 12:23:29 GMT vary: Origin vary: X-Origin cache-control: no-cache, no-store, max-age=0, must-revalidate expires: Mon, 01 Jan 1990 00:00:00 GMT pragma: no-cache content-length: 277 server: UploadServer alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43" { "error": { "code": 404, "message": "No such object: bookcreator-dev-firetool-tests/blah", "errors": [ { "message": "No such object: bookcreator-dev-firetool-tests/blah", "domain": "global", "reason": "notFound" } ] } }

And (non-GET):

curl -i 'https://storage.googleapis.com/storage/v1/b/bookcreator-dev-firetool-tests/o/blah' -X DELETE HTTP/2 404 x-guploader-uploadid: ADPycdt_7t20l4J1rLO0hosF99ICr446vfIK8TBzbt0cf9nQ_eO-0Rlsc_pegIV8VzcCJd3KcYgco2gQjKnOjrzk9P1vN0PN8A content-type: application/json; charset=UTF-8 date: Mon, 09 Aug 2021 12:22:57 GMT vary: Origin vary: X-Origin cache-control: no-cache, no-store, max-age=0, must-revalidate expires: Mon, 01 Jan 1990 00:00:00 GMT pragma: no-cache content-length: 277 server: UploadServer alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43" { "error": { "code": 404, "message": "No such object: bookcreator-dev-firetool-tests/blah", "errors": [ { "message": "No such object: bookcreator-dev-firetool-tests/blah", "domain": "global", "reason": "notFound" } ] } }

rhodgkins · 2021-08-10T10:13:24Z

Test cases for @google-cloud/storages Bucket#getFiles based on querying against real life bucket with same files:

curl -i 'https://storage.googleapis.com/storage/v1/b/bookcreator-dev-firetool-tests/o'

rhodgkins · 2021-08-10T10:30:02Z

The integration tests are only being run for pushes - is there a reason why?

firebase-tools/.github/workflows/node-test.yml

Line 61 in e38d25c

if: github.event_name == 'push'

rhodgkins · 2021-08-11T10:40:35Z

scripts/storage-emulator-integration/tests.ts

+        it("should set null custom metadata values to empty strings", async () => {
+          const [, source] = await testBucket.upload(smallFilePath);
+
+          const file = testBucket.file(copyDestinationFile);
+          const metadata = { foo: "bar", nullMetadata: null };
+          const cacheControl = "private,max-age=10,immutable";
+          // Types for CopyOptions are wrong (@google-cloud/storage sub-dependency needs
+          // update to include https://github.com/googleapis/nodejs-storage/pull/1406
+          // and https://github.com/googleapis/nodejs-storage/pull/1426)
+          const copyOpts: CopyOptions & { [key: string]: unknown } = {
+            metadata,
+            cacheControl,
+          };
+          const [, { resource: metadata1 }] = await testBucket
+            .file(smallFilePath.split("/").slice(-1)[0])
+            .copy(file, copyOpts);
+
+          expect(metadata1).to.deep.include({
+            bucket: source.bucket,
+            contentType: source.contentType,
+            etag: source.etag,
+            crc32c: source.crc32c,
+            metadata: {
+              foo: "bar",
+              // Sets null metadata values to empty strings
+              nullMetadata: "",
+            },
+            cacheControl,
+          });
+
+          // Also double check with a new metadata fetch
+          const [metadata2] = await file.getMetadata();
+          expect(metadata2).to.deep.equal(metadata1);
+        });


This seems weird behaviour but can be confirmed with:

curl -i 'https://storage.googleapis.com/storage/v1/b/bookcreator-dev-firetool-tests/o/prefix/rewriteTo/b/bookcreator-dev-firetool-tests/o/copied' -H "authorization: Bearer $(gcloud auth print-access-token)" -H 'content-type: application/json' --data-raw '{ "metadata": { "hello": "world", "foo": null } }' HTTP/2 200 x-guploader-uploadid: ADPycdtAGoz55T_cY81t5UnGT_MZlTnOeXiBSkCyL6ydj5-1yIPxz4zkg8dPzdIkjZww3q5eKzJN7n_nnVpii6K-iw content-type: application/json; charset=UTF-8 date: Wed, 11 Aug 2021 08:13:45 GMT vary: Origin vary: X-Origin cache-control: no-cache, no-store, max-age=0, must-revalidate expires: Mon, 01 Jan 1990 00:00:00 GMT pragma: no-cache content-length: 974 server: UploadServer alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43" { "kind": "storage#rewriteResponse", "totalBytesRewritten": "5", "objectSize": "5", "done": true, "resource": { "kind": "storage#object", "id": "bookcreator-dev-firetool-tests/copied/1628669625765894", "selfLink": "https://www.googleapis.com/storage/v1/b/bookcreator-dev-firetool-tests/o/copied", "mediaLink": "https://storage.googleapis.com/download/storage/v1/b/bookcreator-dev-firetool-tests/o/copied?generation=1628669625765894&alt=media", "name": "copied", "bucket": "bookcreator-dev-firetool-tests", "generation": "1628669625765894", "metageneration": "1", "storageClass": "STANDARD", "size": "5", "md5Hash": "Z0RBlgyhui3gitTlDJ/emA==", "crc32c": "t0Xs5w==", "etag": "CIaggJTDqPICEAE=", "timeCreated": "2021-08-11T08:13:45.767Z", "updated": "2021-08-11T08:13:45.767Z", "timeStorageClassUpdated": "2021-08-11T08:13:45.767Z", "metadata": { "hello": "world", "foo": "" } } }

(Confirmed same behaviour with copyTo.)

I believe this is correct, yes.

rhodgkins · 2021-08-31T15:28:05Z

Sorry to pester @inlined / @taeold / @mbleigh but is there anyone who can have a look over this please?

inlined · 2021-09-02T18:12:30Z

Thanks for tagging me. I'll poke around internally to see if I can find a good reviewer for you.

abeisgoat · 2021-09-02T18:50:30Z

Wow awesome work! Sorry for having this slip under my radar. It's a lot to digest, but I'll dig through and send thoughts!

rhodgkins · 2021-09-14T08:06:36Z

Thanks @abeisgoat, let me know if you want me to rebase the latest changes from master

rhodgkins · 2021-10-25T08:20:57Z

Bump @abeisgoat & @inlined

src/emulator/storage/files.ts

tonyjhuang · 2022-03-14T15:13:38Z

I'm checking in with @rachelmyers to see if she needs to give a pass. @sam-gc's approval is optional but please hold off on merging until Rachel chimes in, thanks.

tonyjhuang · 2022-03-14T17:01:37Z

@rhodgkins Looks like you're good to go on approvals, thanks again for the high quality PR! Apologies for the very, very long turnaround time. We appreciate you volunteering your time to improve the Firebase Storage emulator!

rhodgkins · 2022-03-14T17:06:15Z

Thanks! Look forward to the release containing this 👍 (I don't have permission to merge this in btw - thought I'd mention as it sounds like, from your comments, that I can click that button!)

sam-gc · 2022-03-14T17:08:39Z

Looks like it can't merge at the moment because this branch is outdated. May need a quick merge/rebase

rhodgkins · 2022-03-14T17:29:57Z

@sam-gc / @tonyjhuang done

tonyjhuang · 2022-03-14T20:19:12Z

@rhodgkins Looks like an integration test failed. Please fix.

rhodgkins · 2022-03-15T08:53:43Z

Fixed - tested locally with npm run clean && npm run test:triggers-end-to-end

rhodgkins · 2022-03-15T09:08:16Z

I've resolved the conflicts too but ping me if there's more by the time you come back to this. (Some what frustrating having to resolve conflicts in a simple CHANGELOG each time 😬)

tonyjhuang · 2022-03-15T15:27:14Z

Sounds good, running integration tests now.

tonyjhuang · 2022-03-15T17:22:23Z

And it's in, thanks again for the high quality contribution, @rhodgkins 🥳

rhodgkins · 2022-03-15T17:48:04Z

~~Thanks @tonyjhuang - any ideas on when they'll be a next a release?~~ ignore me! Thanks again!

tonyjhuang · 2022-03-15T19:05:12Z

Yes I believe this should be included in https://github.com/firebase/firebase-tools/releases/tag/v10.3.0 which we just made the cut off for :)

google-cla bot added the cla: yes Manual indication that this has passed CLA. label Aug 6, 2021

rhodgkins changed the title ~~Emulator fixes~~ Storage emulator fixes Aug 6, 2021

rhodgkins commented Aug 6, 2021

View reviewed changes

rhodgkins force-pushed the emulator-fixes branch from 44b7583 to 3ce3fdc Compare August 6, 2021 11:25

rhodgkins changed the title ~~Storage emulator fixes~~ Storage emulator fixes and added rewriteTo support Aug 6, 2021

rhodgkins commented Aug 6, 2021

View reviewed changes

scripts/storage-emulator-integration/tests.ts Show resolved Hide resolved

rhodgkins force-pushed the emulator-fixes branch from 331fc3a to 3bc428a Compare August 9, 2021 12:21

rhodgkins commented Aug 9, 2021

View reviewed changes

rhodgkins force-pushed the emulator-fixes branch from e7c7f8c to fbc9c07 Compare August 9, 2021 15:44

rhodgkins force-pushed the emulator-fixes branch from 7134fff to 035ff8d Compare August 10, 2021 10:18

rhodgkins force-pushed the emulator-fixes branch from 33febe2 to a0fba75 Compare August 11, 2021 08:18

rhodgkins commented Aug 11, 2021

View reviewed changes

rhodgkins force-pushed the emulator-fixes branch 4 times, most recently from 8f00187 to 7165bdc Compare August 19, 2021 07:38

rhodgkins force-pushed the emulator-fixes branch 2 times, most recently from fb04288 to 2d030f7 Compare August 31, 2021 15:13

abeisgoat self-requested a review September 2, 2021 18:50

rhodgkins force-pushed the emulator-fixes branch from 2d030f7 to f397dbe Compare November 3, 2021 11:40

This was referenced Nov 3, 2021

Admin SDK getFiles() doesn't work on emulator #3778

Closed

file.exists() is always true #3764

Closed

rhodgkins force-pushed the emulator-fixes branch from bd664c6 to 140be80 Compare March 14, 2022 15:02

tonyjhuang reviewed Mar 14, 2022

View reviewed changes

src/emulator/storage/files.ts Outdated Show resolved Hide resolved

Comment on boolean argument

3627742

rachelmyers removed their request for review March 14, 2022 15:42

sam-gc approved these changes Mar 14, 2022

View reviewed changes

Merge branch 'master' into emulator-fixes

b4f1457

tonyjhuang enabled auto-merge (squash) March 14, 2022 17:37

tonyjhuang mentioned this pull request Mar 14, 2022

Emulators start with --import fails whether storage_export/blobs or storage_export/metadata empty folders do not exist #3823

Closed

tonyjhuang linked an issue Mar 14, 2022 that may be closed by this pull request

Emulators start with --import fails whether storage_export/blobs or storage_export/metadata empty folders do not exist #3823

Closed

Fixed end to end test storage import/export

8b3c692

auto-merge was automatically disabled March 15, 2022 08:53
Head branch was pushed to by a user without write access

Merge branch 'master' into emulator-fixes

efa57d4

tonyjhuang merged commit 9795181 into firebase:master Mar 15, 2022

This was referenced Mar 17, 2022

Fix storage emulator tests firebase/firebase-tools-ui#727

Merged

Firebase Storage Emulator crashes unexpectedly since v10.3.0 #4326

Closed

tohhsinpei mentioned this pull request Mar 24, 2022

Fix issue with importing Storage Emulator data with nested directory structure #4358

Merged

tohhsinpei mentioned this pull request Apr 6, 2022

Fix cross-platform incompatibility with Storage Emulator exports #4411

Merged

rhodgkins deleted the emulator-fixes branch November 17, 2022 12:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage emulator fixes and added rewriteTo support #3647

Storage emulator fixes and added rewriteTo support #3647

rhodgkins commented Aug 6, 2021 •

edited

rhodgkins Aug 6, 2021 •

edited

rhodgkins Aug 6, 2021

tonyjhuang Mar 10, 2022

rhodgkins Mar 10, 2022

rhodgkins Mar 10, 2022

rhodgkins Mar 11, 2022

rhodgkins Aug 9, 2021

rhodgkins commented Aug 10, 2021 •

edited

rhodgkins commented Aug 10, 2021

rhodgkins Aug 11, 2021

abeisgoat Jan 5, 2022

rhodgkins commented Aug 31, 2021

inlined commented Sep 2, 2021

abeisgoat commented Sep 2, 2021

rhodgkins commented Sep 14, 2021

rhodgkins commented Oct 25, 2021

tonyjhuang commented Mar 14, 2022

tonyjhuang commented Mar 14, 2022 •

edited

rhodgkins commented Mar 14, 2022

sam-gc commented Mar 14, 2022

rhodgkins commented Mar 14, 2022

tonyjhuang commented Mar 14, 2022

rhodgkins commented Mar 15, 2022

rhodgkins commented Mar 15, 2022

tonyjhuang commented Mar 15, 2022

tonyjhuang commented Mar 15, 2022

rhodgkins commented Mar 15, 2022 •

edited

tonyjhuang commented Mar 15, 2022

Storage emulator fixes and added rewriteTo support #3647

Storage emulator fixes and added rewriteTo support #3647

Conversation

rhodgkins commented Aug 6, 2021 • edited

Description

Downloading (using File#download())

Support for rewriteTo / copyTo

Flattened object storage for buckets

Fixed Objects: list endpoint

Scenarios Tested

Sample Commands

Linked issues

rhodgkins Aug 6, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhodgkins commented Aug 10, 2021 • edited

rhodgkins commented Aug 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhodgkins commented Aug 31, 2021

inlined commented Sep 2, 2021

abeisgoat commented Sep 2, 2021

rhodgkins commented Sep 14, 2021

rhodgkins commented Oct 25, 2021

tonyjhuang commented Mar 14, 2022

tonyjhuang commented Mar 14, 2022 • edited

rhodgkins commented Mar 14, 2022

sam-gc commented Mar 14, 2022

rhodgkins commented Mar 14, 2022

tonyjhuang commented Mar 14, 2022

rhodgkins commented Mar 15, 2022

rhodgkins commented Mar 15, 2022

tonyjhuang commented Mar 15, 2022

tonyjhuang commented Mar 15, 2022

rhodgkins commented Mar 15, 2022 • edited

tonyjhuang commented Mar 15, 2022

rhodgkins commented Aug 6, 2021 •

edited

Downloading (using `File#download()`)

Support for `rewriteTo` / `copyTo`

Fixed `Objects: list` endpoint

rhodgkins Aug 6, 2021 •

edited

rhodgkins commented Aug 10, 2021 •

edited

tonyjhuang commented Mar 14, 2022 •

edited

rhodgkins commented Mar 15, 2022 •

edited