New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Firestore Emulator exports cannot be imported on other platforms #2421
Comments
This issue does not seem to follow the issue template. Make sure you provide all the required information. |
@akshaybabloo thanks for the report! This looks like a bug, we'll have to dig into it. |
Looking at the path printed out: |
@yuchenshi so I did find one place we have a likely bug in the code: firebase-tools/src/emulator/hubExport.ts Line 57 in e10851b
That should use $ cat data/firestore_export/firestore_export.overall_export_metadata
�mDN3�"O
M
Aall_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata osamstern@samstern-macbookpro3 tmp.3ZGMX0TU The overall export metadata points to a path with I'm actually not sure what the best path forward is here, do you know the best practice for handling this kind of thing? Also I have attached a |
I've opened b/160784464 internally to track the work needed in the Firestore Emulator. |
It looks like the path in |
@floogulinc That's a great observation. I'll be mostly likely going down that path when I get more time to work on it. |
I follow these steps to achieve import Firestore data from the windows machine on mac.
|
Hey! [REQUIRED] Environment infofirebase-tools: 8.16.2 Platform: Ubuntu 20.04.1 LTS (importing) and Windows 10 (exporting) [REQUIRED] Steps to reproduceCreate an export on windows and then import it in Ubuntu. [REQUIRED] Expected behaviourShould be able to run with all the imports. [REQUIRED] Actual behaviourI get an IO error: Exception in thread "main" com.google.cloud.datastore.core.exception.DatastoreException: /path/to/project/e2e-testing/emulator_data/minimal_rtdb_populate/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:316)
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.fetchEntities(ExportImportUtil.java:62)
at com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:89)
Caused by: java.io.FileNotFoundException: /home/acro-comp/Dev/acro-companion/e2e-testing/emulator_data/minimal_rtdb_populate/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:211)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:153)
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:312)
... 2 more |
Hi! I can confirm that this is still happening, I cannot read any data exported from Windows into linux (Ubuntu). This is causing me to break my CI flow because some of my team members develops on Windows and the pipeline runs in Linux. I'm considering migrating my pipeline to windows containers.
|
Exception in thread "main" com.google.cloud.datastore.core.exception.DatastoreException: /Users/markgoho/Projects/active-branch-tracker/db/exports/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:316)
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.fetchEntities(ExportImportUtil.java:62)
at com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:89)
Caused by: java.io.FileNotFoundException: /Users/markgoho/Projects/active-branch-tracker/db/exports/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:312)
... 2 more Still happening. @yuchenshi I'm curious why this was tagged a feature request and if that's what has pushed this bug to the bottom of the pile. |
@markgoho Thanks for bring this up. This is still on our roadmap and I've changed the type to bug for clarity. This is indeed a bit involved since it requires work in the Firestore Emulator (non-opensource) and also the CLI, which makes it a bit hard to prioritize. But we're in no way pushing this "to the bottom of the pile", nor do we have policies like fixing bugs strictly before feature requests. |
thanks @yuchenshi - I'm glad this is being recategorized as a bug and I understand this isn't a simple fix |
I can confir that I'm still using the workaround on Elementary OS 6 and Pop!_OS 21.04 when my collegues commit that file from their Windows 10 machines. |
@riker09 |
@krishna-ksharma assuming that means go to |
This should had to be fixed, can see this discussion started since 2020 and many of us macOs users having the same issue |
Sure: Open a webbrowser of your choice. Go to Or in other words: @markgoho was right. EDIT: The typo was originally done by @HenryVillavicencio and I agree. This should have been fixed a while ago. |
@yuchenshi Can you please share some update on this ? how long it gonna take ? |
@HenryVillavicencio's solution made it work again on my mac. |
I just bumped into this issue (windows-macos) and I'm disappointed it got reported over a year ago an still is not fixed :( |
@dex3r agreed it is disappointing, but consider the number of people using Firestore, then those using emulators, then those with exports, then those who also switch between different operating systems, and then compare that final number of affected people to the other issues that this team needs to fix that (presumably) affect a much larger group of people and now consider there is a workaround mentioned here: #2421 (comment) If this were your team, how would you prioritize this? |
I also get this bug importing to gcloud using :
|
Hey, this is still happening. Scenario is that Windows developers save to this file and Mac developers couldn't import the data. |
Agreed. This is killing me and my team. Basically Mac developers cannot write to the emulator DB for windows developers to read. Not to mention Mac developers have to manually copy and paste the latest data. @yuchenshi can you and your team help prioritize this bug fix? |
Can gladly confirm that taking an export from macos and importing it on windows now work! Thank you @yuchenshi et al. !! ❤️ 👍 🍾 |
Fixed in Firebase CLI v10.5.0. |
Are you sure about that. I've just updated to 10.5.0, restarted the emulator and then exported on Windows and my Edit: I've also just noticed that the storage emulator shows a white page when importing the newly exported data. I think it's the same issue as #4322 |
@oobayly First of all, we have not changed the export format, but just updated the Firestore emulator so it recognizes slashes from other OSes. This makes it backwards compatible with all the existing export dumps all of you have worked so hard to create in the past. The Storage Emulator is a separate, unrelated issue. As such, just changing slashes will not help. Please kindly comment on relevant Storage issues such as #4322 or open a new issue with the relevant details. |
Thanks for that clarification. |
Still hitting this error on 10.6.0 (importing on Ubuntu from Windows export) |
@charlesfries Would you mind providing a bit more details like the logs with Also, it will be great if you can upload a sample export zipball that we can look into. Please make sure it does not contain sensitive information you don't want to share with the public though, or you can reach out privately to Firebase Support. |
@yuchenshi Sure thing. Here is a reproduction: https://github.com/charlesfries/firebase-tools-repro-2
// firebase.json
{
"firestore": {
"rules": "firestore.rules",
"indexes": "firestore.indexes.json"
},
"functions": {
"predeploy": "npm --prefix \"$RESOURCE_DIR\" run build"
},
"storage": {
"rules": "storage.rules"
},
"emulators": {
"auth": {
"port": 9099
},
"functions": {
"port": 5001
},
"firestore": {
"port": 8080
},
"storage": {
"port": 9199
},
"ui": {
"enabled": true
}
}
}
As stated I'm running the emulators on Ubuntu (GitHub Actions) using a dataset exported from a Windows machine. Here is the output with Output
|
@charlesfries As we can see from the logs, this is an issue with the Storage emulator, which is a different issue from the OP. As mentioned in the comments above, #4322 tracks the Storage Emulator issue, and please kindly subscribe there and add your repro in the comments. I'll update the title here to indicate this is tracking Firestore only, and I'd encourage anyone running into import issues to try the debug steps above and decide on the right thread to chime in. |
@yuchenshi That is a separate issue. The issue you're referencing is actually a duplicate of another closed issue (#4326) which I also reproduced and observed being fixed via #4358/v10.6.0. Also that issue occurs at runtime, not when data is being imported during the emulator initialization. I think my current issue is more similar to this one, no? |
@charlesfries There are also two related issues involved in #4326, and see #4322 (comment) about the cross-platform part with a response from a team member mentioning they're working on it. I appreciate your patience while we navigate through the matrix of issues. Please be mindful that this is quite an active discussion thread with a lot of subscribers and we can really do everyone a favor by keeping the scope of each issue small and contained. As such, we'll lock this thread for now since the original issue has been fixed. If you still experience issues, please don't hesitate to open a new issue with the details and a collaborator will help triage or mark it as a duplicate. |
Creating an export of windows and importing it on macOS gives an IO error.
[REQUIRED] Environment info
firebase-tools: 8.4.3
Platform: macOS and Windows
[REQUIRED] Steps to reproduce
[REQUIRED] Expected behavior
[REQUIRED] Actual behavior
I get an IO error:
The text was updated successfully, but these errors were encountered: