Firestore emulator crash on export with ~0.5 GB of data #4336
Comments
|
I've found a similar thing with trying to import a 1.5gb production database for local debugging. The emulators just choke even though my machine has 64gb of ram. I opened an issue about it but unfortunately the resolution was "the emulators can't do that". #3929 |
|
i have 64gb of ram and have the same issue. the emulator ought to support this or at least support a flag to give users the option to specify the size of the process to override the defaults |
|
IIRC I tried changing java options to give the process much more ram but larger data sets seem to trigger a bug where it just eats up all available memory then crashes or hangs. I gave it something like 32gb for a 1.5gb data set and it still hung the same way as the default memory limit. For some reason the emulator is incapable of loading a larger data set no matter how many resources it is given. |
|
FYI the JVM behavior for Firestore Emulator can be tweaked using an environment variable, such as |
yuchenshi
changed the title
|
@yuchenshi in #3929 I tried it with -Xmx48g on my machine with 64gb of ram on a 1.5gb data set... It filled 48gb of ram then failed. There's seemingly a bug that once an import reaches a certain size the emulator gets "stuck" and just continues allocating memory until it runs out and crashes no matter how much you give it. |
|
@evelant Thanks for the reply. In order to pinpoint the issue, it'd be great if we can look into the particular dump you're trying to import. Would you mind contacting Firebase Support where you can privately share your exports file for reproducing the bug? |
|
@evelant @yuchenshi let me know the result of this. I can't send over my data because it contains sensitive information |
|
I was able to recreate the issue with the following snippet in Node: const createDay = (
id,
marketplace,
cogs,
estimatedFees,
sales,
cancels,
orders,
units,
storageFees,
date,
productId
) => {
const dayObject = {
AtoZClaims: {
AmazonFees: {},
Other: {},
ProductCharges: 0
},
Orders: {
AmazonFees: {
Commission: 0,
EstimatedFees: 0,
FBAPerOrderFulfillmentFee: 0,
FBAPerUnitFulfillmentFee: 0,
FBAWeightBasedFee: 0,
NonAmazonFBAPerUnitFulfillmentFee: 0,
Other: {}
},
Other: {},
ProductCharges: 0,
PromoRebates: {
Orders: 0,
Shipping: 0
}
},
OtherTransactions: {},
PaidToAmazon: {},
PostedOrders: {
AmazonFees: {
Commission: 0,
EstimatedFees: 0,
FBAPerOrderFulfillmentFee: 0,
FBAPerUnitFulfillmentFee: 0,
FBAWeightBasedFee: 0,
NonAmazonFBAPerUnitFulfillmentFee: 0,
Other: {}
},
Other: {},
ProductCharges: 0,
PromoRebates: {
Orders: 0,
Shipping: 0
}
},
Refunds: {
AmazonFees: {},
Other: {},
ProductCharges: 0,
PromoRebates: {
Orders: 0,
Shipping: 0
}
},
SellingFees: {
CostOfAdvertising: 0,
FBAFees: {},
Other: {
Coupons: 0
}
},
ShippingServices: {},
Ads: {
Clicks: 0,
Conversions: 0,
Cost: 0,
Impressions: 0,
Sales: 0,
Units: 0
},
COGS: cogs,
Counts: {
CanceledOrders: cancels,
Orders: orders,
RefundedUnits: 0,
ShippedOrders: 0,
ShippedUnits: 0,
Units: units
},
Date: date,
FIFOCOGS: 0,
Marketplace: marketplace,
ProductData: productId
? {
HasProduct: true,
ProductId: productId
}
: null,
ShippedCOGS: 0,
StorageFees: storageFees,
UserId: id
};
dayObject.Orders.AmazonFees.EstimatedFees = Number(estimatedFees);
dayObject.Orders.ProductCharges += sales;
return dayObject;
};
const { v4: uuid } = require('uuid');
for (let i = 0; i < 7500; i++) {
console.log(i);
await Promise.all([...Array(100)].map(async (value, j) => {
const collection = `TestCollection${j % 15}`;
await firestore.collection(collection).doc(uuid()).set(createDay(
'testid',
'asdf',
0,
0,
0,
0,
0,
0,
0,
'2020-02-02',
'testid'
));
}));
}After all 7500 loop iterations complete I get the following firestore-debug.log |
|
@jakeleventhal May I ask if you've tried with the memory flags? If so, how much heap space did you start it with? |
|
@yuchenshi i tried with memory flags and per "Activity Monitor" on mac java was only using <1gb ram. i am actually now unable to test this in a timely manner perhaps because of some other bug. In the for loop for my script, the first ~3,000 iterations were lightning fast (>100/s) but once it gets to ~3200 it slows to a crawl. Each iteration of the loop where i console.log(i) takes about 2-5 seconds. And then eventually it crashes with a "DEADLINE_EXCEEDED" error even though it is just the local emulator. I think what I provided is enough for a minimal reproduction case. FWIW i am running on a M1 chip not intel |
|
Edit: It feels like there's a certain limit around those 4.6MB that makes it crash as simply editing some (small) field values, so adding a few bytes can make the difference between it not failing and failing. This is just an empiric observation though. I am having the same problem (with the exact same stacktrace of OS: MacOS 11.5.2 (16GB ram) |
|
@AndreasJJ Would you be able to provide a repro for that BufferOverflowException? This would help the Firestore team identify what's going wrong here. Additionally, are you using a Mac with Intel chips or an M1 Mac? |
|
I am using a Intel chip Mac. So i tried to create a new repo for repro, where i simply copied the entire firebase folder into the new repo. In addition to changing the project id to a new project (as i didnt want my original project id to be shown in a public repo). However trying to edit the dataset in the new repo (which should have the exact same state as the one throwing the BufferOverflowException) doesn't end with an export error. As such i wonder if the error might have something to do with whatever files and states firebase saves locally on my computer that is associated with my original project id. Just for reference, here's the original error from the log from my real repo when i try to export through |
|
@joehan @yuchenshi i am using m1 mac |
[REQUIRED] Environment info
firebase-tools: 10.2.2
Platform: macOS
[REQUIRED] Test case
See below
[REQUIRED] Steps to reproduce
Export ~0.56GB of data for the firebase emulator. The emulator crashes and fails to export due to running out of buffer memory. My machine has 64GB ram so this should not be an issue
[REQUIRED] Expected behavior
Export completes successfully
[REQUIRED] Actual behavior
Export crashes with following error:
The text was updated successfully, but these errors were encountered: