Backup Restoration after transferring EFS and Database to another AWS AccountID

[FrankAllanRJ]

Description
I'm getting 502 Bad Gateway when trying to access Cloud Admin Hubs after creating a new stack using the backup restore option.

The scenario is: two AWS accounts (two AccountIDs), where, in one (we'll call it "A" account) there are several scenarios created (about 300), and the objective is to replicate all this environment, scenarios, etc., in another AWS account (another AccountID. Let's call it account "B"). In this account, the stack is named "test-001".

To Reproduce
Steps to reproduce the behavior:

1. Create a stack on account "B" with the same name as the stack on account "A". Eg: test-001

2. Transfer the file system (StorageEFS) of the "original" stack from account "A", which is populated (using DataSync), into the file system of the stack from account "B". After completing this step, if you access one of the EC2 stacks from account "B", you will see that the size of the /storage directory has increased to the same size as the file system from account "A". Use the df -h command to view the mount points.

3. In account "A", create a DB snapshot of the stack that is populated. It has to be the same as the file system.

4. If your stack on account "B" is in a different region than the stack on account "A", copy the snapshot to the desired region. That is, if your stack in account "A" is in Oregon, but your desire is to lift Hubs Cloud in N. Virginia, copy the Database snapshot to Virginia before sharing it with account "B".
   
   You will also need to create and share an AWS KMS key (on account "A") and share it with account "B".
   NOTE: Do not choose an AWS managed KMS key. Create a new one.

5. Still on account "A", share the snapshot with account "B" informing your accountID.
   

6. In account "B", go to RDS > Snapshots: Shared with me.
   Select the snapshot, click on Actions and Copy Snapshot. Set a name for the snapshot (which will be copied to your "B" account in the Manual tab) and enter the ARN of the KMS key you created in the account

7. Now that the snapshot is available on your "B" account, restore the Database from that snapshot. You can now use the AWS shared key from the KMS.
   In the Connectivity settings, I selected the same stack options I created earlier (item 1).

8. Create a new secret for the DB you just created in AWS Secrets Manager > Secrets. It will be the ARN of this secret that you will use to populate the Restore From Secret Database ARN field in CloudFormation.

9. Now create a new snapshot of this new DB. It will be the name of this snapshot that you will use to fill in the "Restore From Database Snapshot Identifier" field in CloudFormation

10. Now, let's create a new stack in the "B" account, filling in the Mozilla Template Backup Restoration fields in CloudFormation.
    If you don't have any other domains for this new stack, you will need to delete the previous stack (from the "B" account). In my case, as I have two more domains available, I was able to create the new one, keeping the previous stack active.
    Campos de restauração de backup do formulário:

• Restore From Stack Name: test-001 (name of the "empty" stack in account "B" and which is the same as the name in account "A")
• Restore From Database Snapshot Identifier: I filled in the name of the Database that I shared and uploaded from account "A" to account "B".
• Restore From Secret Database ARN: filled in with the Secret ARN I created for the "new" Database (copied from account "A")
• Restore From Vault Name: filled in with the vault name of the original file system of stack test-001, which I created in account "B", which was populated with files from the file system of account "A"
• Restore From Recovery Point ARN: I filled it with the Recovery Point ARN of a backup (AWS Backup) that I created on demand, because, as the stack was created this afternoon, the next automatic backup would only come into existence tomorrow morning (here in Brazil).

New stack "test-001-backup" (on account "B") was created without errors.

11. Now try to access the website referring to your stack. The error will appear
    

Expected behavior
Expected behavior
I expected that, when accessing this new environment, it would already be fully populated, with all the more than 300 scenarios that had been created in the production account (account "A") and everything working normally.

Hardware

• Device: Desktop and phone
• OS: Windows, Android iOS, Linux
• Browser: Firefox, Google Chrome, MS Edge.

[FrankAllanRJ]

I was able to eliminate the 502 Bad Gateway error.

The problem was that the database that downloaded account "A" had no password (or any other).
I discovered this when trying to perform queries through RDS.
I modified the DB and set a password. After this procedure, I managed to carry out the queries, but the system still gave error 502.
I restarted EC2 and after a few minutes the site loaded.

But HC still has problems:

• images open broken;
• does not trigger emails to authenticate us;
• and the only public rooms available, they don't load. Remembering that, in the production environment, everything works normally.

How to website is loading (account "B")

Follow the logs when I try to enter the ANJOS DIGITAIS-EF01..." room...
staging.mtkct.com-1689265375298.log
staging.mtkct.com - Network.har.txt

How to website is loading (account "A"). This it works:

staging.mtkct.com-1689265375298.log
staging.mtkct.com - Network.har.txt

[FrankAllanRJ]

Thanks for getting back to me, @pattersonbl2.

I didn't find the file in any of the buckets referring to the restored stack. This folder structure (/hab/svc/reticulum/config) doesn't even exist, as does the config.toml file.
Which of these buckets should config.toml be in?

I'll try to access EC2 and look for this file.

I take the opportunity to apologize for mentioning many people in the previous post. But I'm quite frustrated with not being able to transfer content between different AWS AccoundIDs and I feel that the Mozilla team doesn't seem to pay much attention to this topic.

Best regards,

Frank Allan.

[pattersonbl2]

Sorry for the confusion, You can find the auth file to set up Google auth for ssh into ec2 in the asset s3 bucket. I have an issue logging into my AWS account to get the file name, but you should be able to use session manager to bypass the primary ssh method to find the config.toml, which would be the config file for the backend of your hubs instance.

[FrankAllanRJ]

Ok, @pattersonbl2

Following your instructions, I managed to access EC2 and made a copy of the config file for my machine (thank you for that).
I am attaching it here. All information that I identified as sensitive, I replaced it with the word "CONFIDENTIAL"
config.toml.csv

From what I've analyzed, there are no mentions of the data source account, nor the previous stack, just the new account. So apparently everything is correct. However, I'm new to the subject and certainly don't have the knowledge of the Mozilla team on the correct settings.

[pattersonbl2]

Yeah, but it looks weird that the DB setting is set to the local host. that would mean it trying to connect to the database installed on the ec2 but it is not installed there. I am still locked out of my account to check my configs at moment.

[FrankAllanRJ]

@pattersonbl2
I had the same impression as you when I accessed the DB for the first time, but I downloaded the config from the production environment (AWS AccountID of origin) and it also contains hostname=localhost for all DB references contained in the file.
I confess that even today I have doubts whether the data I saw accessing the tables from EC2, via localhost, were inside the machine or RDS.

I would like to take the opportunity to answer another question: in which database and table(s) are the URLs referring to the external 3D objects contained in the scenarios (which we see in Spoke, in editing mode) stored?
If, at some point, the transfer of content between accounts works and I manage to upload a new fully populated environment (with data from another AWS AccountID), I will have to correct the URLs/domains of the objects to point to the new environment, since, in the copy of the Database, the data will be with the URLs of the source environment.

Best,
Frank Allan