Pre-Deploy error on Push to CDN

[rafaelromcar-parabol]

Issue - Bug

• How serious is the bug? P2
• OS: macOS / Linux / Windows / ?
• Browser: Chrome / Safari / Lynx / ? + version
• Node version: node --version
• NPM version: npm --version
• RethinkDB version: rethinkdb --version

App version v7.31.0

Error experienced in Staging with data that might have not been consistent

ServerID 177                                                                                                                                                                                                       
🚀 Predeploy Started v7.31.0 sha:a256820b703d822bad970e6e2f86b07b451a2135                                                                                                                                          
👴 RethinkDB Migration Started                                                                                                                                                                                     
[migrate-rethinkdb] No new migrations                                                                                                                                                                              
👴 RethinkDB Migration Complete                                                                                                                                                                                    
🐘 Postgres Migration Started                                                                                                                                                                                      
🔩 Postgres Extension Checks Started                                                                                                                                                                               
   pgvector                                                                                                                                                                                                        
🔩 Postgres Extension Checks Completed                                                                                                                                                                             
PostgreSQL - No migrations to run!                                                                                                                                                                                 
🐘 Postgres Migration Complete                                                                                                                                                                                     
🔗 QueryMap Persistence Started                                                                                                                                                                                    
⛓️ Prime Integrationgs Started                                                                                                                                                                                      
⛅️ Push to CDN Started                                                                                                                                                                                             
⛓️ Prime Integrations Complete                                                                                                                                                                                      
ReqlDriverError: The connection was closed before the query could be completed                                                                                                                                     
ReqlDriverError: The connection was closed before the query could be completed                                                                                                                                     
🔗 QueryMap Persistence Complete: 0 records added                                                                                                                                                                  
/home/node/parabol/dist/preDeploy.js:17107                                                                                                                                                                         
        if (parsedObjToReturn[textNodeName]) {                                                                                                                                                                     
                             ^                                                                                                                                                                                     
                                                                                                                                                                                                                   
TypeError: Cannot read properties of undefined (reading '#text')                                                                                                                                                   
  Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.                                                                                                       
    at /home/node/parabol/dist/preDeploy.js:17107:30                                                                                                                                                               
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)                                                                                                                                  
    at async parseXmlErrorBody (/home/node/parabol/dist/preDeploy.js:17116:19)                                                                                                                                     
    at async de_CommandError (/home/node/parabol/dist/preDeploy.js:6482:15)                                                                                                                                        
    at async /home/node/parabol/dist/preDeploy.js:44487:24                                                                                                                                                         
    at async /home/node/parabol/dist/preDeploy.js:1883:20                                                                                                                                                          
    at async /home/node/parabol/dist/preDeploy.js:44371:46                                                                                                                                                         
    at async /home/node/parabol/dist/preDeploy.js:12205:20                                                                                                                                                         
    at async /home/node/parabol/dist/preDeploy.js:1231:24                                                                                                                                                          
    at async /home/node/parabol/dist/preDeploy.js:1254:20 {                                                                                                                                                        
  '$metadata': { attempts: 1, totalRetryDelay: 0 }                                                                                                                                                                 
}                                                                                                                                                                                                                  
                                                                                                                                                                                                                   
Node.js v20.11.0

Then, after trying to reproduce on Staging and failing to do so, I experienced it again in one of our PPMIs. This was the sequence:

• Upgraded the application from 7.24.0 to 7.31.0 and pre-deploy was executed and ran without any problem. No errors were thrown.
• Redeployed again to modify some configuration parameters and pre-deploy failed with the following:

Warning: Ignoring extra certs from `./packages/server/postgres/cacerts`, load failed: error:80000002:system library::No such file or directory
🚀 Predeploy Started v7.31.0 sha:a256820b703d822bad970e6e2f86b07b451a2135
👴 RethinkDB Migration Started
[migrate-rethinkdb] No new migrations
👴 RethinkDB Migration Complete
🐘 Postgres Migration Started
🔩 Postgres Extension Checks Started
   pgvector: skipping check (POSTGRES_USE_PGVECTOR !== true)
🔩 Postgres Extension Checks Completed
PostgreSQL - No migrations to run!
🐘 Postgres Migration Complete
🔗 QueryMap Persistence Started
⛓️ Prime Integrationgs Started
⛅️ Push to CDN Started
⛓️ Prime Integrations Complete
ReqlDriverError: The connection was closed before the query could be completed
🔗 QueryMap Persistence Complete: 0 records added


/home/node/parabol/dist/preDeploy.js:17107
        if (parsedObjToReturn[textNodeName]) {
                             ^

TypeError: Cannot read properties of undefined (reading '#text')
  Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.
    at /home/node/parabol/dist/preDeploy.js:17107:30
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async parseXmlErrorBody (/home/node/parabol/dist/preDeploy.js:17116:19)
    at async de_CommandError (/home/node/parabol/dist/preDeploy.js:6482:15)
    at async /home/node/parabol/dist/preDeploy.js:44487:24
    at async /home/node/parabol/dist/preDeploy.js:1883:20
    at async /home/node/parabol/dist/preDeploy.js:44371:46
    at async /home/node/parabol/dist/preDeploy.js:12205:20
    at async /home/node/parabol/dist/preDeploy.js:1231:24
    at async /home/node/parabol/dist/preDeploy.js:1254:20 {
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

Node.js v20.11.0

I tried running pre-deploy again and no luck. The client has not complained about anything and it looks like they have been using AWS without any trouble so far. I could not reproduce on Staging or using an on-demand-env.

Acceptance Criteria (optional)

Pre-Deploy does not throw that error
Triage is performed to find the root cause of the bug, timeboxed to ~1 hour.

Estimated effort: 1 hour to triage. More if root cause is already identified.

[jordanh]

Promoting this one to this cycle; thank you for reporting/capturing @rafaelromcar-parabol

[rafaelromcar-parabol]

It happened again on v7.35.0.

[Dschoordsch]

This can be reproduced locally by

yarn build
yarn predeploy

but does not happen every time. It's the s3 client trying to push but getting a 520 response.

[Dschoordsch]

Happens also on 3.588 (latest)

[Dschoordsch]

It happens in the function regionRedirectMiddleware(clientConfig) in both versions.

[Dschoordsch]

The 520 response comes from Cloudflare and the S3 client does not know how to deal with it, so it treats it as 'CLIENT_ERROR' which means it's not retry-able. I'm looking if implementing a custom retry policy could solve this.

[rafaelromcar-parabol]

Happened again in production 😢