composer va:web:build fails on Jenkins/BRD only and not on PR or local environments

[oksana-c]

Problem statement

Issues with vets-website package break the build on BRD only and not on local or PR environments.
This causes unexpected delays in CMS deployment schedule.
Build breaking events like these should be caught in Tugboat / local environments, so that the team can react sooner.

Broken build sample: http://jenkins.vfs.va.gov/job/testing/job/cms-post-deploy-tests-staging/229/consoleFull

Error: Cannot find module 'chokidar'

Acceptance Criteria

• composer va:web:build fails / behaves in the same manner on all types of environments: BRD, Tugboat, local.

[acrollet]

Ok, I believe I have the cause of this issue worked out.

yarn install is failing silently - compare the output in:

• http://jenkins.vfs.va.gov/job/testing/job/cms-post-deploy-tests-staging/227/consoleFull (successful)
• http://jenkins.vfs.va.gov/job/testing/job/cms-post-deploy-tests-staging/228/consoleFull (fails)

In the failing build, the output stops during step [2/4] Fetching packages..., without getting as far as [3/4] Linking dependencies... . However, the command fails silently with a successful exit status. I believe we're experiencing this bug:

yarnpkg/yarn#6915

I highly suspect that it's related to the on-going issues with the production proxies.

This was fixed in yarnpkg/yarn#6968, but the version of yarn we're using (v1.12.3) is older than that.

Working on a resolution...

[indytechcook]

Great find @acrollet !

[oksana-c]

@acrollet @indytechcook is this issue resolved?

[acrollet]

I think so, @indytechcook have we seen this error again?

[oksana-c]

@indytechcook please check comments above ^

[indytechcook]

Yes we did. My current theories are either related to the TIC dropping connections, a routing issue with the requests or how dependabot upgrades vets-website.

[ElijahLynn]

Yeah, this isn't an issue anymore. We should close this and re-open if it comes up again.

[indytechcook]

Reopening since it happened again.

http://jenkins.vfs.va.gov/job/testing/job/cms-post-deploy-tests-staging/268/consoleFull

    $ composer va:web:build | tee /dev/stderr | grep "Failed to pipe Drupal content into Metalsmith!" -B1000 -C8 && echo "tests.yml | composer va:web:build included the Drupal/Metalsmith error." && exit 1 || echo "tests.yml | Front end site was built! Check $DRUPAL_ADDRESS/static for raw output!" Output:/path/to/file
    ┃ > drush state-set va_gov.build va.web_build_stats TRUE -y                                                            
    ┃                                                                                                                      
    ┃ > rm -rf docroot/vendor/va-gov/web/build/vagovdev                                                                    
    ┃                                                                                                                      
    ┃ > cd web && echo "Downloading assets for vets-website ref $(git rev-parse --verify HEAD)"                            
    ┃                                                                                                                      
    ┃ Downloading assets for vets-website ref 1588f0895d2fef4925e72879e1a4c26098fc791d                                     
    ┃                                                                                                                      
    ┃ > cd web && NODE_ENV=production yarn build:content --pull-drupal --no-drupal-proxy --buildtype=vagovdev               ┃ --api=https://dev-api.va.gov --asset-source=$(git rev-parse --verify HEAD)                                           
    ┃                                                                                                                      
    ┃ yarn run v1.21.1                                                                                                     
    ┃                                                                                                                      
    ┃ $ node --max-old-space-size=10240 script/build-content.js --pull-drupal --no-drupal-proxy --buildtype=vagovdev        ┃ --api=https://dev-api.va.gov --asset-source=1588f0895d2fef4925e72879e1a4c26098fc791d                                 
    ┃                                                                                                                      
    ┃ internal/modules/cjs/loader.js:883                                                                                   
    ┃ throw err;                                                                                                           
    ┃ ^                                                                                                                    
    ┃                                                                                                                      
    ┃ Error: Cannot find module 'chokidar'                                                                                 
    ┃ Require stack:                                                                                                       
    ┃ - /var/www/cms/docroot/vendor/va-gov/web/script/build-content.js                                                     
    ┃ at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)                                          
    ┃ at Function.Module._load (internal/modules/cjs/loader.js:725:27)                                                     
    ┃ at Module.require (internal/modules/cjs/loader.js:952:19)                                                            
    ┃ at require (internal/modules/cjs/helpers.js:88:18)                                                                   
    ┃ at Object.<anonymous> (/var/www/cms/docroot/vendor/va-gov/web/script/build-content.js:1:18)                          
    ┃ at Module._compile (internal/modules/cjs/loader.js:1063:30)                                                          
    ┃ at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)                                            
    ┃ at Module.load (internal/modules/cjs/loader.js:928:32)                                                               
    ┃ at Function.Module._load (internal/modules/cjs/loader.js:769:14)                                                     
    ┃ at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12) {                                
    ┃ code: 'MODULE_NOT_FOUND',                                                                                            
    ┃ requireStack: [ '/var/www/cms/docroot/vendor/va-gov/web/script/build-content.js' ]                                   
    ┃ }      

[acrollet]

ok interesting, this time we see error Command failed with exit code 1., but that's not bubbling up to yaml-tests, which still reports:

✔ Process Succeeded in 1.272s Output: /path/to/file
✔ GitHub Status: va/web/build: success

[ElijahLynn]

Earlier this happened >

error An unexpected error occurred: "https://registry.yarnpkg.com/dot-prop/-/dot-prop-5.2.0.tgz: write EPROTO ┃ 140334237591424:error:141A10F4:SSL routines:ossl_statem_client_read_transition:unexpected ┃ message:../deps/openssl/openssl/ssl/statem/statem_clnt.c:395:

Can't reproduce the with regular curl. Still sleuthing.

[ElijahLynn]

K, so we just worked out a solution:

Add the va:web:install step to the Build (B) step in Ansible. The Build step runs bake instances and they don't go through the TIC. This is where the initial composer install runs too.
Composer install still runs on deploys but just because we haven't removed the deploy steps from tests.yml yet because devshop needs them and I was waiting (being lazy) for that so I don't have to update he playbook to run filters.

But, I don't know when DevShop will be EOL so I'll go ahead and do all of that now.

[ElijahLynn]

This is done in #3904 and https://github.com/department-of-veterans-affairs/devops/pull/8557 and we did EOL DevShop on Friday so was able to do it a bit cleaner.