Production deploy failing - slug too big

[Br3nda]

-----> Discovering process types
       Procfile declares types     -> release, web
       Default types for buildpack -> console, rake

-----> Compressing...
 !     Compiled slug size: 548.2M is too large (max is 500M).
 !     See: http://devcenter.heroku.com/articles/slug-size
 !     Push failed

[Br3nda]

I'm gonna try and slim down the slug

[Br3nda]

production deploy via travis is failing with a slug over the hard limit of 500MB on heroku. we need to get it under 300MB soft limit.
Currently running a staging build that has been promoted to production - so we have a smaller slug.

Is there some "writing into the slug" going on? Like some caching or something in the file system?

[Br3nda]

Running bash on ⬢ nzsl... up, run.7341 (Standard-1X)
~ $ du -sh * | sort -hr
406M	vendor
370M	node_modules
55M	public
18M	tmp
8.4M	app
224K	spec
192K	config
148K	db
72K	yarn.lock
40K	lib
36K	LICENSE
36K	bin
12K	README.md
12K	Gemfile.lock
8.0K	log
8.0K	doc
8.0K	CONTRIBUTING.md
4.0K	script
4.0K	renovate.json
4.0K	Rakefile
4.0K	Procfile
4.0K	package.json
4.0K	Gemfile
4.0K	env-example
4.0K	config.ru
4.0K	CODEOWNERS
4.0K	app.json

[Br3nda]

~/node_modules $ du -sh * | sort -hr |head
364M	puppeteer
2.9M	devtools-protocol
872K	@types
196K	readable-stream
160K	unbzip2-stream
156K	ws
116K	debug
100K	tar-fs
100K	buffer
100K	bl

[Br3nda]

i can see c1e0699 is where we added puppeteer.
@danmorrisonNZ i that means to be a production dependency? It's 364M, and our limit for the whole app on heroku is 500M (recommended 300M)

[Br3nda]

the promoted slug, from staging to prod, will have bits missing -- i've found the bits. It's the pdf generation. puppeteer brings in a headless chrome to do it -- and chrome has gotten bigger over time.
right now it's failing becuase it can't find chrome.

2020-09-11T21:19:38.172946+00:00 app[web.1]: I, [2020-09-11T21:19:38.172842 #105]  INFO -- : [c603365b-a436-4d30-b2d3-775bce4398ed] node /app/bin/render-pdf.js /app/tmp/pdf_rendering/1599859178-8ef39f2669570b9850ea-input.html /app/tmp/pdf_rendering/1599859178-8ef39f2669570b9850ea-output.pdf
2020-09-11T21:19:38.342052+00:00 app[web.1]: Input path: file:///app/tmp/pdf_rendering/1599859178-8ef39f2669570b9850ea-input.html
2020-09-11T21:19:38.342744+00:00 app[web.1]: Output path: /app/tmp/pdf_rendering/1599859178-8ef39f2669570b9850ea-output.pdf
2020-09-11T21:19:38.545389+00:00 app[web.1]: Starting PDF conversion
2020-09-11T21:19:39.069837+00:00 app[web.1]: (node:1398) UnhandledPromiseRejectionWarning: Error: Navigation failed because browser has disconnected!
2020-09-11T21:19:39.069857+00:00 app[web.1]: at /app/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:51:147
2020-09-11T21:19:39.069857+00:00 app[web.1]: at /app/node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:47:62
2020-09-11T21:19:39.069858+00:00 app[web.1]: at Array.map (<anonymous>)
2020-09-11T21:19:39.069859+00:00 app[web.1]: at Object.emit (/app/node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:47:43)
2020-09-11T21:19:39.069859+00:00 app[web.1]: at CDPSession.emit (/app/node_modules/puppeteer/lib/cjs/puppeteer/common/EventEmitter.js:72:22)
2020-09-11T21:19:39.069859+00:00 app[web.1]: at CDPSession._onClosed (/app/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:247:14)
2020-09-11T21:19:39.069860+00:00 app[web.1]: at Connection._onClose (/app/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:128:21)
2020-09-11T21:19:39.069860+00:00 app[web.1]: at WebSocket.<anonymous> (/app/node_modules/puppeteer/lib/cjs/puppeteer/common/WebSocketTransport.js:17:30)
2020-09-11T21:19:39.069861+00:00 app[web.1]: at WebSocket.onClose (/app/node_modules/ws/lib/event-target.js:129:16)
2020-09-11T21:19:39.069861+00:00 app[web.1]: at WebSocket.emit (events.js:314:20)
2020-09-11T21:19:39.069862+00:00 app[web.1]: (Use `node --trace-warnings ...` to show where the warning was created)
2020-09-11T21:19:39.069912+00:00 app[web.1]: (node:1398) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
2020-09-11T21:19:39.069973+00:00 app[web.1]: (node:1398) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
2020-09-11T21:19:39.087811+00:00 app[web.1]: I, [2020-09-11T21:19:39.087696 #105]  INFO -- : [c603365b-a436-4d30-b2d3-775bce4398ed] method=GET path=/vocab_sheet/download_pdf format=html controller=VocabSheetsController action=download_pdf status=200 duration=993.84 view=0.00 db=8.98 params={}

[Br3nda]

I rolled back recent puppeteer upgrades - and that doesn't help. it still brings in the current security patched/stable chrome -- and that's too big for us.

I'm gonna leave it with for now.

Someone will need to find a way to generated pdfs without all of chrome (i like wicked pdf gem) -- or something else clever.

[Br3nda]

some last bit of info - the vendor folder is mostly ruby (284 out of 366M)

and heroku counts the compressed size - so you'll see it adds up to over 500M while running, but needs to gzip to less than 500M (preferrably under 300M)

[Br3nda]

also there's some writing to local file system still here that we should remove and put into a service like redis.

###### WARNING:
       You set your `config.active_storage.service` to :local in production.
       If you are uploading files to this app, they will not persist after the app
       is restarted, on one-off dynos, or if the app has multiple dynos.
       Heroku applications have an ephemeral file system. To
       persist uploaded files, please use a service such as S3 and update your Rails
       configuration.





-----> Compressing...
 !     Compiled slug size: 549.8M is too large (max is 500M).
 !     See: http://devcenter.heroku.com/articles/slug-size
 !     Push failed

[eoinkelly]

Wkhtmlpdf uses a super old version of WebKit which means it'll fail to render any reasonably modern CSS. I'm really not sure what to do here, will do some research.

[eoinkelly]

also there's some writing to local file system still here that we should remove and put into a service like redis.

@Br3nda AFAIK this app doesn't use active_storage at all (active_storage is fairly recent in Rails world) so, while we should adjust that setting to silence the warning, active_storage is not really writing to local disk in prod

[eoinkelly]

Seems this is caused by a bug in puppeteer 1.18.0 puppeteer/puppeteer#4610 - fixed in 1.18.1 apparently. I'll do the upgrade and see what happens

[eoinkelly]

Hmmm puppeteer/puppeteer#4610 refers to an older version of Puppeteer (current version is 5.3.0) so there is no Puppeteer upgrade to fix this 😞 .

[eoinkelly]

I had a look at the vendored ruby gems:

~/vendor/bundle/ruby/2.7.0/gems $ du -hs * |sort -h
12K	rails-6.0.3.2
32K	babel-transpiler-0.7.0
32K	sprockets-es6-0.9.2
40K	marcel-0.3.3
48K	websocket-extensions-0.1.5
52K	sass-rails-6.0.0
64K	erubi-1.9.0
72K	globalid-0.4.2
72K	request_store-1.4.1
80K	climate_control-0.2.0
84K	multi_xml-0.6.0
84K	rack-test-1.1.0
88K	sprockets-rails-3.2.1
92K	rails-html-sanitizer-1.3.0
100K	method_source-1.0.0
100K	pandoc-ruby-2.1.4
104K	rails-dom-testing-2.0.3
108K	crass-1.0.6
116K	execjs-2.7.0
116K	modernizr-rails-2.7.1
116K	rb-inotify-0.10.1
120K	faraday_middleware-1.0.0
124K	zeitwerk-2.3.1
128K	multipart-post-2.1.1
132K	responders-3.0.1
144K	lograge-0.11.2
144K	orm_adapter-0.5.0
148K	warden-1.2.8
152K	mini_magick-4.10.1
156K	rack-canonical-host-1.0.0
184K	dalli-2.7.10
188K	actionmailer-6.0.3.2
188K	sass-listen-4.0.0
200K	builder-3.2.4
200K	mini_portile2-2.4.0
208K	terrapin-0.6.0
224K	websocket-driver-0.7.2
228K	mime-types-3.2.2
244K	rb-fsevent-0.10.3
248K	actiontext-6.0.3.2
272K	activejob-6.0.3.2
288K	mini_mime-1.0.2
292K	i18n-1.8.3
296K	loofah-2.6.0
308K	whenever-1.0.0
316K	thor-0.20.3
328K	will_paginate-3.3.0
360K	httparty-0.18.0
368K	minitest-5.14.1
376K	actionmailbox-6.0.3.2
380K	actioncable-6.0.3.2
388K	chronic-0.10.2
404K	bootsnap-1.4.6
420K	activemodel-6.0.3.2
424K	temple-0.8.1
436K	sassc-rails-2.1.2
452K	haml-5.1.2
488K	activestorage-6.0.3.2
492K	public_suffix-4.0.5
520K	bcrypt-3.1.13
520K	rake-13.0.1
528K	skylight-core-4.3.1
532K	tilt-2.0.9
548K	sprockets-4.0.2
556K	videojs_rails-4.12.15
624K	addressable-2.7.0
632K	faraday-1.0.1
664K	raygun4ruby-3.2.1
676K	rack-2.2.3
736K	thread_safe-0.3.6
804K	devise-4.7.2
896K	uglifier-4.2.0
1.1M	actionview-6.0.3.2
1.2M	jquery-ui-rails-6.0.1
1.2M	mime-types-data-3.2018.0812
1.2M	puma-4.3.5
1.2M	tzinfo-1.2.7
1.3M	sass-3.7.4
1.4M	actionpack-6.0.3.2
1.5M	paperclip-6.1.0
1.6M	activesupport-6.0.3.2
1.6M	concurrent-ruby-1.1.6
1.6M	jquery-rails-4.4.0
1.8M	autoprefixer-rails-9.8.4
1.8M	nio4r-2.5.2
2.2M	babel-source-5.8.35
2.3M	newrelic_rpm-6.11.0.365
2.4M	msgpack-1.3.3
2.5M	mimemagic-0.3.5
2.5M	railties-6.0.3.2
2.7M	activerecord-6.0.3.2
3.8M	nokogiri-1.10.10
3.9M	mail-2.7.1
4.3M	pg-1.2.3
8.3M	ffi-1.12.2
8.8M	foundation-rails-6.6.2.0
10M	skylight-4.3.1
35M	libv8-7.3.492.27.1-x86_64-linux
38M	mini_racer-0.2.15
93M	sassc-2.3.0

We could get rid of skylight pretty easily but 10M won't fix this for us.

35M	libv8-7.3.492.27.1-x86_64-linux
38M	mini_racer-0.2.15
93M	sassc-2.3.0

are pretty tied into Rails so removing them seems disruptive. I think we'll have to look at replacing/moving puppeteer somehow.

[eoinkelly]

I installed heroku:repo and ran the following to try and reduce slug size:

HEROKU_APP=nzsl heroku plugins:install heroku-repo
HEROKU_APP=nzsl heroku repo:gc
HEROKU_APP=nzsl heroku repo:purge_cache

I also added the following env var to the production heroku app:

BUNDLE_WITHOUT=development:test

These got the slug size to 427 MB so deploys are working again.

[eoinkelly]

I don't know of a viable alternative to puppeteer for PDF rendering in the app. We could move puppeteer out to a separate service but that's a goodly chunk of work and would increase the monthly hosting costs a bit so I think we want to avoid doing that as long as possible.

[Br3nda]

Wicked pdf gem is a plausible option - have some same-licence opensource code that does this already - i'll have a play and see if it's smaller - we need more room to move.

[Rabid-Dan]

I think JY at Ackama recently did some pdf rendering work with a different gem too and I think he said it went pretty well/was relatively painless so we could always ask him what that was if an alternative is needed.

[eoinkelly]

@Br3nda I've used wicked_pdf on a few projects. The main challenge was that it was based on very old webkit so rendering any kind of modern CSS didn't work (e.g. flexbox). That said, the PDF rendering needs of this app are pretty simple so if you can tweak to the markup for the print view enough then wicked_pdf could be a good option.

[DSRU]

the vocab sheet 'save as PDF' function in production at the moment is not working. It downloads a pdf but then says the file is empty when you try to open it. Saving a pdf via the print menu works fine. Is this related to this issue?