(URGENT) Crashes in Fontist on GHA

[ronaldtse]

From @ghobona at metanorma/metanorma-ogc#651

Error 1:

/usr/local/bundle/gems/fontist-1.20.0/lib/fontist/font.rb:220:in `request_formula_installation': undefined method `each' for nil:NilClass (NoMethodError)

      paths.each do |path|
           ^^^^^
	from /usr/local/bundle/gems/fontist-1.20.0/lib/fontist/font.rb:207:in `block in download_font'
	from /usr/local/bundle/gems/fontist-1.20.0/lib/fontist/font.rb:206:in `each'

Error 2:

/usr/local/bundle/gems/fontist-1.20.0/lib/fontist/indexes/base_index.rb:9:in `from_yaml': Please fetch formulas with `fontist update`. (Fontist::Errors::MainRepoNotFoundError)
	from /usr/local/bundle/gems/fontist-1.20.0/lib/fontist/formula.rb:82:in `find_by_font_file'
	from /usr/local/bundle/gems/fontist-1.20.0/lib/fontist/system_index.rb:217:in `override_font_props'
	from /usr/local/bundle/gems/fontist-1.20.0/lib/fontist/system_index.rb:213:in `parse_font'

[ronaldtse]

@alexeymorozov can you please help with this? Thanks.

[alexeymorozov]

Working on it. I found out which formula fails (google/space_mono) and that Google Fonts started to use the javascript redirection for downloads which are not handled in fontist now.

Curl for Space Mono returns the 200 OK status code but its content is the javascript redirection:

$ curl -v https://fonts.google.com/download\?family\=Space%20Mono
*   Trying 142.251.208.142...
* TCP_NODELAY set
* Connected to fonts.google.com (142.251.208.142) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.google.com
*  start date: Mar  4 06:35:50 2024 GMT
*  expire date: May 27 06:35:49 2024 GMT
*  subjectAltName: host "fonts.google.com" matched cert's "*.google.com"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc53880d200)
> GET /download?family=Space%20Mono HTTP/2
> Host: fonts.google.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< content-type: text/html; charset=utf-8
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: Mon, 01 Jan 1990 00:00:00 GMT
< date: Wed, 03 Apr 2024 18:09:45 GMT
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< cross-origin-opener-policy: same-origin-allow-popups
< content-security-policy: script-src 'nonce-GVEiTdAScLZxHa3X-qcLUw' 'unsafe-inline' 'strict-dynamic' https: http:;object-src 'none';base-uri 'self';report-uri /cspreport
< server: ESF
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< set-cookie: NID=512=VP1j5Xhc4VojM8lt6ih-wa1ubsRd1EDYYWXpgtB6Bng9ZvT3DpKxDrPNDHv354Fot6KIG3MniOtFrf2UqdweFG7CBX6SUGStcEAH7N7zzwXaJlW1hRFUA0dKSbVL1NVQIqfiYOBwD78OMlqTORoXn-KZAk2-eU1IlH1JbvZXf8M; expires=Thu, 03-Oct-2024 18:09:45 GMT; path=/; domain=.google.com; HttpOnly
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< accept-ranges: none
< vary: Accept-Encoding
<
<script data-id="_gd" nonce="GVEiTdAScLZxHa3X-qcLUw">window.WIZ_global_data = {"TSDtV":"%.@.[[null,[[45427734,null,false,null,null,null,\"LWyDvc\"],[45611810,null,false,null,null,null,\"xo3fQd\"],[45618100,null,true,null,null,null,\"Rf6Ob\"],[45616206,null,false,null,null,null,\"cUJvHd\"],[45616047,null,false,null,null,null,\"THDyn\"],[45427735,null,true,null,null,null,\"qwZbqf\"],[45427733,null,false,null,null,null,\"iyX4bd\"],[45620697,null,false,null,null,null,\"Sm4Z2\"]],\"CAMSBh0Dg8KEKg\\u003d\\u003d\"]]]","nQyAE":{}};</script><!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="description" content="Making the web more beautiful, fast, and open through great typography"><meta property="og:site_name" content="Google Fonts"><meta property="og:url" content="https://fonts.google.com/download"><meta property="og:title" content="Google Fonts"><meta property="og:description" content="Making the web more beautiful, fast, and open through great typography"><meta property="og:image" content="https://www.gstatic.com/images/icons/material/apps/fonts/1x/catalog/v5/opengraph_color.png"><meta property="og:image:width" content="1200"><meta property="og:image:height" content="630"><meta property="og:image:type" content="image/png"/><meta name="twitter:card" content="summary_large_image"><meta name="twitter:site" content="@googlefonts"><meta name="twitter:title" content="Google Fonts"><meta name="twitter:description" content="Making the web more beautiful, fast, and open through great typography"><meta name="twitter:image" content="https://www.gstatic.com/images/icons/material/apps/fonts/1x/catalog/v5/opengraph_color.png"><meta property="og:type" content="website"/><meta name="viewport" content="width=device-width, initial-scale=1"><link rel="icon" href="https://www.gstatic.com/images/icons/material/apps/fonts/1x/catalog/v5/favicon.svg"><meta name="apple-mobile-web-app-capable" content="yes"><meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"><link rel="apple-touch-icon" href="https://www.gstatic.com/images/branding/product/1x/google_fonts_64dp.png"><link rel="apple-touch-icon" sizes="96x96" href="https://www.gstatic.com/images/branding/product/1x/google_fonts_96dp.png"><link rel="apple-touch-icon" sizes="128x128" href="https://www.gstatic.com/images/branding/product/2x/google_fonts_64dp.png"><link rel="apple-touch-icon" sizes="192x192" href="https://www.gstatic.com/images/branding/product/2x/google_fonts_96dp.png"><link rel="canonical" href="https://fonts.google.com/download"/><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin><style nonce="zqrRmmpQZRPNuHusFV76oQ">@font-face{font-family:'Google Sans';font-style:normal;font-weight:400;font-display:block;src:url(//fonts.gstatic.com/s/googlesans/v58/4Ua_rENHsxJlGDuGo1OIlJfC6l_24rlCK1Yo_Iqcsih3SAyH6cAwhWdRFD48TE63OOYKtrwEIJllpyw.ttf)format('truetype');}@font-face{font-family:'Google Sans';font-style:normal;font-weight:400;font-display:block;src:url(//fonts.gstatic.com/s/googlesans/v58/4Ua_rENHsxJlGDuGo1OIlJfC6l_24rlCK1Yo_Iqcsih3SAyH6cAwhX9RFD48TE63OOYKtrwEIJllpyw.ttf)format('truetype');}@font-face{font-family:'Google Sans';font-style:normal;font-weight:500;font-display:block;src:url(//fonts.gstatic.com/s/googlesans/v58/4Ua_rENHsxJlGDuGo1OIlJfC6l_24rlCK1Yo_Iqcsih3SAyH6cAwhWdRFD48TE63OOYKtrw2IJllpyw.ttf)format('truetype');}@font-face{font-family:'Google Sans';font-style:normal;font-weight:500;font-display:block;src:url(//fonts.gstatic.com/s/googlesans/v58/4Ua_rENHsxJlGDuGo1OIlJfC6l_24rlCK1Yo_Iqcsih3SAyH6cAwhX9RFD48TE63OOYKtrw2IJllpyw.ttf)format('truetype');}@font-face{font-family:'Google Sans Mono';font-style:normal;font-weight:400;font-display:block;src:url(//fonts.gstatic.com/s/googlesansmono/v15/P5sUzYWFYtnZ_Cg-t0Uq_rfivrdYH4RE8-pZ5gQ1abT53wVQGrw.ttf)format('truetype');}@font-face{font-family:'Google Sans Mono';font-style:normal;font-weight:700;font-display:block;src:url(//fonts.gstatic.com/s/googlesansmono/v15/P5sUzYWFYtnZ_Cg-t0Uq_rfivrdYH4RE8-pZ5gQ1abQe2AVQGrw.ttf)format('truetype');}</style><style nonce="zqrRmmpQZRPNuHusFV76oQ">@font-face {font-family: Tofu;src: url("data:font/ttf;base64,AAEAAAAKAIAAAwAgT1MvMmPgY1IAAAEoAAAAYGNtYXAALvz2AAABkAAAAGRnbHlmLTUfeQAAAfwAAAIaaGVhZCe/nsoAAACsAAAANmhoZWEJgAWAAAAA5AAAACRobXR4BbUASwAAAYgAAAAGbG9jYQENAAAAAAH0AAAABm1heHAABwC3AAABCAAAACBuYW1lDgElPAAABBgAAADccG9zdHBseXQAAAT0AAAAKwABAAAAAQAAFiMNcl8PPPUAAwXcAAAAAOG7K+oAAAAA4bsr6gCI/+IEqgQ9AAAABgACAAAAAAAAAAEAAARM/84AAAUtAIgAgwSqAAEAAAAAAAAAAAAAAAAAAAABAAEAAAACALYABAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAABAUtAZAABQAEA88DhAAAAHEDzwOEAAACDQBLAcIAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAATk9ORQDAAAAAIARM/84AAARMADIAAAABAAAAAALuBBoAAAAgAAAFLQBLAIgAAAAAAAMAAAADAAAAHAADAAEAAAAcAAMACgAAADwABAAgAAAABAAEAAEAAAAB//8AAAAB//8AAAABAAAAAAANAAAAAAAoAAAAAAAAAAIAAAACAAD+DgAAAAEAAP4QABD//wAAAAEAAAAAAQ0AAAAEAIj/4gSqBD0AOgB5AKkAtQAAAQYHBgYHBhcWFxYWFxcHBgYHBgcGFxYXFhceAjI2Njc2NzYnJicmJicnNzY3NicmJicuAyMqAgUWFxYXFhYHBgcGBwYVFBcWFxYXFgcGBgcOAyMiLgInJiYnJjc2NzY3NjU0JyYnJicmNTQ3NjcyNjIyFgcGBwYVFBYWNzA3Njc2NzYyFxYXFhYHBgYHBgYHBgYVFTM1NDY3NjY3NjY3NicmJgMGBhcWFxY3NiYnJgG3RTU9VhASDQ0kCSEHBgoaLQwOAgMzJz87RAhIaHNoSAhrTWEEAg4MLRkKEDUXHRcVeVQNFi9kXD5oQQG2NyxoGQMBAg05GTYKCjYZOQwJDhBlQgsZMlxOT1wyGQtCZRAOCQw5GTYKChISTBwLSEFbCFh6elj4VCwHHx4BCQoLExYKJgorDwQBBAUTHhAcAw4KRQgLAxYNJR4EBAoQZTcVEgcHGB8VEwkYEwQ9Bxwfa0JGREA2DSYGBQkYRyQrLmBTPyckCQEBAQEBAQ5NYYsuKiRHGAkQNkVZWlN7FwQEAQFGBBc2cw88EU44GiAGAQEGIBo4TTU2QGEPAwMBAQEBAwMPYUA2NU04GiAGAQEGCwwyUiQqZ0hBBwEBuwpZDQIBDg0BFBMLEwYCAgspCyQLDxgeEB0FFCciGxQaIBEEGA0kMR8dHzY5/g8IIxYYBwkVEzIMCgAAAAAAAAYATgADAAEECQABAAgAAAADAAEECQACAA4ACAADAAEECQADAC4AFgADAAEECQAEABgARAADAAEECQAFABoAXAADAAEECQAGABgAdgBUAG8AZgB1AFIAZQBnAHUAbABhAHIAMQAuADAAMAAwADsATgBPAE4ARQA7AFQAbwBmAHUALQBSAGUAZwB1AGwAYQByAFQAbwBmAHUAIABSAGUAZwB1AGwAYQByAFYAZQByAHMAaQBvAG4AIAAxAC4AMAAwADAAVABvAGYAdQAtAFIAZQBnAHUAbABhAHIAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAABAgR0b2Z1AA==");}</style><link id="app-css" rel="stylesheet" href=&#47;&#47;www.gstatic.com&#47;_&#47;fonts&#47;_&#47;ss&#47;k&#61;fonts.app.&#45;OPkraAi&#45;50.L.W.O&#47;am&#61;wA&#47;d&#61;0&#47;br&#61;1&#47;rs&#61;AE60u5cLltj_kUTWPieEO7HyO1c6EPULvA nonce="zqrRmmpQZRPNuHusFV76oQ"><title>Google Fonts</title><base href="/" target="_blank"/><script async src="https://www.googletagmanager.com/gtag/js?id=G-XPW1QSKFW4" nonce="GVEiTdAScLZxHa3X-qcLUw"></script><script nonce="GVEiTdAScLZxHa3X-qcLUw">window.dataLayer = window.dataLayer || [];function gtag(){dataLayer.push(arguments);}window['GA_TRACKING_ID'] = 'G-XPW1QSKFW4'; gtag('js', new Date()); gtag('config', window['GA_TRACKING_ID'], {'anonymize_ip': true, 'referrer' : document.referrer.split('?')[0], 'experiment_ids': '94429828',});</script></head><body itemscope itemtype="http://schema.org/WebPage"><gf-root></gf-root><script nonce="GVEiTdAScLZxHa3X-qcLUw">
        const wasPageRefreshed = sessionStorage.getItem('refreshToken') == 'true';
        const tabCount = Number(localStorage.getItem('tabCount'));
        const isDarkTheme = localStorage.getItem('isDarkTheme') === 'true';
        const isOsSettingDarkTheme = window.matchMedia && window.matchMedia('(prefers-color-scheme: dark)').matches;

        if(tabCount == 0 && !wasPageRefreshed){
          if (isOsSettingDarkTheme) {
            document.body.classList.add('gf-dark-theme');
          } else {
            document.body.classList.remove('gf-dark-theme');
          }
        } else {
          if (isDarkTheme) {
            document.body.classList.add('gf-dark-theme');
          } else {
            document.body.classList.remove('gf-dark-theme');
          }
        }
      window['CONFIG'] = 'WyJjb25maWciLG51bGwsIi8iLCIvIiwiLy8iLCJmb250cy5nb29nbGVhcGlzLmNvbSIsMCwwLDAsWzk0NDI5ODI4XSwwLDEsMSwiMTAyNTY5NSJd';
      if ('serviceWorker' in navigator) {
        (function() {
          navigator.serviceWorker.getRegistrations().then(function (registrations) {
            for (let registration of registrations) {
              registration.unregister();
            }
          });
        })();
      }
* Connection #0 to host fonts.google.com left intact
      </script><script id="base-js" src="//www.gstatic.com/_/fonts/_/js/k=fonts.app.en_US.A1r0eMW6XwI.es5.O/am=wA/d=1/br=1/rs=AE60u5c3W4OYB5DeUX_tE95S3ueBeRxTcg/m=core" nonce="GVEiTdAScLZxHa3X-qcLUw"></script><script async src="https://ssl.gstatic.com/external_hosted/lottie/lottie_light.js" nonce="GVEiTdAScLZxHa3X-qcLUw"></script></body></html>* Closing connection 0

[alexeymorozov]

Temporarily as a hotfix and a test case I've replaced the current formula with 4 others for each style with urls which the site uses. I've tried and it works in my fork. The fix is already in the formulas repo so the opengeospatial's repo should work too now.

But we need a proper fix because, as far as I see, all google formulas stopped working. The current archive URL returns javascript which calls API and gets urls for each font style, then downloads them and archives in a zip file after.

API with urls for styles: https://fonts.google.com/download/list?family=Space+Mono

More information on the Google's change: google/fonts#7481

How people are dealing with the change: squidfunk/mkdocs-material#6983

[ronaldtse]

@alexeymorozov I think we have two problems to deal with:

1. Fetching and indexing Google Fonts into Formulas
2. Allowing Fontist users to directly download fonts from Google Fonts in a non-interactive way

For 1, the best choice has to be directly parsing the google/fonts GitHub repository. Changes there may be more frequent than their new Developer API, but also means we can respond to them faster. It would mean we need to parse font metadata ourselves, but that's also reasonable because we need that functionality for fonts sourced outside of Google Fonts.

For 2, users have two needs with Google Fonts:

• Downloading the font file locally. We can link people to the Google Fonts GitHub.
• Providing an HTTP link for web rendering. This we don't serve yet, and we will need to have the WOFF-type link to make this happen. The WOFF file and link are not provided in the GitHub repository, so we will need to use the Developer API.

Conclusion:

1. For most of Formula information, we can populate from their GitHub repository, including for local font download.
2. For public usage of the font we will need to use the API for every font to obtain the public-facing font file links and WOFF links.

Regarding 1:

Interestingly, their metadata file (METADATA.pb in text protobuf) looks eerily similar to our Formula file:

• https://github.com/google/fonts/blob/main/ofl/abhayalibre/METADATA.pb

Except that they have:

• per font weight
• designer
• category: SERIF, SANS_SERIF, DISPLAY, MONOSPACE, ...
• primary_script
• subsets
• optional classifications
• optional stroke
• optional source: lists where the repository/commit the font file and/or archive is from. If there is source they have an upstream.yaml file that lists out:
  • for repository: which branch, location of the license file, location of the TTF file inside that source repository.
  • for archive: URL and location of the license and TTF files
  • NOTE: They have a PR that merges upstream.yaml into METADATA.pb

They also have DESCRIPTION.{lang}.html that provides a description of the font, in HTML. Perhaps we should also supply this on the Formulas site.

Maybe having this information would also be a good choice for Fontist...