Cannot get online files when remote server uses compression

[ooktay]

Description of the problem: I am hosting a mirror repository inside enterprise network. The mirror server supports gzip, deflate and br compression. I expect WDM to be able to download from the mirror, but it fails with a decoding exception.

Browser and version: All

Operating system: All

WebDriverManager version: 4.4.3

WebDriverManager traces:
2021-07-21 15:33:10 DEBUG i.g.b.w.v.VersionDetector.getBrowserVersionFromTheShell:140 - Detecting chrome version using online commands.properties
2021-07-21 15:33:10 DEBUG i.g.b.w.v.Shell.runAndWaitArray:54 - Running command on the shell: [/Applications/Google Chrome.app/Contents/MacOS/Google Chrome, --version]
2021-07-21 15:33:10 DEBUG i.g.b.w.v.Shell.runAndWaitArray:56 - Result: Google Chrome 91.0.4472.164
2021-07-21 15:33:11 ERROR i.g.b.w.o.HttpClient.execute:171 - Error HTTP 404 executing https://my-enterprise/repository/webdriver/chromedriver.xmlLATEST_RELEASE_91
2021-07-21 15:33:11 WARN i.g.b.w.v.VersionDetector.getDriverVersionFromRepository:113 - Exception reading https://my-enterprise/repository/webdriver/chromedriver.xmlLATEST_RELEASE_91 to get latest version of chromedriver (Error HTTP 404 executing https://my-enterprise/repository/webdriver/chromedriver.xmlLATEST_RELEASE_91)
2021-07-21 15:33:11 DEBUG i.g.b.w.v.VersionDetector.getDriverVersionFromProperties:85 - Getting driver version for chrome91 from online versions.properties
2021-07-21 15:33:11 DEBUG i.g.b.w.v.VersionDetector.getDriverVersionFromProperties:90 - Driver for chrome91 not found in online properties (using local versions.properties)
2021-07-21 15:33:11 INFO i.g.b.w.WebDriverManager.resolveDriverVersion:632 - Using chromedriver 91.0.4472.19 (resolved driver for Chrome 91)
2021-07-21 15:33:11 INFO i.g.b.w.WebDriverManager.logSeekRepo:962 - Reading https://my-enterprise/repository/webdriver/chromedriver.xml to seek chromedriver

Error trace:
2021-07-21 15:33:11 ERROR i.g.b.w.WebDriverManager.handleException:808 - There was an error managing chromedriver 91.0.4472.19 (com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: Invalid byte 2 of 2-byte UTF-8 sequence.)
io.github.bonigarcia.wdm.config.WebDriverManagerException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: Invalid byte 2 of 2-byte UTF-8 sequence.
at io.github.bonigarcia.wdm.WebDriverManager.getDriversFromXml(WebDriverManager.java:956)
at io.github.bonigarcia.wdm.managers.ChromeDriverManager.getDriverUrls(ChromeDriverManager.java:100)
at io.github.bonigarcia.wdm.WebDriverManager.createUrlHandler(WebDriverManager.java:830)
at io.github.bonigarcia.wdm.WebDriverManager.download(WebDriverManager.java:677)
at io.github.bonigarcia.wdm.WebDriverManager.manage(WebDriverManager.java:593)
at io.github.bonigarcia.wdm.WebDriverManager.setup(WebDriverManager.java:287)
at com.snapbytes.snaptest.web.WebDriverType.setup(WebDriverType.java:88)

[bonigarcia]

Thanks for reporting and contributing. Nevertheless, there are several aspects I don't see clear about this issue. First, I think the reported version of WDM is incorrect, isn't it? WDM 1.4.3 is very old. Are you actually using that one or is it a typo?

Supposing you are using version 4.4.3, in your trace logs there are errors 404, which is causes for bad URLs, but not for decoding problems, for the best of my knowledge. Can you explain that?

Regarding the PR, I don't know how this change can affect to the regular repositories. In addition, I would like to test it with an repositories supporting compression. How can it be done?

[ooktay]

Hi,
WDM version is 4.4.3, that was a typo.

404 is for LATEST_RELEASE_91, but the problem is really the online versions.properties. I am saving versions.properties on nexus using a cron job. Before 4.4.1, it worked because the http request did not have this line:
httpGet.addHeader("accept-encoding", "gzip, deflate, br");
When Nexus gets this this header, it will compress the response body, but the HttpClient does not have the decompress handlers, so it fails.

For testing, I think the simplest setup would be nginx reverse proxy for https://chromedriver.storage.googleapis.com/. Brotli module should be added. Alternatively cloudflare uses brotli.

---

About the The 404 for LATEST_RELEASE: This is related to how I configure the repository url: I am using nexus to mirror chromedrivers. Nexus does not support mirroring the index of https://chromedriver.storage.googleapis.com/, so I am saving it as chromedriver.xml using a cron job. Then I configure repository url as http://nexus/webdriver/chromedriver.xml. This way, WDM can get the xml from nexus. but it fails to get the LATEST_RELEASE, which deserves another PR.
I am also saving latest versions.properties, so maybe I can change this config now. I'll let you know.

[ooktay]

Nginx config for gzip:

location /repository/webdriver/ {
    proxy_http_version 1.1;
    proxy_pass   https://chromedriver.storage.googleapis.com/;
    add_header  Cache-Control "no-store";

    gzip on;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
    gzip_disable "MSIE [1-6]\.";
}

[ooktay]

For testing nginx setup, look for < Content-Encoding: gzip
curl -v --compressed -H 'Accept-Encoding: gzip' http://localhost/repository/webdriver/ -o chromedrivers.xml