Windows CI image is taking long time to pull

[lizan]

It takes about 20 minutes to pull the docker image and 10 minutes to perform the build + tests:
https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_apis/build/builds/46369/logs/103

From:
https://dev.azure.com/cncf/envoy/_build/results?buildId=46369&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=168f295b-0553-5364-35f7-923225ecd8b3

Because we run everything in RBE, it is possible to run a smaller image in AZP (only bazel + a few deps for extracting dependencies).

Not a high priority but just for tracking.

cc @wrowe @sunjayBhatia

[sunjayBhatia]

The latest Windows image size has been reduced (see https://hub.docker.com/r/envoyproxy/envoy-build-windows2019/tags?page=1&ordering=last_updated) by a GB when compressed (4.73GB - 3.79GB) and uncompressed (10.5GB - 9.64GB)

But yeah, the two things we're fighting against are the base image size (total 4.99GB currently, one layer of which should be cached on the AZP workers, the second may not be if we're ahead or behind on hotfixes) and the size of VS Build tools (seems to be ~5GB).

[sunjayBhatia]

This PR also reduces the image size a bit once we added the Windows 10 SDK back correctly: #95

[wrowe]

When Google moves to Server 2004+ releases we will see notable improvements; https://techcommunity.microsoft.com/t5/containers/windows-server-version-2004-now-available/ba-p/1419194

The problem is that due to a security model change, we won't be able to instantiate docker spaces based on the earlier 1809-1909 images once google makes that transition, and we can't build these in 2004+ to run in the existing GCP hosts. I'm not clear what this successful transition looks like. @sunjayBhatia - ideas?

[sunjayBhatia]

The hard part about moving images to ones based on a semi annual release channel is that you need the host OS to match, not sure if its feasible to force developers to do so (or expect we will get access to those host OSes in AZP and RBE)

[sunjayBhatia]

Maybe we can do all CI builds on a smaller image but publish 2019 images as well for the public (the build image and envoy release image)

[wrowe]

Actually, we might have just found the answer to our mystery "GCP won't run 32 bit windows app" puzzle
https://support.microsoft.com/en-us/help/4542617/you-might-encounter-issues-when-using-windows-server-containers-with-t