Appliance can create recommended StorageClasses for various k8s distributions

[craigfurman]

Background

All PersistentVolumes across all services need to be able to accept a storage class configured by the admin. As Sourcegraph developers, we can't know what cloud-provider-specific storage classes exist in user deployments. One simplifying assumption we've carried forward from the helm chart so far, is that all PVs can use the same storage class (or none).

The helm chart allows the optional creation of a storage class specifically for use with Sourcegraph: https://github.com/sourcegraph/deploy-sourcegraph-helm/blob/main/charts/sourcegraph/templates/storageclass.yaml.

Proposal

The reconciler should be "dumb", and not make assumptions that any non-default storage class should be used unless injected as config. By default, it should not pass storageClassName to PVC templates. This will cause k8s to use the cluster-global default storage class.

The appliance application can be a little smarter, and offer to create a sensible StorageClass named "sourcegraph". Examples of sensible: on GKE, configure a GCE SSD StorageClass. This sort of thing is particularly useful to "box distribution" users (e.g. the AMI - I made up a term because the term I'd normally use for this is "appliance", and that's taken). Of course the appliance might not recognise every managed k8s distribution, but we can always add support for more over time, and the admin can always fall back on injecting the storage class name.

One thing to note is that StorageClass is (naturally) a cluster-scoped resource, and as such this is a "privileged" feature. Users who want to use the appliance in unprivileged mode but who don't want to use cluster-default storage classes can use their admin privilege to create a storageclass out-of-band and inject its name. Note that porting the default behavior of our helm chart would be incompatible with the unprivileged-by-default appliance.