[RFE] Support for an explicit and reasonable way for custom whereabouts.kubeconfig path (e.g. k3s)

[dannypv05261]

Is your feature request related to a problem? Please describe.

When whereabouts.kubeconfig is not located in the default CNI folder /etc/cni/net.d/whereabouts.d in host (k3s e.g. /var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig). There is no explicit and reasonable way to configure kubeconfig value in whereabouts.conf due to mixing up the container internal path and the host file path as a variable.

Below is the snapshot of code in install-cni.sh.
With changing to customize path with env variable CNI_CONF_DIR, I can make kubeconfig value ={custom_path}/whereabouts.d/whereabouts.kubeconfig in whereabouts.kubeconfig (WHEREABOUTS_KUBECONFIG_LITERAL), but no whereabouts.kubeconfig and whereabouts.conf will be in host eventually since changing on CNI_CONF_DIR also affect internal container path

CNI_CONF_DIR=${CNI_CONF_DIR:-"/host/etc/cni/net.d"}
WHEREABOUTS_RECONCILER_CRON=${WHEREABOUTS_RECONCILER_CRON:-30 4 * * *}

# Make a whereabouts.d directory (for our kubeconfig)

mkdir -p $CNI_CONF_DIR/whereabouts.d
WHEREABOUTS_KUBECONFIG=$CNI_CONF_DIR/whereabouts.d/whereabouts.kubeconfig
WHEREABOUTS_FLATFILE=$CNI_CONF_DIR/whereabouts.d/whereabouts.conf # Yuki~ Nikhil's note: imo we should remove "flatfile" from whereabouts vocabulary and call this "WHEREABOUTS_CONF_FILE" instead. Flatfile may be the format but it's confusing naming.
WHEREABOUTS_KUBECONFIG_LITERAL=$(echo "$WHEREABOUTS_KUBECONFIG" | sed -e s'|/host||')

.....

  touch $WHEREABOUTS_FLATFILE
  chmod ${KUBECONFIG_MODE:-600} $WHEREABOUTS_FLATFILE
  cat > $WHEREABOUTS_FLATFILE <<EOF
{
  "datastore": "kubernetes",
  "kubernetes": {
    "kubeconfig": "${WHEREABOUTS_KUBECONFIG_LITERAL}"
  },
  "reconciler_cron_expression": "${WHEREABOUTS_RECONCILER_CRON}"
}

The workaround I used below is to make container internal CNI folder path same as host folder path by charging the volumeMount path cni-net-dir and adding env variable CNI_CONF_DIR with same path as below, but it is not a good practice for users to change volumeMount path. The good practice should have the fixed internal path /host/etc/cni/net.d, users map their volume to the cni-net-dir and define custom settings in env variable.

      env:
        - name: CNI_CONF_DIR
          value: "/var/lib/rancher/k3s/agent/etc/cni/net.d"

        volumeMounts:
        - name: cni-net-dir
          # mountPath: /host/etc/cni/net.d
          mountPath: /var/lib/rancher/k3s/agent/etc/cni/net.d
        - name: cron-scheduler-configmap
          mountPath: /cron-schedule

    volumes:
        - name: cnibin
          hostPath:
            #path: /opt/cni/bin
            path: /var/lib/rancher/k3s/data/current/bin
        - name: cni-net-dir
          hostPath:
            #path: /etc/cni/net.d
            path: /var/lib/rancher/k3s/agent/etc/cni/net.d

Another workaround is to create a link file from /etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig to the custom path. But it seems not a good idea to apply additional changes in host.

Describe the solution you'd like

In install-cni.sh, change variable from WHEREABOUTS_KUBECONFIG_LITERAL to the existing env variable WHEREABOUTS_KUBECONFIG_FILE_HOST

  touch $WHEREABOUTS_FLATFILE
  chmod ${KUBECONFIG_MODE:-600} $WHEREABOUTS_FLATFILE
  cat > $WHEREABOUTS_FLATFILE <<EOF
{
  "datastore": "kubernetes",
  "kubernetes": {
    "kubeconfig": "${WHEREABOUTS_KUBECONFIG_FILE_HOST}"
  },
  "reconciler_cron_expression": "${WHEREABOUTS_RECONCILER_CRON}"
}
EOF

I submitted a request for that in case it is useful. #466

In this way, if I need to use the custom path, the only changes in daemonset-install.yaml are in volumes and env. It aligns with the general practice of the deployment file. It will not affect people using the default path as well.

      env:
        - name: WHEREABOUTS_KUBECONFIG_FILE_HOST
          value: "/var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"

      volumes:
        - name: cnibin
          hostPath:
            #path: /opt/cni/bin
            path: /var/lib/rancher/k3s/data/current/bin
        - name: cni-net-dir
          hostPath:
            #path: /etc/cni/net.d
            path: /var/lib/rancher/k3s/agent/etc/cni/net.d

Describe alternatives you've considered
Maybe mark the workaround in doc / README.md if this PR doesn't accept.

Additional context

[manuelbuil]

I recently updated the K3s docs and now there is a guide on how to deploy Multus + Whereabouts setting the correct path. Please have a look: https://docs.k3s.io/networking/multus-ipams. It'd be great if you could give it a try!