[kibana] Kibana jobs doesn't have tolerations field

[Stephani0106]

Chart version: 8.5.1

Kubernetes version: 1.24.6

Kubernetes provider: Azure Kubernetes Service

Helm Version: 3.10.2

Describe the bug: Kibana jobs 'post-delete' and 'pre-install' doesn't support tolerations field

I'm working on upgrading the ElasticSearch chart to version 8.5.1 in my environment, the previous version of my ElasticSearch helm chart was 7.17.3.

In my environment we work with Azure Spot instances, and all Kubernetes nodes have a Label 'kubernetes.azure.com/scalesetpriority=spot' that says that applications that run on this node should tolerate Spot instances through a toleration.

As follows:

# Label in Nodes
kubernetes.azure.com/scalesetpriority=spot

# Toleration in resources
tolerations:
  ...
  - key: kubernetes.azure.com/scalesetpriority
    operator: Equal
    value: spot
    effect: NoSchedule

Without this tolerations the pods are "hung" and do not find a Node to run. Getting as follows:

With the error:

0/7 nodes are available: 3 node(s) had untolerated taint {CriticalAddonsOnly: true}, 
4 node(s) had untolerated taint {kubernetes.azure.com/scalesetpriority: spot}. 
preemption: 0/7 nodes are available: 7 Preemption is not helpful for scheduling.

Well, during the upgrade of Kibana to version 8.5.1 I noticed that from now on, two jobs called 'pre-install' and 'post-delete' are installed. However, within the template of these Jobs, there is no field of tolerations causing all the jobs I climb in my environment to be "hung" every time.

To try to work around this, I am editing the YAML file in the environment manually, inserting the required tolerance, but that does not help because as soon as I change the YAML of a Job, this Job becomes invalid and another is born in the sequence without the toleration, as follows:

I need to find a way around this problem because it's affecting my environment and I can't do the Kibana installation.

[Stephani0106]

A PR to add this setting has been opened at: #1758