Setting the instance

[nhpip]

Hi,

First, great application.

I'm running into an issue. I am deploying a multi-node instance AWS via Kubernetes. For the dashboards Grafana has a pull-down menu for application instance. The problem is on upgrades the assigned IP addresses / host-names change, but the old ones are still in the pull-down menu.

Is there a way, that you are aware of, to do either:

1. Clear old instances
2. Set the instance sometime after initialization where we can assign a static instance name?

Thanks and good work

[akoutmos]

Hey there! Number one isn't really possible since those entries in the list is what Grafana is able to find in Prometheus for the given time window. In addition, instance is something that Prometheus is attaching to the time series and not something that PromEx is doing (unless you are using the GrafanaAgent integration). The best way to tackle this problem would be to use the relabel_config option in Prometheus itself to dictate how the instance label is generated.

These docs may help:

• https://www.robustperception.io/controlling-the-instance-label/
• https://prometheus.io/docs/concepts/jobs_instances/
• https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

[nhpip]

Thanks, I had thought of relabeling it's just that I have to go through 20 layers of bureaucracy to get any changes like that. 😀. It's a shame that the metrics server can't relabel itself.

[nhpip]

FYI...I fixed it by using the app version. I added this:

   @impl true
    def manual_metrics(_) do

      Manual.build(
        :collection_server_version_manual_metrics,
        {__MODULE__, :get_version, []},
        [
          last_value(
            [:collection_server, :current, :version],
            event_name: [:collection_server, :current, :version],
            description:
              "Indicates the current application version. This is used to handle the 24 hour post-upgrade period where dead instances show up",
            measurement: :status,
            tags: [:version]
          )
        ]
      )
    end

    def get_version() do
      :telemetry.execute(
        [:collection_server, :current, :version],
        %{
          status: 1
        },
        %{
          version: CollectionServer.Mixfile.project()[:version]
        }
      )
    end

Then changed the dashboard to:

      {
        "allValue": null,
        "datasource": "CollectionServer",
        "definition": "label_values(collection_server_prom_ex_prom_ex_status_info, instance)",
        "hide": 0,
        "includeAll": false,
        "label": "Application Instance",
        "multi": false,
        "name": "instance",
        "options": [],
        "query": "label_values(collection_server_current_version{version=\"12.0.0\"}, instance)",
        "refresh": 2,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },

On a new release you just need to change the version number in the dashboard to match what is in mix.exs

[akoutmos]

The dashboards can be dynamically rendered via EEx. So you can probably automate the whole process so you never need to update the dashboard version and it'll automatically pick up the proper version.

[nhpip]

Thanks....I'll give that a go...or use sed :-)

[nhpip]

Oh, one more question. Is there a simple way to delete a single or set of statistics? I mean I did this.....but I don't like it

    :ets.select_delete(Elixir.CollectionServer.PromEx.Metrics, [{{{[:collection_server, :prom_ex, :admin, key, :status], :_}, :_}, [], [true]}])

[akoutmos]

At the moment there is no convenient way to clear out data from ETS. What is your use case specifically?

[nhpip]

The app has several thousands of users. Certain users seem to utilize far more resources than others. So we can go in via the shell and set a user, or list of users to monitor where the user id would be set as a tag in a statistic. e.g.

collection_server_user_file_uploads{user_id="bob@shoes.com"}

I can then set the user id in the dashboard to act as a filter.

The worry is that over time we will end up with lots of users still getting reported that we really don't care about anymore which could overload the system.

[akoutmos]

I would be cautious about using user_id as a label for a timeseries data point (re: https://www.robustperception.io/cardinality-is-key/). The goal is to use labels where the cardinality is bounded and limited to much smaller number than several thousand. Erasing it from ETS also won't do you much good as it is still in Prometheus and will stay in there until the retention time limit is hit.

This kind of free form data I think fits better in logs instead of metrics (at least from what i gather).

[nhpip]

My boss will frown at that ;-)

One final question. Looking, for example, at the cowboy plugin there are expressions/queries like:

"sum(increase(collection_server_prom_ex_plug_cowboy_http_requests_total{job=\"$job\", instance=\"$instance\", status=~\"4..|5..\"}[24h])) / sum(increase(collection_server_prom_ex_plug_cowboy_http_requests_total{job=\"$job\", instance=\"$instance\"}[24h])) OR on() vector(0)"

Now in my case there are several nodes deployed on AWS and in the case of certain plugins I want an aggregate view over all nodes. So I deleted the Application Instance section from the JSON file and changed the expressions to look something like:

"sum(increase(collection_server_prom_ex_plug_cowboy_http_requests_total{job=\"$job\", status=~\"4..|5..\"}[24h])) / sum(increase(collection_server_prom_ex_plug_cowboy_http_requests_total{job=\"$job\"}[24h])) OR on() vector(0)",

Basically I removed instance. I'm guessing that'll work?

[akoutmos]

I figured I would point it out and you can do what you will with the input :P.

I've been meaning to rework the dashboards in order to support aggregating metrics across instances....but I just haven't had the time unfortunately. Some additional funding for PromEx would help me buy some of my time back in order to do that. If you remove that label filter there, you should be able to achieve the result you need.