prometheus: _total is appended when one already exists

[gouthamve]

Description

Declaring a counter with _total or .total suffix ends up exporting the metric with an additional _total suffix.

For example, the following metric exposes:
counter, err := meter.Float64Counter("foo.total", api.WithDescription("a simple counter")) -->

# HELP foo_total_total a simple counter
# TYPE foo_total_total counter
foo_total_total{A="B",C="D",otel_scope_name="github.com/open-telemetry/opentelemetry-go/example/prometheus",otel_scope_version=""} 5

Environment

• OS: Golang
• Architecture: ARM64
• Go Version: go1.20.6 darwin/arm64
• opentelemetry-go version: v1.16.0

Steps To Reproduce

For this code:

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"

	"github.com/prometheus/client_golang/prometheus/promhttp"

	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/prometheus"
	api "go.opentelemetry.io/otel/metric"
	"go.opentelemetry.io/otel/sdk/metric"
)

func main() {
	ctx := context.Background()

	// The exporter embeds a default OpenTelemetry Reader and
	// implements prometheus.Collector, allowing it to be used as
	// both a Reader and Collector.
	exporter, err := prometheus.New()
	if err != nil {
		log.Fatal(err)
	}
	provider := metric.NewMeterProvider(metric.WithReader(exporter))
	meter := provider.Meter("github.com/open-telemetry/opentelemetry-go/example/prometheus")

	// Start the prometheus HTTP server and pass the exporter Collector to it
	go serveMetrics()

	opt := api.WithAttributes(
		attribute.Key("A").String("B"),
		attribute.Key("C").String("D"),
	)

	// This is the equivalent of prometheus.NewCounterVec
	counter, err := meter.Float64Counter("foo.total", api.WithDescription("a simple counter"))
	if err != nil {
		log.Fatal(err)
	}
	counter.Add(ctx, 5, opt)

	ctx, _ = signal.NotifyContext(ctx, os.Interrupt)
	<-ctx.Done()
}

func serveMetrics() {
	log.Printf("serving metrics at localhost:2223/metrics")
	http.Handle("/metrics", promhttp.Handler())
	err := http.ListenAndServe(":2223", nil)
	if err != nil {
		fmt.Printf("error serving http: %v", err)
		return
	}
}

This has been fixed in the past: #3369 but looks like its back?

Also see: open-telemetry/opentelemetry-specification#3581