[BUG] error substring not found in metrics scraper

[aexvir]

Agent Environment
version: 7.49.1
os: linux (container os)
cloud: gcp
orchestrator: kubernetes

Describe what happened:
during the night some of our pods would stop being scraped for metrics, and on the fleet management ui I can see some broken openmetrics integrations, all of which show the same error: substring not found in prometheus_client/parser.py function _parse_sample

full stacktrace

Error: substring not found
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/prometheus_client/parser.py", line 115, in _parse_sample
label_start, label_end = text.index("{"), text.rindex("}")
ValueError: substring not found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/base.py", line 1235, in run
self.check(instance)
File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 142, in check
self.process(scraper_config)
File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 581, in process
for metric in self.scrape_metrics(scraper_config):
File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 542, in scrape_metrics
for metric in self.parse_metric_family(response, scraper_config):
File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 466, in parse_metric_family
for metric in text_fd_to_metric_families(input_gen):
File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/libs/prometheus.py", line 76, in text_fd_to_metric_families
sample = _parse_sample(line)
File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/prometheus_client/parser.py", line 130, in _parse_sample
name_end = text.index(separator)
ValueError: substring not found

Describe what you expected:
metrics

Steps to reproduce the issue:
not sure what on our metrics could be triggering this, can't see that in the ui and we don't have logs persisted at the moment
there has been no significant release on our end in the past two days, when this issue started happening

[aexvir]

small update here, issue found, we had a couple metrics with a tag that could be specified by end users, and some unneeded garbage ended up causing this; we've added validation for this, but I still think that if a metric fails to parse the other ones should still make it, so there probably should be some work to handle this more gracefully and avoid breaking the whole integration

we suspect it's probably the following line

api_hits{endpoint="/something",market="'\"()$@--%23"}

we've gotten to points where only 10% of our pods were being scraped because of this issue