Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scrape timeout with many disks #197

Open
pokab opened this issue Jan 20, 2024 · 2 comments
Open

Scrape timeout with many disks #197

pokab opened this issue Jan 20, 2024 · 2 comments

Comments

@pokab
Copy link

pokab commented Jan 20, 2024 •
edited

I manage a Linux box with around 20-25 HDDs. Some of these disks are faster to reply to smartctl, others are pretty slow, taking around 1-2 seconds.
Setting the scrape interval to 60 seconds and scrape timeout to 40 seconds does not help in avoiding regular scrape timeouts.
A previous solution (not specific to Prometheus) I made spawned the smartctl subprocesses in parallel for all the HDDs, and it works perfectly. Would a solution like this be appropriate for this software? Maybe with an option to enable or disable it?
@SuperQ
Copy link
Contributor

SuperQ commented Jan 22, 2024

This should be possible here with a goroutine worker pool. We can parallellize the data collection.

It doesn't look like the current collector records any timing information, even in debug mode. Something we can improve as well.

SuperQ added a commit that referenced this issue Jan 22, 2024
@SuperQ
Update json reading logging
e21dbef 
Update the smartctl command reading and parsing of json logging to make
for easier debugging of slow devices by adding a duration to the debug
logging.

For #197

Signed-off-by: SuperQ <superq@gmail.com>
@SuperQ SuperQ mentioned this issue Jan 22, 2024
Merged
SuperQ added a commit that referenced this issue Jan 24, 2024
@SuperQ
Update json reading logging
32de524 
Update the smartctl command reading and parsing of json logging to make
for easier debugging of slow devices by adding a duration to the debug
logging.

For #197

Signed-off-by: SuperQ <superq@gmail.com>
SuperQ added a commit that referenced this issue Jan 25, 2024
@SuperQ
Update json reading logging (#198)
77e6b07 
Update the smartctl command reading and parsing of json logging to make
for easier debugging of slow devices by adding a duration to the debug
logging.

For #197

Signed-off-by: SuperQ <superq@gmail.com>
pokab added a commit to idatahu/smartctl_exporter that referenced this issue Feb 27, 2024
@pokab
Use worker pool to make running of smartctl parallel (for issue prome…
19555c3 
…theus-community#197).
@pokab pokab mentioned this issue Feb 27, 2024
Open
pokab added a commit to idatahu/smartctl_exporter that referenced this issue Feb 27, 2024
@pokab
Use worker pool to make running of smartctl parallel (for issue prome…
488ba2b 
…theus-community#197).

Signed-off-by: Póka Balázs <poka@idata.hu>
zxzharmlesszxz pushed a commit to zxzharmlesszxz/smartctl_exporter that referenced this issue Mar 5, 2024
@SuperQ @zxzharmlesszxz
Update json reading logging (prometheus-community#198)
6b5f02b 
Update the smartctl command reading and parsing of json logging to make
for easier debugging of slow devices by adding a duration to the debug
logging.

For prometheus-community#197

Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: mort <mort@syneforge.com>
zxzharmlesszxz pushed a commit to zxzharmlesszxz/smartctl_exporter that referenced this issue Mar 5, 2024
@SuperQ @zxzharmlesszxz
Update json reading logging (prometheus-community#198)
5b02aa0 
Update the smartctl command reading and parsing of json logging to make
for easier debugging of slow devices by adding a duration to the debug
logging.

For prometheus-community#197

Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: Denys <zxzharmlesszxz@gmail.com>
@pokab
Copy link
Author

pokab commented Mar 19, 2024

I've been using my forked version (see #204) for three weeks without any problem. It successfully solved the scrape issue. Could you please give some feedback?

pokab added a commit to idatahu/smartctl_exporter that referenced this issue May 9, 2024
@pokab
Use worker pool to make running of smartctl parallel (for issue prome…
f25870d 
…theus-community#197).

Signed-off-by: Póka Balázs <poka@idata.hu>
@pokab pokab mentioned this issue May 10, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@SuperQ @pokab