Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add avgRTT to NFS operation stats #2633

Closed
wants to merge 1 commit into from
Closed

add avgRTT to NFS operation stats #2633

wants to merge 1 commit into from

Conversation

ksankeerth
Copy link

Hi Team/ @discordianfish,

Please check this PR and let me know if anything has to be changed.
Fixes #2550

We have to merge prometheus/procfs#487 before merging this PR.

Thanks

@ksankeerth
Copy link
Author

Build and tests failed because of missing field in struct. Once we merge procfs PR, I guess these errors won't come.

@ksankeerth
Copy link
Author

Hi Team/ @discordianfish,

Since this PR is merged prometheus/procfs#487, Can you please run CI again?

@discordianfish
Copy link
Member

@ksankeerth You need to update the procfs dependency here first

SuperQ
SuperQ requested changes May 24, 2023
View reviewed changes
Copy link
Member

@SuperQ SuperQ left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a rebase in order to pick up the procfs changes.

collector/mountstats_linux.go:760:15: op.AverageRTTMilliseconds undefined (type procfs.NFSOperationStats has no field or method AverageRTTMilliseconds)

@ethercflow
Copy link

@ksankeerth @discordianfish @SuperQ Sorry to bother you, any plan to merge this?

@discordianfish
Copy link
Member

I've rebased, let see

@discordianfish
Copy link
Member

@ksankeerth Looks like this still needs work

@ksankeerth
Copy link
Author

@ksankeerth Looks like this still needs work

Sorry for missing this. I'll update tonight

@ksankeerth
Copy link
Author

It seems, there are some e2e test failures. I'm looking at them.

@ksankeerth
Copy link
Author

Hi @discordianfish / @SuperQ ,

I tested locally and can see NFS mountstats that I've added. But e2e tests fail because of old fixtures. Can You help me to update fixtures?

I've tried to update fixtures using command make update_fixtures but it doesn't update fixtures with new metrics.

 # HELP node_mountstats_nfs_operations_average_round_trip_time_seconds Duration from the time that the client's kernel sends the RPC request until the time it receives the reply.
# TYPE node_mountstats_nfs_operations_average_round_trip_time_seconds counter
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="ACCESS",protocol="tcp"} 0.003
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="ALLOCATE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="BIND_CONN_TO_SESSION",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CLONE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CLOSE",protocol="tcp"} 0.00075
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="COMMIT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="COPY",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="COPY_NOTIFY",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CREATE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CREATE_SESSION",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DEALLOCATE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DELEGRETURN",protocol="tcp"} 0.0006666666666666666
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DESTROY_CLIENTID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DESTROY_SESSION",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="EXCHANGE_ID",protocol="tcp"} 0.0015
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FREE_STATEID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FSID_PRESENT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FSINFO",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FS_LOCATIONS",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETACL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETATTR",protocol="tcp"} 0.0009210526315789473
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETDEVICEINFO",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETDEVICELIST",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETXATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GET_LEASE_TIME",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTCOMMIT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTERROR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTGET",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTRETURN",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTSTATS",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LINK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LISTXATTRS",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOCK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOCKT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOCKU",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOOKUP",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOOKUPP",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOOKUP_ROOT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="NULL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OFFLOAD_CANCEL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN",protocol="tcp"} 0.0038333333333333336
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN_CONFIRM",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN_DOWNGRADE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN_NOATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="PATHCONF",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READ",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READDIR",protocol="tcp"} 0.002
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READLINK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READ_PLUS",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RECLAIM_COMPLETE",protocol="tcp"} 0.009
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RELEASE_LOCKOWNER",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="REMOVE",protocol="tcp"} 0.0036666666666666666
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="REMOVEXATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RENAME",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RENEW",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SECINFO",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SECINFO_NO_NAME",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SEEK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SEQUENCE",protocol="tcp"} 0.003142857142857143
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SERVER_CAPS",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETACL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETATTR",protocol="tcp"} 0.00425
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETCLIENTID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETCLIENTID_CONFIRM",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETXATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="STATFS",protocol="tcp"} 0.00175
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SYMLINK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="TEST_STATEID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="WRITE",protocol="tcp"} 0.0065

Thanks,
Sankeerthan

@ksankeerth ksankeerth marked this pull request as draft March 27, 2024 03:40
@ksankeerth
This is a combination of 4 commits.
b6415d7 
add avgRTT to NFS operation stats

Signed-off-by: shankeerthan-kasilingam <shankeerthan1995@gmail.com>
@ksankeerth ksankeerth marked this pull request as ready for review March 27, 2024 03:47
@ksankeerth
Copy link
Author

Hi Team,

Updated PR with required changes. Please check and let me know if more changes have to be done.

Thanks

@SuperQ
Copy link
Member

SuperQ commented Mar 28, 2024

I was just reviewing this, and the original procfs PR.

This is not a metric we can accept, as it is a computation against two metrics we already expose.

node_mountstats_operations_response_time_seconds_total / node_mountstats_operations_requests_total

Worse, this is actually an invalid and misleading new metric, because it produces a "since beginning of time" average, rather than the current average response time which is provided when you do the correct PromQL:

rate(node_mountstats_operations_response_time_seconds_total[1m]) / rate(node_mountstats_operations_requests_total[1m])

@SuperQ SuperQ closed this Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SuperQ SuperQ SuperQ requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

avg RTT in mountstats metrics
4 participants
@ksankeerth @discordianfish @ethercflow @SuperQ