Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent data for bricks metrics #175

Open
VLZZZ opened this issue Dec 26, 2019 · 0 comments
Open

Inconsistent data for bricks metrics #175

VLZZZ opened this issue Dec 26, 2019 · 0 comments

Comments

@VLZZZ
Copy link
Contributor

VLZZZ commented Dec 26, 2019

Problem

GlusterFS volume with 6 bricks (Replicated on 3 nodes).
After resizing the volume using heketi-cli for each node additional brick was added with different size.
Anyway, when exporting metrics there's duplicated brick_pids and no metrics for bunch of bricks in volume.

I though that it's the same problem described in #168 but we use the version with hotfix already.

Information

gluster volume info vol_1f6bbd7c2d17834cabc5641414de13d1 :

Volume Name: vol_1f6bbd7c2d17834cabc5641414de13d1
Type: Distributed-Replicate
Volume ID: 3f1f97e1-b3d4-4170-a4ac-772c3dbc62a1
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.134.250:/var/lib/heketi/mounts/vg_0671e452b8dcd045a69f029827ab1fa2/brick_4f4a02fa038f0457c60b44b7460fe8fa/brick
Brick2: 10.70.134.195:/var/lib/heketi/mounts/vg_ce7255e7dd1922ab0ceff493c4451e4f/brick_a7ae30e683d839bf9f5f4e990f97c62c/brick
Brick3: 10.70.134.254:/var/lib/heketi/mounts/vg_f1a0d56f912d9557548602568aa0323e/brick_f42dbabd011649c8db2af17c34ba1309/brick
Brick4: 10.70.134.195:/var/lib/heketi/mounts/vg_ce7255e7dd1922ab0ceff493c4451e4f/brick_ac6f7d0f3de2d7c80ffaa9023c55375f/brick
Brick5: 10.70.134.254:/var/lib/heketi/mounts/vg_f1a0d56f912d9557548602568aa0323e/brick_36c97c15e0476981d1553345a962a16b/brick
Brick6: 10.70.134.250:/var/lib/heketi/mounts/vg_0671e452b8dcd045a69f029827ab1fa2/brick_f1c14b147b8c26dceef726a87a2d7c11/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

Note 6 different bricks pids in the status output:

gluster volume status vol_1f6bbd7c2d17834cabc5641414de13d1:

Status of volume: vol_1f6bbd7c2d17834cabc5641414de13d1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.134.250:/var/lib/heketi/mounts/
vg_0671e452b8dcd045a69f029827ab1fa2/brick_4
f4a02fa038f0457c60b44b7460fe8fa/brick       49156     0          Y       266
Brick 10.70.134.195:/var/lib/heketi/mounts/
vg_ce7255e7dd1922ab0ceff493c4451e4f/brick_a
7ae30e683d839bf9f5f4e990f97c62c/brick       49172     0          Y       264
Brick 10.70.134.254:/var/lib/heketi/mounts/
vg_f1a0d56f912d9557548602568aa0323e/brick_f
42dbabd011649c8db2af17c34ba1309/brick       49156     0          Y       267
Brick 10.70.134.195:/var/lib/heketi/mounts/
vg_ce7255e7dd1922ab0ceff493c4451e4f/brick_a
c6f7d0f3de2d7c80ffaa9023c55375f/brick       49173     0          Y       273
Brick 10.70.134.254:/var/lib/heketi/mounts/
vg_f1a0d56f912d9557548602568aa0323e/brick_3
6c97c15e0476981d1553345a962a16b/brick       49157     0          Y       276
Brick 10.70.134.250:/var/lib/heketi/mounts/
vg_0671e452b8dcd045a69f029827ab1fa2/brick_f
1c14b147b8c26dceef726a87a2d7c11/brick       49157     0          Y       273
Self-heal Daemon on localhost               N/A       N/A        Y       196
Self-heal Daemon on 10.70.134.195           N/A       N/A        Y       193
Self-heal Daemon on 10.70.134.250           N/A       N/A        Y       193

Task Status of Volume vol_1f6bbd7c2d17834cabc5641414de13d1
------------------------------------------------------------------------------
Task                 : Rebalance
ID                   : 398d867f-dcbd-4bc6-8c91-bf1eb46b84dd
Status               : completed

gluster volume status vol_1f6bbd7c2d17834cabc5641414de13d1 detail:

Status of volume: vol_1f6bbd7c2d17834cabc5641414de13d1
------------------------------------------------------------------------------
Brick                : Brick 10.70.134.250:/var/lib/heketi/mounts/vg_0671e452b8dcd045a69f029827ab1fa2/brick_4f4a02fa038f0457c60b44b7460fe8fa/brick
TCP Port             : 49156
RDMA Port            : 0
Online               : Y
Pid                  : 266
File System          : xfs
Device               : /dev/mapper/vg_0671e452b8dcd045a69f029827ab1fa2-brick_4f4a02fa038f0457c60b44b7460fe8fa
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=64,swidth=512,noquota
Inode Size           : 512
Disk Space Free      : 8.5GB
Total Disk Space     : 22.0GB
Inode Count          : 11534080
Free Inodes          : 11493473
------------------------------------------------------------------------------
Brick                : Brick 10.70.134.195:/var/lib/heketi/mounts/vg_ce7255e7dd1922ab0ceff493c4451e4f/brick_a7ae30e683d839bf9f5f4e990f97c62c/brick
TCP Port             : 49172
RDMA Port            : 0
Online               : Y
Pid                  : 264
File System          : xfs
Device               : /dev/mapper/vg_ce7255e7dd1922ab0ceff493c4451e4f-brick_a7ae30e683d839bf9f5f4e990f97c62c
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=64,swidth=512,noquota
Inode Size           : 512
Disk Space Free      : 9.0GB
Total Disk Space     : 22.0GB
Inode Count          : 11534080
Free Inodes          : 11493398
------------------------------------------------------------------------------
Brick                : Brick 10.70.134.254:/var/lib/heketi/mounts/vg_f1a0d56f912d9557548602568aa0323e/brick_f42dbabd011649c8db2af17c34ba1309/brick
TCP Port             : 49156
RDMA Port            : 0
Online               : Y
Pid                  : 267
File System          : xfs
Device               : /dev/mapper/vg_f1a0d56f912d9557548602568aa0323e-brick_f42dbabd011649c8db2af17c34ba1309
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=64,swidth=512,noquota
Inode Size           : 512
Disk Space Free      : 8.5GB
Total Disk Space     : 22.0GB
Inode Count          : 11534080
Free Inodes          : 11493397
------------------------------------------------------------------------------
Brick                : Brick 10.70.134.195:/var/lib/heketi/mounts/vg_ce7255e7dd1922ab0ceff493c4451e4f/brick_ac6f7d0f3de2d7c80ffaa9023c55375f/brick
TCP Port             : 49173
RDMA Port            : 0
Online               : Y
Pid                  : 273
File System          : xfs
Device               : /dev/mapper/vg_ce7255e7dd1922ab0ceff493c4451e4f-brick_ac6f7d0f3de2d7c80ffaa9023c55375f
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=64,swidth=512,noquota
Inode Size           : 512
Disk Space Free      : 470.1MB
Total Disk Space     : 5.0GB
Inode Count          : 970368
Free Inodes          : 962997
------------------------------------------------------------------------------
Brick                : Brick 10.70.134.254:/var/lib/heketi/mounts/vg_f1a0d56f912d9557548602568aa0323e/brick_36c97c15e0476981d1553345a962a16b/brick
TCP Port             : 49157
RDMA Port            : 0
Online               : Y
Pid                  : 276
File System          : xfs
Device               : /dev/mapper/vg_f1a0d56f912d9557548602568aa0323e-brick_36c97c15e0476981d1553345a962a16b
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=64,swidth=512,noquota
Inode Size           : 512
Disk Space Free      : 470.1MB
Total Disk Space     : 5.0GB
Inode Count          : 970376
Free Inodes          : 963005
------------------------------------------------------------------------------
Brick                : Brick 10.70.134.250:/var/lib/heketi/mounts/vg_0671e452b8dcd045a69f029827ab1fa2/brick_f1c14b147b8c26dceef726a87a2d7c11/brick
TCP Port             : 49157
RDMA Port            : 0
Online               : Y
Pid                  : 273
File System          : xfs
Device               : /dev/mapper/vg_0671e452b8dcd045a69f029827ab1fa2-brick_f1c14b147b8c26dceef726a87a2d7c11
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=64,swidth=512,noquota
Inode Size           : 512
Disk Space Free      : 470.1MB
Total Disk Space     : 5.0GB
Inode Count          : 970360
Free Inodes          : 962989

Then when I'm trying to receive metrics the only metric with consistent data is gluster_volume_status_brick_count which is showing 6 bricks

gluster_volume_brick_pid{hostname="10.70.134.195",instance="localhost",peerid="3f6b8a82-8988-42d7-a37c-5cf567a89eab",volume_name="vol_1f6bbd7c2d17834cabc5641414de13d1"} 273
gluster_volume_brick_pid{hostname="10.70.134.250",instance="localhost",peerid="d05b7c42-fd49-4a33-9a06-4a66e4079452",volume_name="vol_1f6bbd7c2d17834cabc5641414de13d1"} 273
gluster_volume_brick_pid{hostname="10.70.134.254",instance="localhost",peerid="07e29ad9-1247-42b2-9db8-512f7baaccce",volume_name="vol_1f6bbd7c2d17834cabc5641414de13d1"} 276
gluster_volume_brick_total_bytes{hostname="10.70.134.195",instance="localhost",peerid="3f6b8a82-8988-42d7-a37c-5cf567a89eab",volume_name="vol_1f6bbd7c2d17834cabc5641414de13d1"} 5.357961216e+09
gluster_volume_brick_total_bytes{hostname="10.70.134.250",instance="localhost",peerid="d05b7c42-fd49-4a33-9a06-4a66e4079452",volume_name="vol_1f6bbd7c2d17834cabc5641414de13d1"} 5.357961216e+09
gluster_volume_brick_total_bytes{hostname="10.70.134.254",instance="localhost",peerid="07e29ad9-1247-42b2-9db8-512f7baaccce",volume_name="vol_1f6bbd7c2d17834cabc5641414de13d1"} 5.357961216e+09
gluster_volume_brick_total_inodes{hostname="10.70.134.195",instance="localhost",peerid="3f6b8a82-8988-42d7-a37c-5cf567a89eab",volume_name="vol_1f6bbd7c2d17834cabc5641414de13d1"} 970368
gluster_volume_brick_total_inodes{hostname="10.70.134.250",instance="localhost",peerid="d05b7c42-fd49-4a33-9a06-4a66e4079452",volume_name="vol_1f6bbd7c2d17834cabc5641414de13d1"} 970360
gluster_volume_brick_total_inodes{hostname="10.70.134.254",instance="localhost",peerid="07e29ad9-1247-42b2-9db8-512f7baaccce",volume_name="vol_1f6bbd7c2d17834cabc5641414de13d1"} 970376
gluster_volume_status_brick_count{instance="localhost",volume_name="vol_1f6bbd7c2d17834cabc5641414de13d1"} 6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant