Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CalicoNodeStatus can't be updated when etcd performance is (temporarily) degraded #8715

Open
alexarefev opened this issue Apr 12, 2024 · 4 comments
Open

CalicoNodeStatus can't be updated when etcd performance is (temporarily) degraded #8715

alexarefev opened this issue Apr 12, 2024 · 4 comments

Comments

@alexarefev
Copy link

Expected Behavior

CalicoNodeStatus resource is updated according to updatePeriodSeconds option

Current Behavior

calico-node stops updating CalicoNodeStatus resource due to the error:

2024-04-12T02:37:59.298523499Z stdout F 2024-04-12 02:37:59.297 [ERROR][246] status-reporter/customresource.go 179: Error updating resource Key=CalicoNodeStatus(support-master-left-1) Name="support-master-left-1" Resource="CalicoNodeStatuses" Value=&v3.CalicoNodeStatus{TypeMeta:v1.TypeMeta{Kind:"CalicoNodeStatus", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"support-master-left-1", GenerateName:"", Namespace:"", SelfLink:"", UID:"d28291b6-6bcf-4c44-bd82-c2866b244e4a", ResourceVersion:"280175376", Generation:0, CreationTimestamp:time.Date(2023, time.September, 13, 14, 48, 15, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubectl-create", Operation:"Update", APIVersion:"projectcalico.org/v3", Time:time.Date(2023, time.September, 13, 14, 48, 15, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0007f99b0), Subresource:""}}}, Spec:v3.CalicoNodeStatusSpec{Node:"support-master-left-1", Classes:[]v3.NodeStatusClassType{"Agent", "BGP", "Routes"}, UpdatePeriodSeconds:(*uint32)(0xc000962a5c)}, Status:v3.CalicoNodeStatusStatus{LastUpdated:time.Date(2024, time.April, 12, 2, 37, 52, 284751692, time.Local), Agent:v3.CalicoNodeAgentStatus{BIRDV4:v3.BGPDaemonStatus{State:"Ready", Version:"v0.3.3+birdv1.6.8", RouterID:"192.168.1.4", LastBootTime:"2024-04-04 09:13:33", LastReconfigurationTime:"2024-04-04 09:13:33"}, BIRDV6:v3.BGPDaemonStatus{State:"Ready", Version:"v0.3.3+birdv1.6.8", RouterID:"192.168.1.4", LastBootTime:"2024-04-04 09:13:34", LastReconfigurationTime:"2024-04-04 09:13:34"}}, BGP:v3.CalicoNodeBGPStatus{NumberEstablishedV4:7, NumberNotEstablishedV4:6, NumberEstablishedV6:0, NumberNotEstablishedV6:0, PeersV4:[]v3.CalicoNodePeer{v3.CalicoNodePeer{PeerIP:"192.168.1.8", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.16", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.11", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.22", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.17", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.12", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.1", Type:"GlobalPeer", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.11", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.12", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.16", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.17", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.22", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.8", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}}, PeersV6:[]v3.CalicoNodePeer{}}, Routes:v3.CalicoNodeBGPRouteStatus{RoutesV4:[]v3.CalicoNodeRoute{v3.CalicoNodeRoute{Type:"FIB", Destination:"0.0.0.0/0", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Kernel", PeerIP:""}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"169.254.169.254/32", Gateway:"192.168.1.3", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Kernel", PeerIP:""}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"172.31.162.221/32", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.128.138.230/32", Gateway:"N/A", Interface:"calibe5d6b6261d", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Kernel", PeerIP:""}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.135.60.192/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.128.51.192/26", Gateway:"192.168.1.11", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.11"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.128.51.192/26", Gateway:"192.168.1.11", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.130.67.0/26", Gateway:"192.168.1.8", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.8"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.130.67.0/26", Gateway:"192.168.1.8", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.128.72.192/26", Gateway:"192.168.1.12", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.12"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.128.72.192/26", Gateway:"192.168.1.12", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.133.88.64/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"192.168.0.0/24", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.130.99.192/26", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.17"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.130.99.192/26", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"192.168.1.0/24", Gateway:"N/A", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Direct", PeerIP:""}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"192.168.1.0/24", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"192.168.2.0/24", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.130.100.0/26", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.17"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.130.100.0/26", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.128.138.192/26", Gateway:"N/A", Interface:"blackhole", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Static", PeerIP:""}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.128.138.192/32", Gateway:"N/A", Interface:"tunl0", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Direct", PeerIP:""}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.131.168.64/26", Gateway:"192.168.1.22", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.22"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.131.168.64/26", Gateway:"192.168.1.22", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.130.183.64/26", Gateway:"192.168.1.16", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.16"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.130.183.64/26", Gateway:"192.168.1.16", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.132.160.192/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.134.168.64/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"172.30.251.59/32", Gateway:"192.168.1.11", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.11"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.251.59/32", Gateway:"192.168.1.11", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.251.59/32", Gateway:"192.168.1.12", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.12"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.251.59/32", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.17"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.251.59/32", Gateway:"192.168.1.22", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.22"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"172.31.0.0/16", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"172.30.0.0/16", Gateway:"N/A", Interface:"blackhole", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Static", PeerIP:""}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.12", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.12"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.17"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.11", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.11"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.22", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.22"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.8", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.8"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.16", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.16"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.135.243.0/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.134.252.0/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.133.226.192/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}}, RoutesV6:[]v3.CalicoNodeRoute{}}}} error=etcdserver: request timed out
2024-04-12T02:37:59.298780451Z stdout F 2024-04-12 02:37:59.298 [WARNING][246] status-reporter/reporter.go 235: Failed to update node status resource; will retry error=etcdserver: request timed out
2024-04-12T02:38:02.492958127Z stdout F 2024-04-12 02:38:02.492 [ERROR][246] status-reporter/customresource.go 179: Error updating resource Key=CalicoNodeStatus(support-master-left-1) Name="support-master-left-1" Resource="CalicoNodeStatuses" Value=&v3.CalicoNodeStatus{TypeMeta:v1.TypeMeta{Kind:"CalicoNodeStatus", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"support-master-left-1", GenerateName:"", Namespace:"", SelfLink:"", UID:"d28291b6-6bcf-4c44-bd82-c2866b244e4a", ResourceVersion:"280175376", Generation:0, CreationTimestamp:time.Date(2023, time.September, 13, 14, 48, 15, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubectl-create", Operation:"Update", APIVersion:"projectcalico.org/v3", Time:time.Date(2023, time.September, 13, 14, 48, 15, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0007f99b0), Subresource:""}}}, Spec:v3.CalicoNodeStatusSpec{Node:"support-master-left-1", Classes:[]v3.NodeStatusClassType{"Agent", "BGP", "Routes"}, UpdatePeriodSeconds:(*uint32)(0xc000962a5c)}, Status:v3.CalicoNodeStatusStatus{LastUpdated:time.Date(2024, time.April, 12, 2, 38, 2, 345238739, time.Local), Agent:v3.CalicoNodeAgentStatus{BIRDV4:v3.BGPDaemonStatus{State:"Ready", Version:"v0.3.3+birdv1.6.8", RouterID:"192.168.1.4", LastBootTime:"2024-04-04 09:13:33", LastReconfigurationTime:"2024-04-04 09:13:33"}, BIRDV6:v3.BGPDaemonStatus{State:"Ready", Version:"v0.3.3+birdv1.6.8", RouterID:"192.168.1.4", LastBootTime:"2024-04-04 09:13:34", LastReconfigurationTime:"2024-04-04 09:13:34"}}, BGP:v3.CalicoNodeBGPStatus{NumberEstablishedV4:7, NumberNotEstablishedV4:6, NumberEstablishedV6:0, NumberNotEstablishedV6:0, PeersV4:[]v3.CalicoNodePeer{v3.CalicoNodePeer{PeerIP:"192.168.1.8", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.16", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.11", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.22", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.17", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.12", Type:"NodeMesh", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.1", Type:"GlobalPeer", State:"Established", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.11", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.12", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.16", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.17", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.22", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}, v3.CalicoNodePeer{PeerIP:"192.168.1.8", Type:"NodePeer", State:"Idle", Since:"2024-04-04"}}, PeersV6:[]v3.CalicoNodePeer{}}, Routes:v3.CalicoNodeBGPRouteStatus{RoutesV4:[]v3.CalicoNodeRoute{v3.CalicoNodeRoute{Type:"FIB", Destination:"0.0.0.0/0", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Kernel", PeerIP:""}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"169.254.169.254/32", Gateway:"192.168.1.3", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Kernel", PeerIP:""}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"172.31.162.221/32", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.128.138.230/32", Gateway:"N/A", Interface:"calibe5d6b6261d", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Kernel", PeerIP:""}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.135.60.192/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.128.51.192/26", Gateway:"192.168.1.11", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.11"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.128.51.192/26", Gateway:"192.168.1.11", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.130.67.0/26", Gateway:"192.168.1.8", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.8"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.130.67.0/26", Gateway:"192.168.1.8", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.128.72.192/26", Gateway:"192.168.1.12", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.12"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.128.72.192/26", Gateway:"192.168.1.12", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.133.88.64/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"192.168.0.0/24", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.130.99.192/26", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.17"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.130.99.192/26", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"192.168.1.0/24", Gateway:"N/A", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Direct", PeerIP:""}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"192.168.1.0/24", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"192.168.2.0/24", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.130.100.0/26", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.17"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.130.100.0/26", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.128.138.192/26", Gateway:"N/A", Interface:"blackhole", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Static", PeerIP:""}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.128.138.192/32", Gateway:"N/A", Interface:"tunl0", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Direct", PeerIP:""}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.131.168.64/26", Gateway:"192.168.1.22", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.22"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.131.168.64/26", Gateway:"192.168.1.22", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.130.183.64/26", Gateway:"192.168.1.16", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.16"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"10.130.183.64/26", Gateway:"192.168.1.16", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.132.160.192/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.134.168.64/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"172.30.251.59/32", Gateway:"192.168.1.11", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.11"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.251.59/32", Gateway:"192.168.1.11", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.251.59/32", Gateway:"192.168.1.12", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.12"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.251.59/32", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.17"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.251.59/32", Gateway:"192.168.1.22", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.22"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"172.31.0.0/16", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"172.30.0.0/16", Gateway:"N/A", Interface:"blackhole", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"Static", PeerIP:""}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.12", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.12"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.17", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.17"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.11", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.11"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.22", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.22"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.8", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.8"}}, v3.CalicoNodeRoute{Type:"RIB", Destination:"172.30.0.0/16", Gateway:"192.168.1.16", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"NodeMesh", PeerIP:"192.168.1.16"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.135.243.0/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.134.252.0/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}, v3.CalicoNodeRoute{Type:"FIB", Destination:"10.133.226.192/26", Gateway:"192.168.1.1", Interface:"ens3", LearnedFrom:v3.CalicoNodeRouteLearnedFrom{SourceType:"BGPPeer", PeerIP:"192.168.1.1"}}}, RoutesV6:[]v3.CalicoNodeRoute{}}}} error=Operation cannot be fulfilled on caliconodestatuses.crd.projectcalico.org "support-master-left-1": the object has been modified; please apply your changes to the latest version and try again

# kubectl get caliconodestatus
NAME                    NODE                    CLASSES            UPDATE INTERVAL   AGE    LAST UPDATED
support-master-left-1   support-master-left-1   Agent,BGP,Routes   10s               211d   8h ago
support-master-left-2   support-master-left-2   Agent,BGP,Routes   10s               211d   4s ago
support-master-left-3   support-master-left-3   Agent,BGP,Routes   10s               211d   4s ago
support-worker-left-1   support-worker-left-1   Agent,BGP,Routes   10s               211d   2s ago
support-worker-left-2   support-worker-left-2   Agent,BGP,Routes   10s               211d   4s ago
support-worker-left-3   support-worker-left-3   Agent,BGP,Routes   10s               211d   6s ago
support-worker-left-4   support-worker-left-4   Agent,BGP,Routes   10s               211d   6s ago

As far as I understand the issue is initiated by temporary ETCD performance degradation and the issue doesn't recover after the ETCD performance recovery.

Possible Solution

Reconciliation loop should process the Operation cannot be fulfilled... error

Steps to Reproduce

I'm not sure that is so easy

  1. Create CalicoNodeStatus resource
  2. Invoke ETCD slowdown

Context

We are using CalicoNodeStatus as a source for external BGP sessions monitoring

Your Environment

  • Calico v3.26.1
  • Kubernetes v1.26.7
  • Ubuntu 22.04
@caseydavenport
Copy link
Member

the object has been modified; please apply your changes to the latest version and try again

This error tends to mean that the object in the API has been modified by another client between when Calico queried the resource and when Calico attempted to update the resource. Looking at that code, it appears like it does attempt to refresh its internal state when it sees CalicoNodeStatus changes, so I would expect this case to be handled. Do you know if there is another entity that is modifying these resources?

Invoke ETCD slowdown

Are you doing something in particular to trigger this?

@alexarefev
Copy link
Author

Hi @caseydavenport
Thank you for response

the object has been modified; please apply your changes to the latest version and try again

This error tends to mean that the object in the API has been modified by another client between when Calico queried the resource and when Calico attempted to update the resource. Looking at that code, it appears like it does attempt to refresh its internal state when it sees CalicoNodeStatus changes, so I would expect this case to be handled. Do you know if there is another entity that is modifying these resources?

It seems there is no any controller that might change CalicoNodeStatus except the calico-node. At least, I can't find anything suspicious in the kube-apiserver logs

Invoke ETCD slowdown

Are you doing something in particular to trigger this?

Decrease ETCD storage IOPS, if it's possible

@coutinhop coutinhop changed the title CalicoNodeStatus can't be updated CalicoNodeStatus can't be updated when etcd performance is (temporarily) degraded May 7, 2024
@alexarefev
Copy link
Author

Hi
The ETCD storage IOPS could be reduced in the following manner:

  1. Get major and minor numbers of block device where etcd data is located
  2. Find out the etcd PID
  3. Get the cgroup path of etcd
  4. Limit the write bytes per second for that cgroup

For Ubuntu 22.04 it's something similar to that one

cat /proc/partitions
ETCD_PID=$(ps -ef | grep 'etcd ' | grep -v 'grep' | awk '{ print $2 }')
CGROUP=$(cat /proc/${ETCD_PID}/cgroup | sed 's/^.\+:://')
echo "<Major>:<Minor> wbps=1024" > /sys/fs/cgroup/${CGROUP}/io.max

@caseydavenport
Copy link
Member

Degraded etcd performance is going to impact every Kubernetes API.

The interesting piece is that it doesn't recover after etcd is functioning again, which suggests we might not be handling an error case correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@caseydavenport @alexarefev