node should not be ready when klog flush deamon in kubelet is block in fsync

[divanodestiny]

What happened?

1. some issues cause a part of fsync syscalls to persist in blocking
2. klog flush deamon hold the lock, call flushAll and block in fsync syscall

kubernetes/vendor/k8s.io/klog/v2/klog.go

Lines 1223 to 1228 in a309fad

	// lockAndFlushAll is like flushAll but locks l.mu first.
	func (l *loggingT) lockAndFlushAll() {
	l.mu.Lock()
	l.flushAll()
	l.mu.Unlock()
	}

4. other goroutines in kubelet will be block when print log
   

   kubernetes/vendor/k8s.io/klog/v2/klog.go

   Lines 885 to 894 in a309fad

   	// output writes the data to the log files and releases the buffer.
   	func (l *loggingT) output(s severity.Severity, logger *logWriter, buf *buffer.Buffer, depth int, file string, line int, alsoToStderr bool) {
   	var isLocked = true
   	l.mu.Lock()
   	defer func() {
   	if isLocked {
   	// Unlock before returning in case that it wasn't done already.
   	l.mu.Unlock()
   	}
   	}()

5. kubelet can renew node lease normally because this goroutine do not print log

6. node controller think this node is ready

What did you expect to happen?

node should be not ready

How can we reproduce it (as minimally and precisely as possible)?

maybe use ptrace to block fsync?

Anything else we need to know?

No response

Kubernetes version

1.26.0

$ kubectl version
# paste output here

Cloud provider

none

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[T-Lakshmi]

/sig node

[yuzhiquan]

I think this related to this issue, maybe we can wait for the solution on klog side.
Feel free to reopen this.
/close

[k8s-ci-robot]

@yuzhiquan: Closing this issue.

In response to this:

I think this related to this issue, maybe we can wait for the solution on klog side.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.