[Bug] When restart oap got interruptedException

[crazychengmm]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

OAP server (apache/skywalking) 9.4

What happened

restart OAP Service failed

What you expected to happen

We reboot the OAP service, sometimes OAP would be start success, but 90% would get a interrupted exception.

Check the info log, it seems list pod, timeout with k8s api.
So i checked the skywalking source code, noticed that connect k8s using io.fabric8.kubernetes.client,
found timeout can change.

public static final String KUBERNETES_CONNECTION_TIMEOUT_SYSTEM_PROPERTY = "kubernetes.connection.timeout";
public static final String KUBERNETES_UPLOAD_REQUEST_TIMEOUT_SYSTEM_PROPERTY = "kubernetes.upload.request.timeout";
public static final String KUBERNETES_REQUEST_TIMEOUT_SYSTEM_PROPERTY = "kubernetes.request.timeout";

kubernetes.connection.timeout
kubernetes.request.timeout

Default value is 10000ms, try to set 15000ms, not work for me.

Here is my questions.

1. This issue is happed with restart the OAP
2. 2024-84-29 17:14:55, 599 io.kubernetes.client.informer.cache.ProcessorListener 96 ［pool-6-thread-1) ERROR 11 - [9.4.9-52005311 processor

using io.fabric8.kubernetes.client ... why came out error with io.kubernetes.client?

How to reproduce

how to slove the issue.

Anything else

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[crazychengmm]

Noticed that 9.4 using client with io.kubernetes.client. I was checking wrong source code which is 8.9.

[crazychengmm]

About timeout, OkhttpClient is used by io.kubernetes.client, which is set by 10s.

If k8s is big enough, api return over 10s, may get error with time out.

[lujiajing1126]

Simply add timeout may not help. What about using limit?

https://kubernetes.io/docs/reference/using-api/api-concepts/#retrieving-large-results-sets-in-chunks

[crazychengmm]

Simply add timeout may not help. What about using limit?

https://kubernetes.io/docs/reference/using-api/api-concepts/#retrieving-large-results-sets-in-chunks

KubernetesPods.java

@SneakyThrows
    private KubernetesPods() {
        KubernetesClient.setDefault();

        final CoreV1Api coreV1Api = new CoreV1Api();
        final CacheBuilder<Object, Object> cacheBuilder =
            CacheBuilder.newBuilder()
                .expireAfterAccess(Duration.ofMinutes(5));

        podByIP = cacheBuilder.build(new CacheLoader<String, Optional<V1Pod>>() {
            @Override
            public Optional<V1Pod> load(String ip) throws Exception {
                return coreV1Api
                    .listPodForAllNamespaces(
                        null, null, String.format(FIELD_SELECTOR_PATTERN_POD_IP, ip),
                        null, null, null, null, null, null, null)
                    .getItems()
                    .stream()
                    .findFirst();
            }
        });

limit was not config, limit was set null.
If using limit should change the code.Is there any other idea, Thanks

[crazychengmm]

@wu-sheng @lujiajing1126 Big boss, 求翻牌

[wu-sheng]

I didn't face this, you could change codes to see whether it fixes.