[Support] Need Help with Kubernetes and Trino Setup #21960

soumilshah1995 · 2024-05-13T17:32:03Z

soumilshah1995
May 13, 2024

Hello I am trying to run K8 and trino locally
and I am following the guide listed here https://trino.io/docs/current/installation/kubernetes.html

helm repo add trino https://trinodb.github.io/charts
helm install example-trino-cluster trino/trino

POD_NAME=$(kubectl get pods -l "app=trino,release=example-trino-cluster,component=coordinator" -o name)
kubectl port-forward $POD_NAME 8080:8080

Example.yaml

image:
  tag: "447"
server:
  workers: 3
coordinator:
  jvm:
    maxHeapSize: "8G"
worker:
  jvm:
    maxHeapSize: "8G"

helm upgrade example-trino-cluster trino/trino -f example.yaml

Not sure why I dont see workers here everything looks good to me not sure what stop im missing here

XuPengfei-1020 · 2024-05-14T06:35:50Z

XuPengfei-1020
May 14, 2024
Collaborator

Check whether the works pod has started up in k8s. kubectl get pods
Check whether the works has registered in discovery(The uri default as http(s)://$host:$port/v1/service)

0 replies

soumilshah1995 · 2024-05-14T12:31:13Z

soumilshah1995
May 14, 2024
Author

I tried following

image:
  tag: "400"
server:
  workers: 3
coordinator:
  jvm:
    maxHeapSize: "8G"
  coordinatorExtraConfig: |-
    internal-communication.shared-secret=whateveryouwant
worker:
  jvm:
    maxHeapSize: "8G"
  workerExtraConfig: |-
    internal-communication.shared-secret=whateveryouwant

#logs from worker pods

2024-05-14T12:28:19.116Z INFO main Bootstrap exchange.source-max-files-per-reader     25       25
2024-05-14T12:28:19.116Z INFO main Bootstrap exchange.max-output-partition-count      50       50
2024-05-14T12:28:19.116Z INFO main Bootstrap exchange.max-page-storage-size           16MB     16MB                                              Max storage size of a page written to a sink, including the page itself and its size represented as an int
2024-05-14T12:28:19.187Z INFO main io.airlift.bootstrap.LifeCycleManager Life cycle starting...
2024-05-14T12:28:19.187Z INFO main io.airlift.bootstrap.LifeCycleManager Life cycle started
2024-05-14T12:28:19.187Z INFO main io.trino.exchange.ExchangeManagerRegistry -- Loaded exchange manager filesystem --
2024-05-14T12:28:21.449Z INFO main io.trino.server.Server ======== SERVER STARTED ========
2024-05-14T12:28:21.449Z ERROR Discovery-0 io.airlift.discovery.client.CachingServiceSelector Cannot connect to discovery server for refresh (trino/general)
io.airlift.discovery.client.DiscoveryException: Lookup of trino failed for http://example-trino-cluster:8080
 at io.airlift.discovery.client.HttpDiscoveryLookupClient$DiscoveryResponseHandler.handleException(HttpDiscoveryLookupClient.java:218)
 at io.airlift.http.client.jetty.JettyResponseFuture.failed(JettyResponseFuture.java:120)
 at io.airlift.http.client.jetty.BufferingResponseListener.onComplete(BufferingResponseListener.java:85)
 at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:218)
 at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:210)
 at org.eclipse.jetty.client.HttpExchange.notifyFailureComplete(HttpExchange.java:276)
 at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:247)
 at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:164)
 at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:821)
 at org.eclipse.jetty.client.HttpDestination.abort(HttpDestination.java:559)
 at org.eclipse.jetty.client.HttpDestination.failed(HttpDestination.java:313)
 at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.failed(AbstractConnectionPool.java:572)
 at org.eclipse.jetty.client.HttpClient$1.failed(HttpClient.java:648)
 at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:161)
 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.TimeoutException: DNS timeout 5000 ms
 ... 7 more
2024-05-14T12:28:21.452Z ERROR Announcer-0 io.airlift.discovery.client.Announcer Cannot connect to discovery server for announce
io.airlift.discovery.client.DiscoveryException: Announcement failed for http://example-trino-cluster:8080
 at io.airlift.discovery.client.HttpDiscoveryAnnouncementClient$DiscoveryResponseHandler.handleException(HttpDiscoveryAnnouncementClient.java:195)
 at io.airlift.http.client.jetty.JettyResponseFuture.failed(JettyResponseFuture.java:120)
 at io.airlift.http.client.jetty.BufferingResponseListener.onComplete(BufferingResponseListener.java:85)
 at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:218)
 at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:210)
 at org.eclipse.jetty.client.HttpExchange.notifyFailureComplete(HttpExchange.java:276)
 at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:247)
 at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:164)
 at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:821)
 at org.eclipse.jetty.client.HttpDestination.abort(HttpDestination.java:559)
 at org.eclipse.jetty.client.HttpDestination.failed(HttpDestination.java:313)
 at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.failed(AbstractConnectionPool.java:572)
 at org.eclipse.jetty.client.HttpClient$1.failed(HttpClient.java:648)
 at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:161)
 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.TimeoutException: DNS timeout 5000 ms
 ... 7 more
Logs from May 14, 2024 to May 14, 2024 UTC

Still does not work

1 reply

nineinchnick May 14, 2024
Collaborator

The workers can't resolve the example-trino-cluster host. Can you check if a example-trino-cluster k8s service exists?

Do you use any custom DNS configuration in your Kubernetes cluster?

soumilshah1995 · 2024-05-14T12:42:45Z

soumilshah1995
May 14, 2024
Author

tried


image:
  tag: "400"
server:
  workers: 3
  workerExtraConfig: |-
    internal-communication.shared-secret=internal-access
  coordinatorExtraConfig: |-
    internal-communication.shared-secret=internal-access

worker logs

2024-05-14T12:28:39.666Z INFO main io.airlift.bootstrap.LifeCycleManager Life cycle starting...
2024-05-14T12:28:39.666Z INFO main io.airlift.bootstrap.LifeCycleManager Life cycle started
2024-05-14T12:28:39.666Z INFO main io.trino.exchange.ExchangeManagerRegistry -- Loaded exchange manager filesystem --
2024-05-14T12:28:41.105Z INFO main io.trino.server.Server ======== SERVER STARTED ========
2024-05-14T12:28:41.107Z ERROR Announcer-0 io.airlift.discovery.client.Announcer Cannot connect to discovery server for announce
io.airlift.discovery.client.DiscoveryException: Announcement failed for http://example-trino-cluster:8080
 at io.airlift.discovery.client.HttpDiscoveryAnnouncementClient$DiscoveryResponseHandler.handleException(HttpDiscoveryAnnouncementClient.java:195)
 at io.airlift.http.client.jetty.JettyResponseFuture.failed(JettyResponseFuture.java:120)
 at io.airlift.http.client.jetty.BufferingResponseListener.onComplete(BufferingResponseListener.java:85)
 at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:218)
 at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:210)
 at org.eclipse.jetty.client.HttpExchange.notifyFailureComplete(HttpExchange.java:276)
 at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:247)
 at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:164)
 at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:821)
 at org.eclipse.jetty.client.HttpDestination.abort(HttpDestination.java:559)
 at org.eclipse.jetty.client.HttpDestination.failed(HttpDestination.java:313)
 at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.failed(AbstractConnectionPool.java:572)
 at org.eclipse.jetty.client.HttpClient$1.failed(HttpClient.java:648)
 at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:192)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
 at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.UnknownHostException: example-trino-cluster
 at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
 at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:887)
 at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519)
 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1377)
 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1305)
 at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:170)
 ... 3 more
2024-05-14T12:28:41.109Z ERROR Discovery-0 io.airlift.discovery.client.CachingServiceSelector Cannot connect to discovery server for refresh (trino/general)
io.airlift.discovery.client.DiscoveryException: Lookup of trino failed for http://example-trino-cluster:8080
 at io.airlift.discovery.client.HttpDiscoveryLookupClient$DiscoveryResponseHandler.handleException(HttpDiscoveryLookupClient.java:218)
 at io.airlift.http.client.jetty.JettyResponseFuture.failed(JettyResponseFuture.java:120)
 at io.airlift.http.client.jetty.BufferingResponseListener.onComplete(BufferingResponseListener.java:85)
 at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:218)
 at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:210)
 at org.eclipse.jetty.client.HttpExchange.notifyFailureComplete(HttpExchange.java:276)
 at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:247)
 at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:164)
 at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:821)
 at org.eclipse.jetty.client.HttpDestination.abort(HttpDestination.java:559)
 at org.eclipse.jetty.client.HttpDestination.failed(HttpDestination.java:313)
 at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.failed(AbstractConnectionPool.java:572)
 at org.eclipse.jetty.client.HttpClient$1.failed(HttpClient.java:648)
 at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:192)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
 at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.UnknownHostException: example-trino-cluster: Temporary failure in name resolution
 at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
 at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:933)
 at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529)
 at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
 at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519)
 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1377)
 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1305)
 at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:170)
 ... 3 more
Logs from May 14, 2024 to May 14, 2024 UTC

0 replies

soumilshah1995 · 2024-05-14T12:44:16Z

soumilshah1995
May 14, 2024
Author

0 replies

soumilshah1995 · 2024-05-14T12:54:32Z

soumilshah1995
May 14, 2024
Author

Slack Thread https://trinodb.slack.com/archives/CP1MUNEUX/p1715619267445559

0 replies

soumilshah1995 · 2024-05-14T16:48:59Z

soumilshah1995
May 14, 2024
Author

got it resolved had to restart k8

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Support] Need Help with Kubernetes and Trino Setup #21960

{{title}}

Replies: 6 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[Support] Need Help with Kubernetes and Trino Setup #21960

soumilshah1995 May 13, 2024

Replies: 6 comments · 1 reply

XuPengfei-1020 May 14, 2024 Collaborator

soumilshah1995 May 14, 2024 Author

nineinchnick May 14, 2024 Collaborator

soumilshah1995 May 14, 2024 Author

soumilshah1995 May 14, 2024 Author

soumilshah1995 May 14, 2024 Author

soumilshah1995 May 14, 2024 Author

soumilshah1995
May 13, 2024

Replies: 6 comments 1 reply

XuPengfei-1020
May 14, 2024
Collaborator

soumilshah1995
May 14, 2024
Author

nineinchnick May 14, 2024
Collaborator

soumilshah1995
May 14, 2024
Author

soumilshah1995
May 14, 2024
Author

soumilshah1995
May 14, 2024
Author

soumilshah1995
May 14, 2024
Author