Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Session expired (The phenomenon of being shut down due to hbase Session expire) I can't solve it, so I ask for your help.. #10625

Open
1 task done
mingeonho1 opened this issue Jan 19, 2024 · 2 comments
Open
1 task done

Session expired (The phenomenon of being shut down due to hbase Session expire) I can't solve it, so I ask for your help.. #10625

mingeonho1 opened this issue Jan 19, 2024 · 2 comments

Comments

@mingeonho1
Copy link

mingeonho1 commented Jan 19, 2024 •
edited

Prerequisites

Please check the FAQ, and search existing issues for similar questions before creating a new issue.YOU MAY DELETE THIS PREREQUISITES SECTION.

  • I have checked the FAQ, and issues and found no answer.
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/ip-***.ap-northeast-2.compute.internal,37985,1705565641224
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:292)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
        at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:198)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:310)
        at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
        at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2201)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1085)
        at java.lang.Thread.run(Thread.java:750)
2024-01-18 17:42:46,053 ERROR [RS:0;ip-****:37985] zookeeper.ZooKeeperWatcher: regionserver:37985-0x18d1ba24ff50003, quorum=localhost:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/ip-****.ap-northeast-2.compute.internal,37985,1705565641224
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:292)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
        at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:198)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:310)
        at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
        at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2201)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1085)
        at java.lang.Thread.run(Thread.java:750)
2024-01-18 17:42:46,129 INFO  [RS:0;ip-****:37985] ipc.RpcServer: Stopping server on 37985
2024-01-18 17:42:46,129 INFO  [RpcServer.listener,port=37985] ipc.RpcServer: RpcServer.listener,port=37985: stopping
2024-01-18 17:42:46,130 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2024-01-18 17:42:46,130 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2024-01-18 17:42:46,134 WARN  [RS:0;ip-****:37985] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/ip-****.ap-northeast-2.compute.internal,37985,1705565641224
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1236)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1225)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1438)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1093)
        at java.lang.Thread.run(Thread.java:750)
2024-01-18 17:42:46,135 INFO  [RS:0;ip-****:37985] regionserver.HRegionServer: stopping server ip-****.ap-northeast-2.compute.internal,37985,1705565641224; zookeeper connection closed.
2024-01-18 17:42:46,135 INFO  [RS:0;ip-****:37985] regionserver.HRegionServer: RS:0;ip-****:37985 exiting
2024-01-18 17:42:46,140 INFO  [Thread-5] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@436c81a3
2024-01-18 17:42:46,141 INFO  [Thread-5] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2024-01-18 17:42:46,141 INFO  [Thread-5] regionserver.ShutdownHook: Shutdown hook finished.

https://ploz.tistory.com/entry/10-%EC%98%A4%EB%A5%98-%EC%88%98%EC%A0%95-hbase-Session-expire%EB%A1%9C-shutdown-%EB%90%98%EB%8A%94-%ED%98%84%EC%83%81

I've referred to this post and several others for guidance, and tried modifying my hbase-site.xml accordingly. However, it still functions well for a while but then keeps shutting down.

Below is my hbase-site.xml configuration.

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->
<configuration>
        <property>
                <name>zookeeper.session.timeout</name>
                <value>120000</value>
        </property>
        <property>
                <name>hbase.client.scanner.timeout.period</name>
                <value>604800000</value> <!-- 밀리초 단위, 여기서는 7일로 설정 -->
        </property>
        <property>
                <name>hbase.security.authentication</name>
                <value>simple</value>
        </property>
        <property>
                <name>hbase.security.authorization</name>
                <value>true</value>
        </property>
        <property>
                <name>hbase.coprocessor.master.classes</name>
                <value>org.apache.hadoop.hbase.security.access.AccessController</value>
        </property>
        <property>
                <name >hbase.coprocessor.region.classes</name>
                <value>org.apache.hadoop.hbase.security.access.AccessController</value>
        </property>
        <property>
                <name>hbase.coprocessor.regionserver.classes</name>
                <value>org.apache.hadoop.hbase.security.access.AccessController</value>
        </property>
</configuration>
@mingeonho1 mingeonho1 changed the title Session expired Session expired (hbase Session expire로 shutdown 되는 현상) Jan 19, 2024
@mingeonho1 mingeonho1 changed the title Session expired (hbase Session expire로 shutdown 되는 현상) Session expired (hbase Session expire로 shutdown 되는 현상) 해결이 안 돼서 도움 부탁드립니다 .. ㅠ Jan 19, 2024
@smilu97
Copy link
Member

smilu97 commented Jan 22, 2024 •
edited

Hi hoya, I'm not very sure about this but I'd like to recommend you to check if there is enough disk space.

There might be intensive disk write when hbase executes compaction jobs

Also It will be great if you translate your comments into english for us! we're trying to maintain issues to be written in english as possible.

@mingeonho1 mingeonho1 changed the title Session expired (hbase Session expire로 shutdown 되는 현상) 해결이 안 돼서 도움 부탁드립니다 .. ㅠ Session expired (The phenomenon of being shut down due to hbase Session expire) I can't solve it, so I ask for your help.. Jan 22, 2024
@mingeonho1
Copy link
Author

Hi hoya, I'm not very sure about this but I'd like to recommend you to check if there is enough disk space.

There might be intensive disk write when hbase executes compaction jobs

Also It will be great if you translate your comments into english for us! we're trying to maintain issues to be written in english as possible.

Thank you.

@mingeonho1 mingeonho1 changed the title Session expired (The phenomenon of being shut down due to hbase Session expire) I can't solve it, so I ask for your help.. Session expired (The phenomenon of being shut down due to hbase Session expire) I can't solve it, so I ask for your help..) Jan 22, 2024
@mingeonho1 mingeonho1 changed the title Session expired (The phenomenon of being shut down due to hbase Session expire) I can't solve it, so I ask for your help..) Session expired (The phenomenon of being shut down due to hbase Session expire) I can't solve it, so I ask for your help.. Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@smilu97 @mingeonho1