[HBASE-6748] Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.94.6, 0.95.0
• Fix Version/s: 0.94.6, 0.95.0
• Component/s: master
• Hadoop Flags: Reviewed
• Assignee: Jeffrey Zhong
• Reporter: Jieshan Bean
• Created: 10/Sep/12 12:00
• Updated: 23/Mar/13 04:53
• Resolved: 23/Feb/13 06:02

Description

You can ealily understand the problem from the below logs:

[2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3
[2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2
[2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1
[2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0
[2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
[2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
[2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807
[2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806
[2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805
[2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804
[2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775803
...................
[2012-09-01 11:41:03,307] [ERROR] [MASTER_SERVER_OPERATIONS-xh03,20000,1339549619270-1] [org.apache.zookeeper.ClientCnxn 623] Caught unexpected throwable
java.lang.StackOverflowError

Issue Links

• relates to: HBASE-7914 Port the fix of HBASE-6748 into 0.94 branch