[HBASE-5606] SplitLogManger async delete node hangs log splitting when ZK connection is lost

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.92.0
• Fix Version/s: 0.92.2, 0.94.0, 0.95.0
• Component/s: wal
• Hadoop Flags: Reviewed
• Assignee: Prakash Khemani
• Reporter: Gopinathan A
• Created: 21/Mar/12 04:27
• Updated: 18/Sep/13 22:18
• Resolved: 03/Apr/12 22:19

Description

1. One rs died, the servershutdownhandler found it out and started the distributed log splitting;
2. All tasks are failed due to ZK connection lost, so the all the tasks were deleted asynchronously;
3. Servershutdownhandler retried the log splitting;
4. The asynchronously deletion in step 2 finally happened for new task
5. This made the SplitLogManger in hanging state.

This leads to .META. region not assigened for long time

hbase-root-master-HOST-192-168-47-204.log.2012-03-14"(55413,79):2012-03-14 19:28:47,932 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task at znode /hbase/splitlog/hdfs%3A%2F%2F192.168.47.205%3A9000%2Fhbase%2F.logs%2Flinux-114.site%2C60020%2C1331720381665-splitting%2Flinux-114.site%252C60020%252C1331720381665.1331752316170
hbase-root-master-HOST-192-168-47-204.log.2012-03-14"(89303,79):2012-03-14 19:34:32,387 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task at znode /hbase/splitlog/hdfs%3A%2F%2F192.168.47.205%3A9000%2Fhbase%2F.logs%2Flinux-114.site%2C60020%2C1331720381665-splitting%2Flinux-114.site%252C60020%252C1331720381665.1331752316170

hbase-root-master-HOST-192-168-47-204.log.2012-03-14"(80417,99):2012-03-14 19:34:31,196 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/hdfs%3A%2F%2F192.168.47.205%3A9000%2Fhbase%2F.logs%2Flinux-114.site%2C60020%2C1331720381665-splitting%2Flinux-114.site%252C60020%252C1331720381665.1331752316170
hbase-root-master-HOST-192-168-47-204.log.2012-03-14"(89456,99):2012-03-14 19:34:32,497 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/hdfs%3A%2F%2F192.168.47.205%3A9000%2Fhbase%2F.logs%2Flinux-114.site%2C60020%2C1331720381665-splitting%2Flinux-114.site%252C60020%252C1331720381665.1331752316170