[HBASE-6736] Distributed Split: a split tasks can be mark as DONE but keep unassigned

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.95.2
• Component/s: master
• Assignee: Unassigned
• Reporter: Nicolas Liochon
• Created: 07/Sep/12 14:36
• Updated: 14/Jun/22 17:52
• Resolved: 15/Oct/13 09:04

Description

Real cluster, scenario mentioned on HBASE-5843.

Got it once out of 5 tests on 0.96
Didn't get it on 0.94 after 3 tests.

It seems we have a race condition on split logs: the task was nearly simultaneously marked as done and resubmitted. Then it remained in the unassigned state.
2012-09-04 17:27:06,237 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 0
2012-09-04 17:27:06,237 INFO org.apache.hadoop.hbase.master.SplitLogManager: resubmitted 1 out of 1 tasks
2012-09-04 17:27:06,237 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2FBOX1%3A9000%2Fhbase%2F.logs%2FBOX0%2C60020%2C1346772046399-splitting%2FBOX0%252C60020%252C1346772046399.1346772046609 ver = 7
2012-09-04 17:27:06,314 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/RESCAN0000000002 entered state: DONE BOX1,60000,1346771990737
2012-09-04 17:27:06,337 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/RESCAN0000000002
2012-09-04 17:27:06,337 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: deleted task without in memory state /hbase/splitlog/RESCAN0000000002
2012-09-04 17:27:07,226 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 1

Issue Links

• is required by: HBASE-5843 Improve HBase MTTR - Mean Time To Recover