[HBASE-4580] Some invalid zk nodes were created when a clean cluster restarts

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.92.0
• Fix Version/s: 0.92.0
• Component/s: master
• Hadoop Flags: Reviewed
• Assignee: gaojinchao
• Reporter: gaojinchao
• Due: 12/Oct/11
• Created: 12/Oct/11 00:52
• Updated: 20/Nov/15 11:52
• Resolved: 20/Oct/11 19:41

Description

The below logs said that we created a invalid zk node when restarted a cluster.
it mistakenly believed that the regions belong to a dead server.

2011-10-11 05:05:29,127 INFO org.apache.hadoop.hbase.master.HMaster: Meta updated status = true
2011-10-11 05:05:29,127 INFO org.apache.hadoop.hbase.master.HMaster: ROOT/Meta already up-to date with new HRI.
2011-10-11 05:05:29,151 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for 771d63e9327383159553619a4f2dc74f with OFFLINE state
2011-10-11 05:05:29,161 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for 3cf860dd323fe6360f571aeafc129f95 with OFFLINE state
2011-10-11 05:05:29,170 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for 4065350214452a9d5c55243c734bef08 with OFFLINE state
2011-10-11 05:05:29,178 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for 4e81613f82a39fc6e5e89f96e7b3ccc4 with OFFLINE state
2011-10-11 05:05:29,187 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for e21b9e1545a28953aba0098fda5c9cd9 with OFFLINE state
2011-10-11 05:05:29,195 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for 5cd9f55eecd43d088bbd505f6795131f with OFFLINE state
2011-10-11 05:05:29,229 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for db5f641452a70b09b85a92970e4198c7 with OFFLINE state
2011-10-11 05:05:29,237 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for a7b20a653919e7f41bfb2ed349af7d21 with OFFLINE state
2011-10-11 05:05:29,253 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Creating (or updating) unassigned node for c9385619425f737eab1a6624d2e097a8 with OFFLINE state

// we cleaned all zk nodes.
2011-10-11 05:05:29,262 INFO org.apache.hadoop.hbase.master.AssignmentManager: Clean cluster startup. Assigning userregions
2011-10-11 05:05:29,262 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Deleting any existing unassigned nodes
2011-10-11 05:05:29,367 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 9 region(s) across 1 server(s), retainAssignment=true
2011-10-11 05:05:29,369 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Timeout-on-RIT=9000
2011-10-11 05:05:29,369 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 9 region(s) to C3S3,54366,1318323920153
2011-10-11 05:05:29,369 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for 771d63e9327383159553619a4f2dc74f with OFFLINE state
2011-10-11 05:05:29,371 INFO org.apache.hadoop.hbase.master.HMaster: Master has completed initialization
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for 3cf860dd323fe6360f571aeafc129f95 with OFFLINE state
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for 4065350214452a9d5c55243c734bef08 with OFFLINE state
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for 4e81613f82a39fc6e5e89f96e7b3ccc4 with OFFLINE state
2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for e21b9e1545a28953aba0098fda5c9cd9 with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for 5cd9f55eecd43d088bbd505f6795131f with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for db5f641452a70b09b85a92970e4198c7 with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for a7b20a653919e7f41bfb2ed349af7d21 with OFFLINE state
2011-10-11 05:05:29,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:58198-0x132f23a9a380000 Async create of unassigned node for c9385619425f737eab1a6624d2e097a8 with OFFLINE state