Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9738

Delete table and loadbalancer interference

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Labels:
      None

      Description

      I have noticed that when the balancer is computing a plan for region moves, and a delete table is issued, there is some interference.

      1. At time t1, user deleted the table.

      2. This led to the master updating the meta table to remove the line for the regioninfo for a region f2a9e2e9d70894c03f54ee5902bebee6.

      2013-10-04 08:42:52,495 INFO  [MASTER_TABLE_OPERATIONS-hor15n05:60000-0] catalog.MetaEditor: Deleted [{ENCODED => f2a9e2e9d70894c03f54ee5902bebee6, NAME => 'usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6.', STARTKEY => '', ENDKEY => ''}]
      

      3. However around the same time, the balancer kicked in, and reassigned the region and made it online somewhere. It didn't check the fact (nor anyone else did) that the table was indeed deleted.

      2013-10-04 08:42:53,215 INFO  [hor15n05.gq1.ygridcore.net,60000,1380869262259-BalancerChore] master.HMaster: balance hri=usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6., src=hor15n09.gq1.ygridcore.net,60020,1380869263722, dest=hor15n11.gq1.ygridcore.net,60020,1380869263682
      

      .....

      2013-10-04 08:42:53,592 INFO  [AM.ZK.Worker-pool2-t829] master.RegionStates: Onlined f2a9e2e9d70894c03f54ee5902bebee6 on hor15n11.gq1.ygridcore.net,60020,1380869263682
      

      4. Henceforth, all the drop tables started giving warnings like

      2013-10-04 08:45:17,587 INFO  [RpcServer.handler=8,port=60000] master.HMaster: Client=hrt_qa//68.142.246.151 delete usertable
      2013-10-04 08:45:17,631 DEBUG [RpcServer.handler=8,port=60000] lock.ZKInterProcessLockBase: Acquired a lock for /hbase/table-lock/usertable/write-master:600000000000000
      2013-10-04 08:45:17,637 WARN  [RpcServer.handler=8,port=60000] catalog.MetaReader: No serialized HRegionInfo in keyvalues={usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6./info:seqnumDuringOpen/1380876173509/Put/vlen=8/mvcc=0, usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6./info:server/1380876173509/Put/vlen=32/mvcc=0, usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6./info:serverstartcode/1380876173509/Put/vlen=8/mvcc=0}
      

      5. The create of the same table also fails since there is still state (reincarnated, maybe) about the table in the master.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ddas Devaraj Das
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: