Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22626

Master assigns the region successfully, but updates the state of region failed, and then keeping the state of the region is OPENNING in zookeeper, If master restarted, those OPENNING regions will not be assign forever.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0
    • Fix Version/s: None
    • Component/s: Region Assignment
    • Labels:
      None

      Description

      Problem Description
      (1)One of the region server restarts causes all the regions that have been assigned to be migrated.

      (2)The master checks these regions and assigns them to other region servers, and assigns other region server assignments successfully, but the update state fails.

      2019-06-22 16:44:20,065 INFO [PEWorker-8] procedure2.ProcedureExecutor: Finished pid=5038, ppid=4488, state=SUCCESS; AssignProcedure table=test061910, region=379c730490b4848f3db065fb25b87452 in 10.5680sec
      ... ...
      2019-06-22 16:44:38,725 INFO [PEWorker-4] procedure.MasterProcedureScheduler: pid=5038, ppid=4488, state=SUCCESS; AssignProcedure table=test061910, region=379c730490b4848f3db065fb25b87452 checking lock on 379c730490b4848f3db065fb25b87452
      2019-06-22 16:44:38,725 ERROR [PEWorker-4] procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception for pid=5038, ppid=4488, state=SUCCESS; AssignProcedure table=test061910, region=379c730490b4848f3db065fb25b87452
      java.lang.UnsupportedOperationException: Unhandled state REGION_TRANSITION_FINISH; there is no rollback for assignment unless we cancel the operation by dropping/disabling the table
      at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.rollback(RegionTransitionProcedure.java:412)
      at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.rollback(RegionTransitionProcedure.java:95)
      at org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:864)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1373)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1329)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1198)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
      at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1761)

      (3)The master restarted, and the old region state will be loaded after restarting. The region for the OPENNING state is not reassigned forever, and the following log is printed.

      2019-06-22 16:46:26,210 INFO [master/hdp0:16000] assignment.RegionStateStore: Load hbase:meta entry region=379c730490b4848f3db065fb25b87452, regionState=OPENING, lastHost=hdp2,16020,1561190163476, regionLocation=hdp0,16020,1561190163887, openSeqNum=69448
      ... ...
      2019-06-22 16:51:28,514 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=hdp0,16020,1561190163887, table=test061910, region=379c730490b4848f3db065fb25b87452
      ... ....
      2019-06-22 16:49:28,483 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=hdp0,16020,1561190163887, table=test061910, region=379c730490b4848f3db065fb25b87452

      (4) The state in zookeeper
      test061910,00000000000000000031457280,1560943902323.379c730490b4848f3db065fb25b87452. column=info:regioninfo, timestamp=1561192609882, value={ENCODED => 379c730490b4848f3db065fb25b87452, NAME => 'test061910,00000000000000000031457280,1560943902323.379c730490b4848f3db065fb25b87452.', STARTKEY => '00000000000000000031457280', ENDKEY => '00000000000000000041943040'}
      test061910,00000000000000000031457280,1560943902323.379c730490b4848f3db065fb25b87452. column=info:seqnumDuringOpen, timestamp=1561190288613, value=\x00\x00\x00\x00\x00\x01\x0FH
      test061910,00000000000000000031457280,1560943902323.379c730490b4848f3db065fb25b87452. column=info:server, timestamp=1561190288613, value=hdp2:16020
      test061910,00000000000000000031457280,1560943902323.379c730490b4848f3db065fb25b87452. column=info:serverstartcode, timestamp=1561190288613, value=1561190163476
      test061910,00000000000000000031457280,1560943902323.379c730490b4848f3db065fb25b87452. column=info:sn, timestamp=1561192609882, value=hdp0,16020,1561190163887
      test061910,00000000000000000031457280,1560943902323.379c730490b4848f3db065fb25b87452. column=info:state, timestamp=1561192609882, value=OPENING

       

      Problem Recurrence
      (1) Modify the state of a normal region from OPEN to OPENNING.
      (2) Restart the master and region servers, view the master log and hbase web.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                chenwandong chenwandong
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: