Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21519

Namespace region is never assigned in a HM failover scenario when multiwal is enabled

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.1.1
    • Fix Version/s: None
    • Component/s: master, wal
    • Labels:
      None

      Description

      In our test env we found that namespace region is never be assigned on HM failover scenario when multiwal feature is enabled,

      2018-11-28 01:38:28,085 WARN [master/HM-1:16000:becomeActiveMaster] master.HMaster: hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. is NOT online; state=\{31f6d3383af09e18e1e81ca02a93de15 state=OPEN, ts=1543340156928, server=RS-2,16020,1543339824397}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
      

      And finally HM abort with following error,

      2018-11-28 01:39:16,858 ERROR [ActiveMasterInitializationMonitor-1543338648565] master.HMaster: Master failed to complete initialization after 240000ms. Please consider submitting a bug report including a thread dump of this process.
      2018-11-28 01:39:18,980 ERROR [ActiveMasterInitializationMonitor-1543338648565] master.HMaster: Zombie Master exiting. Thread dump to stdout
      

      Stack trace:

      Thread 102 (master/HM-1:16000:becomeActiveMaster):
       State: TIMED_WAITING
       Blocked count: 100
       Waited count: 246
       Stack:
       java.lang.Thread.sleep(Native Method)
       org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:148)
       org.apache.hadoop.hbase.master.HMaster.isRegionOnline(HMaster.java:1166)
       org.apache.hadoop.hbase.master.HMaster.waitForNamespaceOnline(HMaster.java:1187)
       org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1044)
       org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2285)
       org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:590)
       org.apache.hadoop.hbase.master.HMaster$$Lambda$40/1078246575.run(Unknown Source)
       java.lang.Thread.run(Thread.java:745)
      

       

      Step to reproduce:
      1) Setup a HBase cluster with 1/2 HM (say HM-1) and 2 RS(say RS-1 & RS-2)
      2) Enable multiwal feature with following configuration setting and start the cluster,

       <property>
       <name>hbase.wal.provider</name>
       <value>multiwal</value>
       </property>
      
      <property>
       <name>hbase.wal.regiongrouping.strategy</name>
       <value>identity</value>
       </property>
      

      3) Make sure meta and namespace regions are assigned on different RS, suppose RS-1 & RS-2 respectively.
      4) Create table 't1'
      5) Flush the meta table explicitly
      6) Kill the RS-2, so during RS-2 SCP all regions including namespace region will be assigned to RS-1.
      7) Now Kill RS-1 before meta flush happen. Here both RS-2 & RS-1 are shutdown now.
      8) Stop the HM and start RS-1 & RS-2.
      9) Now start the HM.

      Meta region is assigned successfully but HM is keep waiting for the namespace region onlline (Master startup cannot progress, in holding-pattern until region onlined) and abort with timeout.

      Observation:
      1) After step-3 namespace region was assigned to RS-2 and meta entry was as follows,

       hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. column=info:server, timestamp=1543339860920, value=RS-2:16020
       hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. column=info:serverstartcode, timestamp=1543339860920, value=1543339824397
      

      2) After step-6 namespace region was assigned to RS-1 and meta entry was as follows,

       hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. column=info:server, timestamp=1543339880920, value=RS-1:16020
       hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. column=info:serverstartcode, timestamp=1543339880920, value=1543339829288
      

      3) After Step-9, meta entry for namespace region was as follows,

       hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. column=info:server, timestamp=1543339860920, value=RS-2:16020
       hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. column=info:serverstartcode, timestamp=1543339860920, value=1543339824397
      

      During SCP we do meta log split based on filter,

       /**
       * Specialized method to handle the splitting for meta WAL
       * @param serverNames logs belonging to these servers will be split
       */
       public void splitMetaLog(final Set<ServerName> serverNames) throws IOException {
       splitLog(serverNames, META_FILTER);
       }
      

      So in this case meta log split will be skipped because of multiwal provider, as the suffix wont be .meta.

      I will analyze it further.

        Attachments

        1. HBASE-21519.branch-2.patch
          7 kB
          Pankaj Kumar
        2. providerId_value.png
          314 kB
          Pankaj Kumar

          Issue Links

            Activity

              People

              • Assignee:
                pankaj2461 Pankaj Kumar
                Reporter:
                pankaj2461 Pankaj Kumar
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: