Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24556

HBase upgrade from 2.2 to 2.3 fails due to failure in ZooKeeper

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: Zookeeper
    • Labels:
      None

      Description

      When we tested upgrading HBase from 2.2 to 2.3, the upgraded node failed to start due to a known zookeeper failure - ZOOKEEPER-3056.

      The error message is shown below:

       

      2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master exiting2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master exitingjava.io.IOException: No snapshot found, but there are log entries. Something is broken! at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240) at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290) at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450) at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764) at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98) at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:259) at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:189) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:210) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3035)
      

       

      It can be reproduced through the following steps:

      • Start a single-node HBase 2.2 with the following configuration.
          <property>
            <name>hbase.rootdir</name>
            <value>/var/lib/hbase</value>
          </property>  <property>
            <name>hbase.zookeeper.property.dataDir</name>
            <value>/var/lib/zookeeper</value>
          </property>
      • Use HBase performance evaluation tool to generate data.
        /hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1
      • Upgrade the node to 2.3 with the same configuration. The new version node failed to start because of the zookeeper.

      HBase 2.2 is using zookeeper-3.4.10.jar, and Kafka 2.3 are using zookeeper-3.5.7.jar.

      The bug has been fixed by ZOOKEEPER-3056 in zookeeper-3.6.0. Should we upgrade the dependency of HBase 2.3 to use zookeeper-3.6.0.jar? or upgrade the configuration file of zookeeper-3.5.7.jar?

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Zhuqi1108 Zhuqi Jin
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: