Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-2587

"Convert Cuboid Data to HFile" failed on EMR 5.5

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Workaround
    • Affects Version/s: v2.0.0
    • Fix Version/s: None
    • Component/s: Job Engine
    • Labels:
      None
    • Environment:
      EMR 5.5, EMR 5.4

      Description

      Create a EMR 5.5 HBase cluser, download and start Kylin on the master node, and then create the sample cube. The cube build got many errors in the reducer of "Convert Cuboid Data to HFile" step. Error trace is:

      2017-05-05 08:02:13,057 WARN [main] org.apache.hadoop.hbase.zookeeper.ZKUtil: hconnection-0x4a67318f0x0, quorum=localhost:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase/hbaseid)
      org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
      	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
      	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
      	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
      	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:220)
      	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:420)
      	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
      	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
      	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:913)
      	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:697)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
      	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
      	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
      	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:213)
      	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167)
      	at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:566)
      	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
      	at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
      	at org.apache.hadoop.hbase.mapreduce.KeyValueSortReducer.reduce(KeyValueSortReducer.java:53)
      	at org.apache.hadoop.hbase.mapreduce.KeyValueSortReducer.reduce(KeyValueSortReducer.java:36)
      	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
      	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:635)
      	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      

      I checked the environment, the hbase-site.xml is in /etc/hbase/conf folder, which appeared at the start position of "hbase classpath", looks good, but the reducer couldn't get zookeeper's info, so using the default value "localhost:2181".

      I bypassed this error by copy "hbase.zookeeper.quorum" from hbase-site.xml to $KYLIN_HOME/conf/kylin_job_conf.xml:

      <property>
      <name>hbase.zookeeper.quorum</name>
      <value>ip-nn-nn-nn-nn.ap-northeast-2.compute.internal</value>
      </property>

      It need a further investigation. Maybe is related with some change in hbase 1.3 (EMR 5.5)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              shaofengshi Shao Feng Shi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: