Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1120

MapReduce job read local meta issue

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: v1.0, v1.1
    • Fix Version/s: v1.2
    • Component/s: Storage - HBase
    • Labels:
      None
    • Environment:
      Kylin-1.x-staging branch
    • Flags:
      Patch

      Description

      set the property "kylin.metadata.url" in kylin.properties like this:

      kylin.metadata.url=kylin_metadata_test1.2@hbase
      

      1.Create a test cube and build.
      2.In the step 2:"Fact_Distinct_Columns" got exception:

      2015-11-02 17:23:50,354 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.NullPointerException
      	at org.apache.kylin.job.hadoop.cube.FactDistinctColumnsMapper.setup(FactDistinctColumnsMapper.java:73)
      	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      

      reason:
      In the MapReduce job Kylin set the local dir to metadataUrl for read metadata.

      FactDistinctColumnsMapper.java
      KylinConfig config = AbstractHadoopJob.loadKylinPropsAndMetadata(conf);
      

      But the ResourceStore.java return a "HBaseResourceStore" instance from "getStore" method.
      Because the HBaseResourceStore's constructor will not throw exception when the metadataUrl is local address not "xxx@hbase"
      "HBaseResourceStore" will read the default HBase table "kylin_metadata" to get the cube metadata and My metadata is in "kylin_metadata_test1.2" table,so got the NPE.

      I modify the class add order in ResourceStore's static code to fixed this issue.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                liyang.gmt8@gmail.com liyang
                Reporter:
                wangxiaoyu Xiaoyu Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: