Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7745

NullPointerException when turn on hive.optimize.union.remove, hive.merge.mapfiles and hive.merge.mapredfiles [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: spark-branch
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:
      None

      Description

      When the hive.optimize.union.remove, hive.merge.mapfiles and hive.merge.mapredfiles are turned on, it throws NullPointerException when I do the following queries:

      create table inputTbl1(key string, val string) stored as textfile;
      create table outputTbl1(key string, values bigint) stored as rcfile;
      load data local inpath '../../data/files/T1.txt' into table inputTbl1;
      
      explain
      insert overwrite table outputTbl1
      SELECT * FROM
      (
      select key, count(1) as values from inputTbl1 group by key 
      union all
      select * FROM (
        SELECT key, 1 as values from inputTbl1 
        UNION ALL
        SELECT key, 2 as values from inputTbl1
      ) a
      )b;
      

      If the hive.merge.mapfiles and hive.merge.mapredfiles are turned off, I do not see any error.

      Here is the stack trace:

      2014-08-16 01:32:26,849 ERROR [main]: ql.Driver (SessionState.java:printError(681)) - FAILED: NullPointerException null
      java.lang.NullPointerException
              at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMoveTask(GenMapRedUtils.java:1738)
              at org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.processFileSink(GenSparkUtils.java:281)
              at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.generateTaskTree(SparkCompiler.java:187)
              at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:199)
              at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9508)
              at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:208)
              at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
              at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:208)
              at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:414)
              at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:310)
              at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1005)
              at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1070)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:942)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:932)
              at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
              at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:198)
              at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
              at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
              at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
              at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
      

        Attachments

        1. HIVE-7745-spark.patch
          1 kB
          Na Yang

          Issue Links

            Activity

              People

              • Assignee:
                nyang Na Yang
                Reporter:
                nyang Na Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: