Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7530

Go thru the common code to find references to HIVE_EXECUCTION_ENGINE to make sure conditions works with Spark [Spark Branch]

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • spark-branch
    • 1.1.0
    • Spark
    • None

    Description

      In common code, such as Utilities.java, I found a lot of references to this conf variable and special handling to a specific engine such as following:

                if (!HiveConf.getVar(job, ConfVars.HIVE_EXECUTION_ENGINE).equals("tez")
                    && isEmptyPath(job, path, ctx)) {
                  path = createDummyFileForEmptyPartition(path, job, work,
                       hiveScratchDir, alias, sequenceNumber++);
      
                }
      

      We need to make sure the condition still holds after a new execution engine such as "spark" is introduced.

      Attachments

        1. HIVE-7530-spark.patch
          2 kB
          Na Yang

        Issue Links

          Activity

            brocknoland Brock Noland added a comment -

            Thank you for the contribution!! I have committed this to spark!

            brocknoland Brock Noland added a comment - Thank you for the contribution!! I have committed this to spark!
            brocknoland Brock Noland added a comment -

            +1

            brocknoland Brock Noland added a comment - +1
            hiveqa Hive QA added a comment -

            Overall: -1 at least one tests failed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12662579/HIVE-7530-spark.patch

            ERROR: -1 due to 4 failed/errored test(s), 5917 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
            org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
            org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
            

            Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/59/testReport
            Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/59/console
            Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-59/

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 4 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12662579

            hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662579/HIVE-7530-spark.patch ERROR: -1 due to 4 failed/errored test(s), 5917 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/59/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/59/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-59/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12662579
            brocknoland Brock Noland added a comment -

            We've been hitting this one by one which is quite slow thus I think we should knock these out. I also think we should evaluate where we are casting to Tez specific classes (as evidenced by HIVE-7591).

            ./ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:      TezWork work = (TezWork)task.getWork();
            ./ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:        ((TezWork)work).add(cplan);
            ./ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:      TezWork work = (TezWork) currTask.getWork();
            ./ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java:          work = (MapWork) ((TezWork) mrTask.getWork()).getAllWork().get(0);
            
            brocknoland Brock Noland added a comment - We've been hitting this one by one which is quite slow thus I think we should knock these out. I also think we should evaluate where we are casting to Tez specific classes (as evidenced by HIVE-7591 ). ./ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java: TezWork work = (TezWork)task.getWork(); ./ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java: ((TezWork)work).add(cplan); ./ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java: TezWork work = (TezWork) currTask.getWork(); ./ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java: work = (MapWork) ((TezWork) mrTask.getWork()).getAllWork().get(0);

            People

              nyang Na Yang
              xuefuz Xuefu Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: