Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9351

Running Hive Jobs with Tez cause templeton to never report percent complete

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 1.1.0
    • Component/s: WebHCat
    • Labels:
      None

      Description

      Currently, when submitting Hive jobs through WebHCat and Hive is configured to use Tez, the "percentComplete" field returned by WebHCat is empty.

      LaunchMapper in WebHCat parses stderr of the process that it launches to extract map = 100%, reduce = 100% for Map/Reduce case. With Tez the content of stderr looks like

        Map 1: -/-	Reducer 2: 0/1	
        Map 1: -/-	Reducer 2: 0(+1)/1	
        Map 1: -/-	Reducer 2: 1/1
      

      WebHCat should handle that as well.
      WebHCat will follow HIVE-8495 and report (completed tasks)/(total tasks) as a percentage.

      1. HIVE-9351.patch
        9 kB
        Eugene Koifman

        Activity

        Hide
        leftylev Lefty Leverenz added a comment -

        Okay, then no doc needed. Thanks Eugene Koifman.

        Show
        leftylev Lefty Leverenz added a comment - Okay, then no doc needed. Thanks Eugene Koifman .
        Hide
        ekoifman Eugene Koifman added a comment -

        none of the other steps in this README are on wiki (AFAIK), so I don't think so.
        If there is a section on wiki at all about running webhcat e2e tests, it should have a pointer to this readme but otherwise there is nothing to document here.

        Show
        ekoifman Eugene Koifman added a comment - none of the other steps in this README are on wiki (AFAIK), so I don't think so. If there is a section on wiki at all about running webhcat e2e tests, it should have a pointer to this readme but otherwise there is nothing to document here.
        Hide
        leftylev Lefty Leverenz added a comment -

        Should the new guidelines in hcatalog/src/test/e2e/templeton/README.txt be documented in the wiki?

        +Running on Tez
        +1. set up Tez as in http://tez.apache.org/install.html
        +2. set hive.execution.engine=tez in hive-site.xml (actually is this needed?)
        +3. add hive.execution.engine=tez to templeton.hive.properties in webhcat-site.xml
        +4. add to mapred-env.sh/yarn-env.sh (as you defined these in step 1)
        +export TEZ_VERSION=0.5.3
        +export TEZ_JARS=/Users/ekoifman/dev/apache-tez-client-${TEZ_VERSION}
        +export TEZ_CONF_DIR=${TEZ_JARS}/conf
        +export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH}
        +(w/o this you'll see something like "java.lang.NoClassDefFoundError: org/apache/tez/dag/api/SessionNotRunning")
        
        Show
        leftylev Lefty Leverenz added a comment - Should the new guidelines in hcatalog/src/test/e2e/templeton/README.txt be documented in the wiki? +Running on Tez +1. set up Tez as in http: //tez.apache.org/install.html +2. set hive.execution.engine=tez in hive-site.xml (actually is this needed?) +3. add hive.execution.engine=tez to templeton.hive.properties in webhcat-site.xml +4. add to mapred-env.sh/yarn-env.sh (as you defined these in step 1) +export TEZ_VERSION=0.5.3 +export TEZ_JARS=/Users/ekoifman/dev/apache-tez-client-${TEZ_VERSION} +export TEZ_CONF_DIR=${TEZ_JARS}/conf +export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH} +(w/o this you'll see something like "java.lang.NoClassDefFoundError: org/apache/tez/dag/api/SessionNotRunning" )
        Hide
        ekoifman Eugene Koifman added a comment -

        Thanks Thejas M Nair for the review

        Show
        ekoifman Eugene Koifman added a comment - Thanks Thejas M Nair for the review
        Hide
        ekoifman Eugene Koifman added a comment -

        This is a WebHCat only change and test failures above are not related. They also failed in prior bot run.

        Show
        ekoifman Eugene Koifman added a comment - This is a WebHCat only change and test failures above are not related. They also failed in prior bot run.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12691730/HIVE-9351.patch

        ERROR: -1 due to 4 failed/errored test(s), 7309 tests executed
        Failed tests:

        TestCustomAuthentication - did not produce a TEST-*.xml file
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
        org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
        org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2345/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2345/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2345/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 4 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12691730 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12691730/HIVE-9351.patch ERROR: -1 due to 4 failed/errored test(s), 7309 tests executed Failed tests: TestCustomAuthentication - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2345/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2345/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2345/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12691730 - PreCommit-HIVE-TRUNK-Build
        Hide
        thejas Thejas M Nair added a comment -

        +1

        Show
        thejas Thejas M Nair added a comment - +1

          People

          • Assignee:
            ekoifman Eugene Koifman
            Reporter:
            ekoifman Eugene Koifman
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development