Hive
  1. Hive
  2. HIVE-6521

WebHCat cannot fetch correct percentComplete for Hive jobs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.14.0
    • Component/s: WebHCat
    • Labels:
      None

      Description

      WebHCat E2E test TestHive_7 failed because percentComplete wasn't returned as expected.

      check_job_percent_complete failed. got percentComplete "map 0% reduce 0%",  expected  "map 100% reduce 100%"
      

      So, there are two problems here.

      1. The log parsing is broken for status of percentComplete. In the stderr of the job we see:
        Launching Job 1 out of 1
        Number of reduce tasks is set to 0 since there's no reduce operator
        Starting Job = job_1393486488858_0691, Tracking URL = http://ambari-sec-1393480847-others-2-4.cs1cloud.internal:8088/proxy/application_1393486488858_0691/
        Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1393486488858_0691
        Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
        2014-02-27 18:40:50,166 Stage-1 map = 0%,  reduce = 0%
        2014-02-27 18:40:56,599 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.87 sec
        2014-02-27 18:40:57,656 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.87 sec
        2014-02-27 18:40:58,706 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.87 sec
        MapReduce Total cumulative CPU time: 870 msec
        Ended Job = job_1393486488858_0691
        MapReduce Jobs Launched: 
        Job 0: Map: 1   Cumulative CPU: 0.87 sec   HDFS Read: 305 HDFS Write: 0 SUCCESS
        Total MapReduce CPU Time Spent: 870 msec
        

        The assumption in the code is that the line containing the percent status will end after "reduce = \d+%" but that fails with the above.

      2. The last status from Hive job is "map = 100%, reduce = 0%" instead of expected "map = 100%, reduce = 100%".
      1. HIVE-6521.patch
        2 kB
        Deepesh Khandelwal
      2. HIVE-6521.2.patch
        2 kB
        Deepesh Khandelwal

        Activity

        Hide
        Thejas M Nair added a comment -

        This has been fixed in 0.14 release. Please open new jira if you see any issues.

        Show
        Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
        Hide
        Deepesh Khandelwal added a comment -

        Thanks Eugene for the review and Sushanth for the committ.

        Show
        Deepesh Khandelwal added a comment - Thanks Eugene for the review and Sushanth for the committ.
        Hide
        Sushanth Sowmyan added a comment -

        Committed to trunk. Thanks, Deepesh for the patch, and Thanks Eugene, for the review.

        Show
        Sushanth Sowmyan added a comment - Committed to trunk. Thanks, Deepesh for the patch, and Thanks Eugene, for the review.
        Hide
        Sushanth Sowmyan added a comment -

        Okay, nvm, my apologies, this patch is okay - my issue was from another source. +1 to this patch.

        Show
        Sushanth Sowmyan added a comment - Okay, nvm, my apologies, this patch is okay - my issue was from another source. +1 to this patch.
        Hide
        Sushanth Sowmyan added a comment -

        I think there's an issue here with a unit test failure with -Phadoop-2, looking into this.

        Show
        Sushanth Sowmyan added a comment - I think there's an issue here with a unit test failure with -Phadoop-2, looking into this.
        Hide
        Eugene Koifman added a comment -

        +1

        Show
        Eugene Koifman added a comment - +1
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12631797/HIVE-6521.2.patch

        ERROR: -1 due to 2 failed/errored test(s), 5185 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1560/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1560/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12631797

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12631797/HIVE-6521.2.patch ERROR: -1 due to 2 failed/errored test(s), 5185 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1560/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1560/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12631797
        Hide
        Deepesh Khandelwal added a comment -

        The previous patch was getting the "Cumulative CPU ..." in the percentComplete string. New patch does better job in parsing.

        Show
        Deepesh Khandelwal added a comment - The previous patch was getting the "Cumulative CPU ..." in the percentComplete string. New patch does better job in parsing.
        Hide
        Deepesh Khandelwal added a comment -

        Attached patch fixes two things:

        1. The regular expression to comply with the Hive percent status logging.
        2. Modified the test so that we also exercise the reduce step.
          Please review.
        Show
        Deepesh Khandelwal added a comment - Attached patch fixes two things: The regular expression to comply with the Hive percent status logging. Modified the test so that we also exercise the reduce step. Please review.

          People

          • Assignee:
            Deepesh Khandelwal
            Reporter:
            Deepesh Khandelwal
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development