Hive
  1. Hive
  2. HIVE-6130

Webhcat resource /templeton/v1/status doesn't track Hive job progress

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Problem: Unable to track Hive job progress through webhcat.
      Cause:
      TempletonUtils has code to parse PIG and JAR child jobid and percentage progress but none for Hive jobs. extractPercentComplete() and extractChildJobId() are supposed to do this parsing.
      Effect:
      /templeton/v1/queue/$job_id?user.name=$user returns no job progress info. The jobid param in the json result that's supposed to contain the child hive jobid contains the templetoncontrollerjob id instead leaving the parent jobid null.

      1. HIVE-6130.1.patch
        4 kB
        Shuaishuai Nie

        Issue Links

          Activity

          Shuaishuai Nie created issue -
          Shuaishuai Nie made changes -
          Field Original Value New Value
          Link This issue depends upon HCATALOG-647 [ HCATALOG-647 ]
          Hide
          Shuaishuai Nie added a comment -

          Create this JIRA to keep tracking of HCATALOG-647. Gautam Kowshik can you please re-base the patch in HCATALOG-647 to hive trunk? This is an important fix and we want to get this fix committed. If it is inconvenience for you, I can help you with it. Thanks

          Show
          Shuaishuai Nie added a comment - Create this JIRA to keep tracking of HCATALOG-647 . Gautam Kowshik can you please re-base the patch in HCATALOG-647 to hive trunk? This is an important fix and we want to get this fix committed. If it is inconvenience for you, I can help you with it. Thanks
          Shuaishuai Nie made changes -
          Link This issue is a clone of HCATALOG-647 [ HCATALOG-647 ]
          Shuaishuai Nie made changes -
          Link This issue depends upon HCATALOG-647 [ HCATALOG-647 ]
          Hide
          Eugene Koifman added a comment -

          TempletonUtils already has logic for Hive percentComplete.

          Also, inspite of the title for this bug, I assume you are not planning to change what ./templeton/v1/status method does, correct?

          Show
          Eugene Koifman added a comment - TempletonUtils already has logic for Hive percentComplete. Also, inspite of the title for this bug, I assume you are not planning to change what ./templeton/v1/status method does, correct?
          Hide
          Shuaishuai Nie added a comment -

          Hi Eugene Koifman. Yes, but we still need the part in the original patch witch parse the hive job id correctly. Attached the re-based patch.

          Show
          Shuaishuai Nie added a comment - Hi Eugene Koifman . Yes, but we still need the part in the original patch witch parse the hive job id correctly. Attached the re-based patch.
          Shuaishuai Nie made changes -
          Attachment HIVE-6130.1.patch [ 12621645 ]
          Hide
          Eugene Koifman added a comment -

          Shuaishuai Nie
          1. the patch contains changes LaunchMapper.java - how is this relevant?
          2. if you grep .conf files in templeton e2e tests, for percentComplete and percent_complete, you'll see that we have tests for pig,jar and hive. Do you believe any of them fail? If so, should the tests be adjusted?
          3. I'm not aware of any ID generated when we submit a hive query

          Show
          Eugene Koifman added a comment - Shuaishuai Nie 1. the patch contains changes LaunchMapper.java - how is this relevant? 2. if you grep .conf files in templeton e2e tests, for percentComplete and percent_complete, you'll see that we have tests for pig,jar and hive. Do you believe any of them fail? If so, should the tests be adjusted? 3. I'm not aware of any ID generated when we submit a hive query
          Hide
          Eugene Koifman added a comment -

          Ignore #3 above. I guess the intent is to parse out the hadoop job ID.

          4. I don't think the unit tests here are very useful. The tests for these changes should be e2e tests as you are basically parsing log files, which may change w/o warning from release to release and break the functionality. the unit test which has a hardcoded string in it will not catch this.

          Show
          Eugene Koifman added a comment - Ignore #3 above. I guess the intent is to parse out the hadoop job ID. 4. I don't think the unit tests here are very useful. The tests for these changes should be e2e tests as you are basically parsing log files, which may change w/o warning from release to release and break the functionality. the unit test which has a hardcoded string in it will not catch this.
          Hide
          Shuaishuai Nie added a comment -

          Hi Eugene Koifman,
          1. the patch contains changes LaunchMapper.java - how is this relevant?
          The changes from launchMapper.java is just from the original patch I ported. Since wirter.flush() not actually doing anything for hdfs,
          2. if you grep .conf files in templeton e2e tests, for percentComplete and percent_complete, you'll see that we have tests for pig,jar and hive. Do you believe any of them fail? If so, should the tests be adjusted?
          If the e2e tests already covers hive cases and the similar fix is checked in from HIVE-5511, I think this patch is no longer necessary.
          Thanks,
          Shuaishuai

          Show
          Shuaishuai Nie added a comment - Hi Eugene Koifman , 1. the patch contains changes LaunchMapper.java - how is this relevant? The changes from launchMapper.java is just from the original patch I ported. Since wirter.flush() not actually doing anything for hdfs, 2. if you grep .conf files in templeton e2e tests, for percentComplete and percent_complete, you'll see that we have tests for pig,jar and hive. Do you believe any of them fail? If so, should the tests be adjusted? If the e2e tests already covers hive cases and the similar fix is checked in from HIVE-5511 , I think this patch is no longer necessary. Thanks, Shuaishuai
          Hide
          Eugene Koifman added a comment -

          If you believe that ID for a Hive job is necessary, I think you should make patch with that, but I think it needs a proper test.

          Show
          Eugene Koifman added a comment - If you believe that ID for a Hive job is necessary, I think you should make patch with that, but I think it needs a proper test.
          Hide
          Eugene Koifman added a comment -

          The tests that I mentioned only deal with percentComplete attribute - I see that you made some changes to regex for that as well.

          Show
          Eugene Koifman added a comment - The tests that I mentioned only deal with percentComplete attribute - I see that you made some changes to regex for that as well.
          Hide
          shanyu zhao added a comment -

          Shuaishuai Nie I think you should remove this line, we shouldn't call flush() for every line in stdout, this will cause performance issue. For example, flush() will cause a packet being sent out for WASB file system.
          + writer.flush();

          Also, is this change intended?

          • public static final Pattern HIVE_COMPLETE = Pattern.compile(" map = \\d+%,
            s+reduce =
            d+%$");
            + public static final Pattern HIVE_COMPLETE = Pattern.compile(" map = \\d+%,
            s+reduce =
            d+%");
          Show
          shanyu zhao added a comment - Shuaishuai Nie I think you should remove this line, we shouldn't call flush() for every line in stdout, this will cause performance issue. For example, flush() will cause a packet being sent out for WASB file system. + writer.flush(); Also, is this change intended? public static final Pattern HIVE_COMPLETE = Pattern.compile(" map = \\d+%, s+reduce = d+%$"); + public static final Pattern HIVE_COMPLETE = Pattern.compile(" map = \\d+%, s+reduce = d+%");

            People

            • Assignee:
              Unassigned
              Reporter:
              Shuaishuai Nie
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development