Hive
  1. Hive
  2. HIVE-5519

Use paging mechanism for templeton get requests.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: WebHCat
    • Labels:
      None

      Description

      Issuing a command to retrieve the jobs field using

      "https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/<job_id>?user.name=admin&fields=*" --user u
      will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out.

      1. HIVE-5519.3.patch.txt
        8 kB
        Hari Sankar Sivarama Subramaniyan
      2. HIVE-5519.2.patch.txt
        7 kB
        Hari Sankar Sivarama Subramaniyan
      3. HIVE-5519.1.patch.txt
        4 kB
        Hari Sankar Sivarama Subramaniyan

        Activity

        Hide
        Eugene Koifman added a comment -

        I think it would be useful to add some JavaDoc on the params added in Server.java (at least the new ones)

        Show
        Eugene Koifman added a comment - I think it would be useful to add some JavaDoc on the params added in Server.java (at least the new ones)
        Hide
        Eugene Koifman added a comment -

        I think it would be useful to add some JavaDoc on the params added in Server.java (at least the new ones)

        Show
        Eugene Koifman added a comment - I think it would be useful to add some JavaDoc on the params added in Server.java (at least the new ones)
        Hide
        Hari Sankar Sivarama Subramaniyan added a comment -

        Eugene Koifman Agree with what you say. Also TestDriverCurl.pm needs some modification to replace the deprecated APIs with the new ones. I will make these changes as part of HIVE-5617

        Show
        Hari Sankar Sivarama Subramaniyan added a comment - Eugene Koifman Agree with what you say. Also TestDriverCurl.pm needs some modification to replace the deprecated APIs with the new ones. I will make these changes as part of HIVE-5617
        Show
        Hari Sankar Sivarama Subramaniyan added a comment - https://reviews.apache.org/r/14916/
        Hide
        Hari Sankar Sivarama Subramaniyan added a comment -

        Addressed the comments made by Eugene Koifman

        Show
        Hari Sankar Sivarama Subramaniyan added a comment - Addressed the comments made by Eugene Koifman
        Hide
        Eugene Koifman added a comment -

        Great job on the doc. It could be useful to add 1 line how these param can be used to support pagination since it may not be obvious

        +1

        Show
        Eugene Koifman added a comment - Great job on the doc. It could be useful to add 1 line how these param can be used to support pagination since it may not be obvious +1
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12610402/HIVE-5519.2.patch.txt

        SUCCESS: +1 4483 tests passed

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1251/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1251/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12610402/HIVE-5519.2.patch.txt SUCCESS: +1 4483 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1251/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1251/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
        Hide
        Hari Sankar Sivarama Subramaniyan added a comment -

        Thanks for the review, Thejas M Nair Updating the change as per Thejas M Nair ' comment. The bug with the previous upload was that even though the counter was incremented correctly based on the filter condition, the records which do not satisfy the filter condition do not get skipped. The fix is to continue and skip the current record, if the filter condition is not satisfied in Server.showJobList().

        Show
        Hari Sankar Sivarama Subramaniyan added a comment - Thanks for the review, Thejas M Nair Updating the change as per Thejas M Nair ' comment. The bug with the previous upload was that even though the counter was incremented correctly based on the filter condition, the records which do not satisfy the filter condition do not get skipped. The fix is to continue and skip the current record, if the filter condition is not satisfied in Server.showJobList().
        Hide
        Thejas M Nair added a comment -

        Looks like the patch needs change to handle filtering out of the jobs before starttime.

        Show
        Thejas M Nair added a comment - Looks like the patch needs change to handle filtering out of the jobs before starttime.
        Hide
        Thejas M Nair added a comment -

        I think it would be more useful to use the jobid instead of the start time in the API. One problem with the time is that it is start time of jobtracker server, which won't change until restart. Also, webhcat will be isolated from changes to the jobid string format if we use the entire string.

        Show
        Thejas M Nair added a comment - I think it would be more useful to use the jobid instead of the start time in the API. One problem with the time is that it is start time of jobtracker server, which won't change until restart. Also, webhcat will be isolated from changes to the jobid string format if we use the entire string.
        Hide
        Hari Sankar Sivarama Subramaniyan added a comment -

        Addressing Thejas M Nair ' comments.

        Show
        Hari Sankar Sivarama Subramaniyan added a comment - Addressing Thejas M Nair ' comments.
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12610734/HIVE-5519.3.patch.txt

        SUCCESS: +1 4512 tests passed

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1275/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1275/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12610734/HIVE-5519.3.patch.txt SUCCESS: +1 4512 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1275/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1275/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12610734/HIVE-5519.3.patch.txt

        SUCCESS: +1 4514 tests passed

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1283/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1283/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12610734/HIVE-5519.3.patch.txt SUCCESS: +1 4514 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1283/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1283/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
        Hide
        Thejas M Nair added a comment -

        Looks good. Nice work on the javadoc and comments! +1 .
        Can you also please update the wiki page once this is in (mark it as a feature that will be available in next release (0.13)?

        Show
        Thejas M Nair added a comment - Looks good. Nice work on the javadoc and comments! +1 . Can you also please update the wiki page once this is in (mark it as a feature that will be available in next release (0.13)?
        Hide
        Thejas M Nair added a comment -

        Patch committed to trunk.
        Thanks for the contribution Hari!

        Show
        Thejas M Nair added a comment - Patch committed to trunk. Thanks for the contribution Hari!
        Hide
        Lefty Leverenz added a comment -

        Added jobid & numrecords parameters to the Jobs wikidoc, but it still needs examples (with outputs).

        Show
        Lefty Leverenz added a comment - Added jobid & numrecords parameters to the Jobs wikidoc, but it still needs examples (with outputs). Web HCat Reference: Jobs

          People

          • Assignee:
            Hari Sankar Sivarama Subramaniyan
            Reporter:
            Hari Sankar Sivarama Subramaniyan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development