Hive
  1. Hive
  2. HIVE-2127

Improve stats gathering reliability by retries on failures with hive.stats.retries.max and hive.stats.retries.wait

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Query Processor, Statistics
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Stats publishing and aggregation only try once and if there is any exception it will fail and return. If many mappers/reducers updating stats at the same time, it is very common to get lock timeout. We should make stats more reliable by retry when there is an SQLException.

      1. HIVE-2127.patch
        19 kB
        Ning Zhang
      2. HIVE-2127.2.patch
        30 kB
        Ning Zhang

        Activity

        Ning Zhang created issue -
        Ning Zhang made changes -
        Field Original Value New Value
        Attachment HIVE-2127.patch [ 12477320 ]
        Hide
        Ning Zhang added a comment -
        Show
        Ning Zhang added a comment - Review board: https://reviews.apache.org/r/664/
        Ning Zhang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Ning Zhang added a comment -

        Paul has an offline comment about it cannot handle Connection exceptions. I'm working on a new patch and will update it soon.

        Show
        Ning Zhang added a comment - Paul has an offline comment about it cannot handle Connection exceptions. I'm working on a new patch and will update it soon.
        Ning Zhang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Ning Zhang made changes -
        Attachment HIVE-2127.2.patch [ 12477343 ]
        Hide
        Ning Zhang added a comment -

        Updated the review board.

        Show
        Ning Zhang added a comment - Updated the review board.
        Ning Zhang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Namit Jain added a comment -

        Comments in review-board

        Show
        Namit Jain added a comment - Comments in review-board
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Namit Jain added a comment -

        Also add the new configuration variables in the name of the jira

        Show
        Namit Jain added a comment - Also add the new configuration variables in the name of the jira
        Hide
        Ning Zhang added a comment -

        @Namit, what does the new configuration variable do? Do you mean to define a variable to disable retry? If so set hive.stats.retries.max = 0 will do.

        Show
        Ning Zhang added a comment - @Namit, what does the new configuration variable do? Do you mean to define a variable to disable retry? If so set hive.stats.retries.max = 0 will do.
        Hide
        Namit Jain added a comment -

        What I meant was:

        Change the subject of the jira:
        Improve stats gathering reliability by retries on failures

        for better searching

        Show
        Namit Jain added a comment - What I meant was: Change the subject of the jira: Improve stats gathering reliability by retries on failures for better searching
        Hide
        Namit Jain added a comment -

        Looks good otherwise

        Show
        Namit Jain added a comment - Looks good otherwise
        Ning Zhang made changes -
        Summary Improve stats gathering reliability by retries on failures Improve stats gathering reliability by retries on failures with hive.stats.retries.max and hive.stats.retries.wait
        Hide
        Ning Zhang added a comment -

        Changed the JIRA subject

        Show
        Ning Zhang added a comment - Changed the JIRA subject
        Ning Zhang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Namit Jain added a comment -

        Committed. Thanks Ning

        Show
        Namit Jain added a comment - Committed. Thanks Ning
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Carl Steinbach made changes -
        Fix Version/s 0.8.0 [ 12316178 ]
        Component/s Query Processor [ 12312586 ]
        Component/s Statistics [ 12314312 ]
        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        1d 9h 35m 2 Namit Jain 27/Apr/11 06:14
        Open Open Patch Available Patch Available
        2d 17h 16m 3 Ning Zhang 27/Apr/11 07:11
        Patch Available Patch Available Resolved Resolved
        8h 8m 1 Namit Jain 27/Apr/11 15:19
        Resolved Resolved Closed Closed
        233d 9h 36m 1 Carl Steinbach 16/Dec/11 23:56

          People

          • Assignee:
            Ning Zhang
            Reporter:
            Ning Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development