Hive
  1. Hive
  2. HIVE-2969

Log Time To Submit metric with PerfLogger

    Details

    • Type: Wish Wish
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Logging
    • Labels:
      None

      Description

      Logging the time from when Driver.run starts to when we begin submitting jobs to map reduce would be helpful in determining how much of the lag in starting a query is due to Hive vs. Hadoop.

        Activity

        Kevin Wilfong created issue -
        Hide
        Phabricator added a comment -

        kevinwilfong requested code review of "HIVE-2969 [jira] Log Time To Submit metric with PerfLogger".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-2969

        Added Time To Submit to the PerfLogger, beginning just after the PerfLogger is first initialized, and ending just before the loop which starts submitting jobs to map reduce.

        Logging the time from when Driver.run starts to when we begin submitting jobs to map reduce would be helpful in determining how much of the lag in starting a query is due to Hive vs. Hadoop.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D2919

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/6651/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - kevinwilfong requested code review of " HIVE-2969 [jira] Log Time To Submit metric with PerfLogger". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2969 Added Time To Submit to the PerfLogger, beginning just after the PerfLogger is first initialized, and ending just before the loop which starts submitting jobs to map reduce. Logging the time from when Driver.run starts to when we begin submitting jobs to map reduce would be helpful in determining how much of the lag in starting a query is due to Hive vs. Hadoop. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2919 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6651/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Phabricator made changes -
        Field Original Value New Value
        Attachment HIVE-2969.D2919.1.patch [ 12523602 ]
        Kevin Wilfong made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1474 (See https://builds.apache.org/job/Hive-trunk-h0.21/1474/)
        HIVE-2969. Log Time To Submit metric with PerfLogger. (kevinwilfong reviewed by ashutoshc) (Revision 1347870)

        Result = FAILURE
        kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347870
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1474 (See https://builds.apache.org/job/Hive-trunk-h0.21/1474/ ) HIVE-2969 . Log Time To Submit metric with PerfLogger. (kevinwilfong reviewed by ashutoshc) (Revision 1347870) Result = FAILURE kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347870 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
        Hide
        Kevin Wilfong added a comment -

        Committed

        Show
        Kevin Wilfong added a comment - Committed
        Kevin Wilfong made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Ashutosh Chauhan made changes -
        Fix Version/s 0.10.0 [ 12320745 ]
        Affects Version/s 0.10.0 [ 12320745 ]
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2969. Log Time To Submit metric with PerfLogger. (kevinwilfong reviewed by ashutoshc) (Revision 1347870)

        Result = ABORTED
        kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347870
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2969 . Log Time To Submit metric with PerfLogger. (kevinwilfong reviewed by ashutoshc) (Revision 1347870) Result = ABORTED kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1347870 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development