Hive
  1. Hive
  2. HIVE-2823

Add logging of total run time of Driver

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We are currently logging the run time of several components of the Driver, e.g. compilation and map reduce execution time. It would be helpful to log the total run time of the Driver's run method as well.

        Activity

        Hide
        Phabricator added a comment -

        kevinwilfong requested code review of "HIVE-2823 [jira] Add logging of total run time of Driver".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-2823

        Updated the PerfLogger to now log the total runtime of Driver.run now. I also added methods to the PerfLogger to get the start and finish times, so that these can be logged if desired.

        We are currently logging the run time of several components of the Driver, e.g. compilation and map reduce execution time. It would be helpful to log the total run time of the Driver's run method as well.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1977

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/4203/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - kevinwilfong requested code review of " HIVE-2823 [jira] Add logging of total run time of Driver". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2823 Updated the PerfLogger to now log the total runtime of Driver.run now. I also added methods to the PerfLogger to get the start and finish times, so that these can be logged if desired. We are currently logging the run time of several components of the Driver, e.g. compilation and map reduce execution time. It would be helpful to log the total run time of the Driver's run method as well. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1977 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4203/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        kevinwilfong updated the revision "HIVE-2823 [jira] Add logging of total run time of Driver".
        Reviewers: JIRA, njain

        After discussing this with Namit Jain, we decided that since I ultimately intend to access the Driver.run start time from a post execution hook, it would make the most sense in terms of dependencies, if the start time was added to the QueryPlan, from which it can be accessed by the hook.

        REVISION DETAIL
        https://reviews.facebook.net/D1977

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
        ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java

        Show
        Phabricator added a comment - kevinwilfong updated the revision " HIVE-2823 [jira] Add logging of total run time of Driver". Reviewers: JIRA, njain After discussing this with Namit Jain, we decided that since I ultimately intend to access the Driver.run start time from a post execution hook, it would make the most sense in terms of dependencies, if the start time was added to the QueryPlan, from which it can be accessed by the hook. REVISION DETAIL https://reviews.facebook.net/D1977 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        Hide
        Phabricator added a comment -

        njain has accepted the revision "HIVE-2823 [jira] Add logging of total run time of Driver".

        running tests

        REVISION DETAIL
        https://reviews.facebook.net/D1977

        BRANCH
        svn

        Show
        Phabricator added a comment - njain has accepted the revision " HIVE-2823 [jira] Add logging of total run time of Driver". running tests REVISION DETAIL https://reviews.facebook.net/D1977 BRANCH svn
        Hide
        Namit Jain added a comment -

        Committed. Thanks Kevin

        Show
        Namit Jain added a comment - Committed. Thanks Kevin
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Phabricator added a comment -

        kevinwilfong has closed the revision "HIVE-2823 [jira] Add logging of total run time of Driver".

        REVISION DETAIL
        https://reviews.facebook.net/D1977

        To: JIRA, njain, kevinwilfong

        Show
        Phabricator added a comment - kevinwilfong has closed the revision " HIVE-2823 [jira] Add logging of total run time of Driver". REVISION DETAIL https://reviews.facebook.net/D1977 To: JIRA, njain, kevinwilfong
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2823 Add logging of total run time in Driver
        (Kevin Wilfong via namit) (Revision 1294492)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294492
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2823 Add logging of total run time in Driver (Kevin Wilfong via namit) (Revision 1294492) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294492 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development