Hive
  1. Hive
  2. HIVE-2236

Cli: Print Hadoop's CPU milliseconds

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: CLI
    • Labels:
      None

      Description

      CPU Milliseonds information is available from Hadoop's framework. Printing it out to Hive CLI when executing a job will help users to know more about their jobs.

      1. HIVE-2236.1.patch
        17 kB
        Siying Dong
      2. HIVE-2236.2.patch
        18 kB
        Siying Dong
      3. HIVE-2236.3.patch
        18 kB
        Siying Dong
      4. HIVE-2236.4.patch
        17 kB
        Siying Dong

        Activity

        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Carl Steinbach made changes -
        Fix Version/s 0.8.0 [ 12316178 ]
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #848 (See https://builds.apache.org/job/Hive-trunk-h0.21/848/)
        HIVE-2236: Print Hadoop's CPU milliseconds in Cli. (Siying Dong via He Yongqiang)

        heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150945
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #848 (See https://builds.apache.org/job/Hive-trunk-h0.21/848/ ) HIVE-2236 : Print Hadoop's CPU milliseconds in Cli. (Siying Dong via He Yongqiang) heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150945 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
        He Yongqiang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        He Yongqiang added a comment -

        committed, thanks Siying!

        Show
        He Yongqiang added a comment - committed, thanks Siying!
        Siying Dong made changes -
        Attachment HIVE-2236.4.patch [ 12487738 ]
        Hide
        He Yongqiang added a comment -

        siying, can you rebase this patch? thanks!

        Show
        He Yongqiang added a comment - siying, can you rebase this patch? thanks!
        Hide
        He Yongqiang added a comment -

        +1, will commit after tests pass

        Show
        He Yongqiang added a comment - +1, will commit after tests pass
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/948/
        -----------------------------------------------------------

        (Updated 2011-07-21 17:30:55.228025)

        Review request for hive, Yongqiang He, Ning Zhang, and namit jain.

        Changes
        -------

        fix a bug

        Summary
        -------

        In hive CLI, print out CPU msec from Hadoop MapReduce coutners.

        This addresses bug HIVE-2236.
        https://issues.apache.org/jira/browse/HIVE-2236

        Diffs (updated)


        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1148623
        trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1148623
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1148623
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1148623
        trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1148623

        Diff: https://reviews.apache.org/r/948/diff

        Testing
        -------

        run the updated codes against real clusters and make sure it printing is correct.

        Thanks,

        Siying

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/948/ ----------------------------------------------------------- (Updated 2011-07-21 17:30:55.228025) Review request for hive, Yongqiang He, Ning Zhang, and namit jain. Changes ------- fix a bug Summary ------- In hive CLI, print out CPU msec from Hadoop MapReduce coutners. This addresses bug HIVE-2236 . https://issues.apache.org/jira/browse/HIVE-2236 Diffs (updated) trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1148623 Diff: https://reviews.apache.org/r/948/diff Testing ------- run the updated codes against real clusters and make sure it printing is correct. Thanks, Siying
        Siying Dong made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siying Dong made changes -
        Attachment HIVE-2236.3.patch [ 12487326 ]
        Hide
        Siying Dong added a comment -

        fix a bug

        Show
        Siying Dong added a comment - fix a bug
        Siying Dong made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/948/
        -----------------------------------------------------------

        (Updated 2011-07-20 06:27:19.820431)

        Review request for hive, Yongqiang He, Ning Zhang, and namit jain.

        Changes
        -------

        remove MapRedStats from DriverContext and add more counters to it.

        Summary
        -------

        In hive CLI, print out CPU msec from Hadoop MapReduce coutners.

        This addresses bug HIVE-2236.
        https://issues.apache.org/jira/browse/HIVE-2236

        Diffs (updated)


        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1148623
        trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1148623
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1148623
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1148623
        trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1148623

        Diff: https://reviews.apache.org/r/948/diff

        Testing
        -------

        run the updated codes against real clusters and make sure it printing is correct.

        Thanks,

        Siying

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/948/ ----------------------------------------------------------- (Updated 2011-07-20 06:27:19.820431) Review request for hive, Yongqiang He, Ning Zhang, and namit jain. Changes ------- remove MapRedStats from DriverContext and add more counters to it. Summary ------- In hive CLI, print out CPU msec from Hadoop MapReduce coutners. This addresses bug HIVE-2236 . https://issues.apache.org/jira/browse/HIVE-2236 Diffs (updated) trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1148623 Diff: https://reviews.apache.org/r/948/diff Testing ------- run the updated codes against real clusters and make sure it printing is correct. Thanks, Siying
        Siying Dong made changes -
        Attachment HIVE-2236.2.patch [ 12487110 ]
        Hide
        Siying Dong added a comment -

        remove the MapRedStat list from DriverContext and add more counters.

        Show
        Siying Dong added a comment - remove the MapRedStat list from DriverContext and add more counters.
        Siying Dong made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/948/#review909
        -----------------------------------------------------------

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        <https://reviews.apache.org/r/948/#comment1962>

        Because I use this in finally block.

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        <https://reviews.apache.org/r/948/#comment1963>

        What's the concern?

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
        <https://reviews.apache.org/r/948/#comment1964>

        We might not. If it doesn't support it will return null. That's why I have a null logic next to ignore null results and don't record anything.

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
        <https://reviews.apache.org/r/948/#comment1965>

        Yes.

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
        <https://reviews.apache.org/r/948/#comment1966>

        Why?

        • Siying

        On 2011-06-23 07:41:33, Siying Dong wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/948/

        -----------------------------------------------------------

        (Updated 2011-06-23 07:41:33)

        Review request for hive, Yongqiang He, Ning Zhang, and namit jain.

        Summary

        -------

        In hive CLI, print out CPU msec from Hadoop MapReduce coutners.

        This addresses bug HIVE-2236.

        https://issues.apache.org/jira/browse/HIVE-2236

        Diffs

        -----

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1138748

        trunk/ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 1138748

        trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1138748

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1138748

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1138748

        trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1138748

        Diff: https://reviews.apache.org/r/948/diff

        Testing

        -------

        run the updated codes against real clusters and make sure it printing is correct.

        Thanks,

        Siying

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/948/#review909 ----------------------------------------------------------- trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java < https://reviews.apache.org/r/948/#comment1962 > Because I use this in finally block. trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java < https://reviews.apache.org/r/948/#comment1963 > What's the concern? trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java < https://reviews.apache.org/r/948/#comment1964 > We might not. If it doesn't support it will return null. That's why I have a null logic next to ignore null results and don't record anything. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java < https://reviews.apache.org/r/948/#comment1965 > Yes. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java < https://reviews.apache.org/r/948/#comment1966 > Why? Siying On 2011-06-23 07:41:33, Siying Dong wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/948/ ----------------------------------------------------------- (Updated 2011-06-23 07:41:33) Review request for hive, Yongqiang He, Ning Zhang, and namit jain. Summary ------- In hive CLI, print out CPU msec from Hadoop MapReduce coutners. This addresses bug HIVE-2236 . https://issues.apache.org/jira/browse/HIVE-2236 Diffs ----- trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1138748 Diff: https://reviews.apache.org/r/948/diff Testing ------- run the updated codes against real clusters and make sure it printing is correct. Thanks, Siying
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/948/#review904
        -----------------------------------------------------------

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        <https://reviews.apache.org/r/948/#comment1947>

        why do you need to pull this var out of the code block?

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        <https://reviews.apache.org/r/948/#comment1948>

        can you not store the MapRedStatsList in DriverContext?

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        <https://reviews.apache.org/r/948/#comment1949>

        format the code here

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
        <https://reviews.apache.org/r/948/#comment1959>

        how do you know the hadoop support this counter

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
        <https://reviews.apache.org/r/948/#comment1960>

        are u going to print this info if this job failed?

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
        <https://reviews.apache.org/r/948/#comment1958>

        remove 'ms'

        • Yongqiang

        On 2011-06-23 07:41:33, Siying Dong wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/948/

        -----------------------------------------------------------

        (Updated 2011-06-23 07:41:33)

        Review request for hive, Yongqiang He, Ning Zhang, and namit jain.

        Summary

        -------

        In hive CLI, print out CPU msec from Hadoop MapReduce coutners.

        This addresses bug HIVE-2236.

        https://issues.apache.org/jira/browse/HIVE-2236

        Diffs

        -----

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1138748

        trunk/ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 1138748

        trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1138748

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1138748

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1138748

        trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1138748

        Diff: https://reviews.apache.org/r/948/diff

        Testing

        -------

        run the updated codes against real clusters and make sure it printing is correct.

        Thanks,

        Siying

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/948/#review904 ----------------------------------------------------------- trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java < https://reviews.apache.org/r/948/#comment1947 > why do you need to pull this var out of the code block? trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java < https://reviews.apache.org/r/948/#comment1948 > can you not store the MapRedStatsList in DriverContext? trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java < https://reviews.apache.org/r/948/#comment1949 > format the code here trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java < https://reviews.apache.org/r/948/#comment1959 > how do you know the hadoop support this counter trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java < https://reviews.apache.org/r/948/#comment1960 > are u going to print this info if this job failed? trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java < https://reviews.apache.org/r/948/#comment1958 > remove 'ms' Yongqiang On 2011-06-23 07:41:33, Siying Dong wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/948/ ----------------------------------------------------------- (Updated 2011-06-23 07:41:33) Review request for hive, Yongqiang He, Ning Zhang, and namit jain. Summary ------- In hive CLI, print out CPU msec from Hadoop MapReduce coutners. This addresses bug HIVE-2236 . https://issues.apache.org/jira/browse/HIVE-2236 Diffs ----- trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1138748 Diff: https://reviews.apache.org/r/948/diff Testing ------- run the updated codes against real clusters and make sure it printing is correct. Thanks, Siying
        He Yongqiang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Siying Dong made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/948/
        -----------------------------------------------------------

        Review request for hive, Yongqiang He, Ning Zhang, and namit jain.

        Summary
        -------

        In hive CLI, print out CPU msec from Hadoop MapReduce coutners.

        This addresses bug HIVE-2236.
        https://issues.apache.org/jira/browse/HIVE-2236

        Diffs


        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1138748
        trunk/ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 1138748
        trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1138748
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1138748
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1138748
        trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1138748

        Diff: https://reviews.apache.org/r/948/diff

        Testing
        -------

        run the updated codes against real clusters and make sure it printing is correct.

        Thanks,

        Siying

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/948/ ----------------------------------------------------------- Review request for hive, Yongqiang He, Ning Zhang, and namit jain. Summary ------- In hive CLI, print out CPU msec from Hadoop MapReduce coutners. This addresses bug HIVE-2236 . https://issues.apache.org/jira/browse/HIVE-2236 Diffs trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1138748 trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1138748 Diff: https://reviews.apache.org/r/948/diff Testing ------- run the updated codes against real clusters and make sure it printing is correct. Thanks, Siying
        Siying Dong made changes -
        Attachment HIVE-2236.1.patch [ 12483553 ]
        Siying Dong made changes -
        Field Original Value New Value
        Assignee Siying Dong [ sdong ]
        Siying Dong created issue -

          People

          • Assignee:
            Siying Dong
            Reporter:
            Siying Dong
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development