Hive
  1. Hive
  2. HIVE-2364

Make performance logging configurable.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.

      1. HIVE-2364.1.patch.txt
        16 kB
        Kevin Wilfong
      2. HIVE-2364.2.patch.txt
        17 kB
        Kevin Wilfong
      3. HIVE-2364.3.patch.txt
        17 kB
        Kevin Wilfong

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #894 (See https://builds.apache.org/job/Hive-trunk-h0.21/894/)
        HIVE-2364. Make performance logging configurable. (Kevin Wilfong via Ning Zhang)

        nzhang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157532
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
        • /hive/trunk/conf/hive-default.xml
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #894 (See https://builds.apache.org/job/Hive-trunk-h0.21/894/ ) HIVE-2364 . Make performance logging configurable. (Kevin Wilfong via Ning Zhang) nzhang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157532 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java /hive/trunk/conf/hive-default.xml
        Hide
        Ning Zhang added a comment -

        Committed. Thanks Kevin!

        Show
        Ning Zhang added a comment - Committed. Thanks Kevin!
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1443/
        -----------------------------------------------------------

        (Updated 2011-08-14 00:54:16.595577)

        Review request for hive and Ning Zhang.

        Changes
        -------

        Updated patch.

        Summary
        -------

        I created a new class PerfLogger, which wraps the old functionality of Utilities' PerfLogBegin and PerfLogBegin methods. I also added a config variable hive.exec.perf.logger. The value of this variable can be changed to point to a subclass of PerfLogger which can customize the code in PerfLogBegin and PerfLogEnd.

        This addresses bug HIVE-2364.
        https://issues.apache.org/jira/browse/HIVE-2364

        Diffs (updated)


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1157444
        trunk/conf/hive-default.xml 1157444
        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1157444
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1157444
        trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1157444

        Diff: https://reviews.apache.org/r/1443/diff

        Testing
        -------

        I ran the code as it is and made sure it continued to log the performance messages as before.

        I also created a subclass of PerfLogger and used its class name as the value of hive.exec.perf.logger, and verified the subclass's code was run.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1443/ ----------------------------------------------------------- (Updated 2011-08-14 00:54:16.595577) Review request for hive and Ning Zhang. Changes ------- Updated patch. Summary ------- I created a new class PerfLogger, which wraps the old functionality of Utilities' PerfLogBegin and PerfLogBegin methods. I also added a config variable hive.exec.perf.logger. The value of this variable can be changed to point to a subclass of PerfLogger which can customize the code in PerfLogBegin and PerfLogEnd. This addresses bug HIVE-2364 . https://issues.apache.org/jira/browse/HIVE-2364 Diffs (updated) trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1157444 trunk/conf/hive-default.xml 1157444 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1157444 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1157444 trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1157444 Diff: https://reviews.apache.org/r/1443/diff Testing ------- I ran the code as it is and made sure it continued to log the performance messages as before. I also created a subclass of PerfLogger and used its class name as the value of hive.exec.perf.logger, and verified the subclass's code was run. Thanks, Kevin
        Hide
        Ning Zhang added a comment -

        Kevin, this patch is out of date. Could you update it and repost it? Please ping me after you update it and I'll start testing right away.

        Show
        Ning Zhang added a comment - Kevin, this patch is out of date. Could you update it and repost it? Please ping me after you update it and I'll start testing right away.
        Hide
        Ning Zhang added a comment -

        +1. Will commit if tests pass.

        Show
        Ning Zhang added a comment - +1. Will commit if tests pass.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1443/
        -----------------------------------------------------------

        (Updated 2011-08-10 00:18:50.035513)

        Review request for hive and Ning Zhang.

        Changes
        -------

        Forgot to add the licensing info to the top of PerfLogger.

        Summary
        -------

        I created a new class PerfLogger, which wraps the old functionality of Utilities' PerfLogBegin and PerfLogBegin methods. I also added a config variable hive.exec.perf.logger. The value of this variable can be changed to point to a subclass of PerfLogger which can customize the code in PerfLogBegin and PerfLogEnd.

        This addresses bug HIVE-2364.
        https://issues.apache.org/jira/browse/HIVE-2364

        Diffs (updated)


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155439
        trunk/conf/hive-default.xml 1155439
        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155439
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1155439
        trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1155439

        Diff: https://reviews.apache.org/r/1443/diff

        Testing
        -------

        I ran the code as it is and made sure it continued to log the performance messages as before.

        I also created a subclass of PerfLogger and used its class name as the value of hive.exec.perf.logger, and verified the subclass's code was run.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1443/ ----------------------------------------------------------- (Updated 2011-08-10 00:18:50.035513) Review request for hive and Ning Zhang. Changes ------- Forgot to add the licensing info to the top of PerfLogger. Summary ------- I created a new class PerfLogger, which wraps the old functionality of Utilities' PerfLogBegin and PerfLogBegin methods. I also added a config variable hive.exec.perf.logger. The value of this variable can be changed to point to a subclass of PerfLogger which can customize the code in PerfLogBegin and PerfLogEnd. This addresses bug HIVE-2364 . https://issues.apache.org/jira/browse/HIVE-2364 Diffs (updated) trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155439 trunk/conf/hive-default.xml 1155439 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155439 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1155439 trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1155439 Diff: https://reviews.apache.org/r/1443/diff Testing ------- I ran the code as it is and made sure it continued to log the performance messages as before. I also created a subclass of PerfLogger and used its class name as the value of hive.exec.perf.logger, and verified the subclass's code was run. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1443/
        -----------------------------------------------------------

        Review request for hive and Ning Zhang.

        Summary
        -------

        I created a new class PerfLogger, which wraps the old functionality of Utilities' PerfLogBegin and PerfLogBegin methods. I also added a config variable hive.exec.perf.logger. The value of this variable can be changed to point to a subclass of PerfLogger which can customize the code in PerfLogBegin and PerfLogEnd.

        This addresses bug HIVE-2364.
        https://issues.apache.org/jira/browse/HIVE-2364

        Diffs


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155439
        trunk/conf/hive-default.xml 1155439
        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155439
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1155439
        trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1155439

        Diff: https://reviews.apache.org/r/1443/diff

        Testing
        -------

        I ran the code as it is and made sure it continued to log the performance messages as before.

        I also created a subclass of PerfLogger and used its class name as the value of hive.exec.perf.logger, and verified the subclass's code was run.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1443/ ----------------------------------------------------------- Review request for hive and Ning Zhang. Summary ------- I created a new class PerfLogger, which wraps the old functionality of Utilities' PerfLogBegin and PerfLogBegin methods. I also added a config variable hive.exec.perf.logger. The value of this variable can be changed to point to a subclass of PerfLogger which can customize the code in PerfLogBegin and PerfLogEnd. This addresses bug HIVE-2364 . https://issues.apache.org/jira/browse/HIVE-2364 Diffs trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155439 trunk/conf/hive-default.xml 1155439 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155439 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1155439 trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1155439 Diff: https://reviews.apache.org/r/1443/diff Testing ------- I ran the code as it is and made sure it continued to log the performance messages as before. I also created a subclass of PerfLogger and used its class name as the value of hive.exec.perf.logger, and verified the subclass's code was run. Thanks, Kevin
        Show
        Kevin Wilfong added a comment - https://reviews.apache.org/r/1443/

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development