Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4361

Fix detailed metrics for protobuf-based RPC on 0.23

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      RPC detailed metrics for any protobuf-based RPC ports are always zero. ProtoOverHadoopRpcEngine needs the same detailed metric logic as in WritableRpcEngine. This is effectively the same change as in HADOOP-8085 except tailored for branch-0.23 which didn't take the full protobuf branch changes that went into branch-2 and trunk.

        Activity

        Hide
        Jason Lowe added a comment -

        Patch that ports the relevant portions of HADOOP-8085 to branch-0.23. I manually tested this by bringing up a single-node cluster and visiting the ResourceManager JMX page. The detailed metrics for YARN interfaces like NodeHeartbeatNumOps were non-zero, whereas they were zero before applying this patch.

        I also manually ran the unit tests in yarn-common, and they all passed.

        Show
        Jason Lowe added a comment - Patch that ports the relevant portions of HADOOP-8085 to branch-0.23. I manually tested this by bringing up a single-node cluster and visiting the ResourceManager JMX page. The detailed metrics for YARN interfaces like NodeHeartbeatNumOps were non-zero, whereas they were zero before applying this patch. I also manually ran the unit tests in yarn-common, and they all passed.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12533103/MAPREDUCE-4361.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2500//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533103/MAPREDUCE-4361.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2500//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        test-patch doesn't know how to test patches that aren't based on trunk. Manually ran test-patch on branch-0.23 and got these results:

        -1 overall.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 18 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        The javadoc warnings are unrelated to the patch (mostly involving issues in Rumen and Hadoop Extras).

        Show
        Jason Lowe added a comment - test-patch doesn't know how to test patches that aren't based on trunk. Manually ran test-patch on branch-0.23 and got these results: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 18 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. The javadoc warnings are unrelated to the patch (mostly involving issues in Rumen and Hadoop Extras).
        Hide
        Thomas Graves added a comment -

        +1 lgtm. Thanks Jason!

        Show
        Thomas Graves added a comment - +1 lgtm. Thanks Jason!
        Hide
        Thomas Graves added a comment -

        I committed this to branch 0.23, the change is not applicable to trunk or branch-2 since it already works there.

        Show
        Thomas Graves added a comment - I committed this to branch 0.23, the change is not applicable to trunk or branch-2 since it already works there.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #298 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/298/)
        MAPREDUCE-4361. Fix detailed metrics for protobuf-based RPC on 0.23 (Jason Lowe via tgraves) (Revision 1353580)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353580
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #298 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/298/ ) MAPREDUCE-4361 . Fix detailed metrics for protobuf-based RPC on 0.23 (Jason Lowe via tgraves) (Revision 1353580) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353580 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java

          People

          • Assignee:
            Jason Lowe
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development