Hadoop Common
  1. Hadoop Common
  2. HADOOP-8227

Allow RPC to limit ephemeral port range.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.2
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: None
    • Labels:
      None

      Description

      This is a sub task of MAPREDUCE-4079. For security reasons we would like to limit the range of ports that are used when some RPC servers select a port.

      1. HADOOP-8227-trunk.txt
        21 kB
        Robert Joseph Evans
      2. HADOOP-8227-trunk.txt
        22 kB
        Robert Joseph Evans
      3. HADOOP-8227-trunk.txt
        25 kB
        Robert Joseph Evans

        Issue Links

          Activity

          Hide
          Robert Joseph Evans added a comment -

          Tom just checked the full patch into branch-0.23 as well so resolving this.

          Show
          Robert Joseph Evans added a comment - Tom just checked the full patch into branch-0.23 as well so resolving this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1053 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1053/)
          HADOOP-8227. Allow RPC to limit ephemeral port range. (bobby) (Revision 1327127)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327127
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1053 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1053/ ) HADOOP-8227 . Allow RPC to limit ephemeral port range. (bobby) (Revision 1327127) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327127 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1018 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1018/)
          HADOOP-8227. Allow RPC to limit ephemeral port range. (bobby) (Revision 1327127)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327127
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1018 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1018/ ) HADOOP-8227 . Allow RPC to limit ephemeral port range. (bobby) (Revision 1327127) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327127 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2100 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2100/)
          HADOOP-8227. Allow RPC to limit ephemeral port range. (bobby) (Revision 1327127)

          Result = ABORTED
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327127
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2100 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2100/ ) HADOOP-8227 . Allow RPC to limit ephemeral port range. (bobby) (Revision 1327127) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327127 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2086 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2086/)
          HADOOP-8227. Allow RPC to limit ephemeral port range. (bobby) (Revision 1327127)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327127
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2086 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2086/ ) HADOOP-8227 . Allow RPC to limit ephemeral port range. (bobby) (Revision 1327127) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327127 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2159/)
          HADOOP-8227. Allow RPC to limit ephemeral port range. (bobby) (Revision 1327127)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327127
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2159/ ) HADOOP-8227 . Allow RPC to limit ephemeral port range. (bobby) (Revision 1327127) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327127 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
          Hide
          Robert Joseph Evans added a comment -

          I have merged this into trunk and branch-2. I am leaving the patch open until I get a +1 on MAPREDUCE-4079 at which point I hope to also put it into branch-0.23, and then I will resolve both JIRAs.

          Show
          Robert Joseph Evans added a comment - I have merged this into trunk and branch-2. I am leaving the patch open until I get a +1 on MAPREDUCE-4079 at which point I hope to also put it into branch-0.23, and then I will resolve both JIRAs.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522785/HADOOP-8227-trunk.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/854//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/854//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522785/HADOOP-8227-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/854//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/854//console This message is automatically generated.
          Hide
          Thomas Graves added a comment -

          +1 lgtm

          Show
          Thomas Graves added a comment - +1 lgtm
          Hide
          Robert Joseph Evans added a comment -

          This patch addresses Tom's comments.

          combined branch-0.23 patch is at https://issues.apache.org/jira/secure/attachment/12522783/MR-4079-full-branch-0.23.txt

          Show
          Robert Joseph Evans added a comment - This patch addresses Tom's comments. combined branch-0.23 patch is at https://issues.apache.org/jira/secure/attachment/12522783/MR-4079-full-branch-0.23.txt
          Hide
          Robert Joseph Evans added a comment -

          Canceling patch to address Tom's Comments.

          Show
          Robert Joseph Evans added a comment - Canceling patch to address Tom's Comments.
          Hide
          Thomas Graves added a comment -

          Mostly looks good. Couple minor things:

          you misspelled queue in javadocs in RpcEngine.java for getServer. (+ * @param queueSizePerHandler the size of the queu per hander thread)
          Can you also indent the second line of the description of portRangeConfig

          It would also be nice to have a test for the RangeNumberIterator, to make sure it hits entire range and doesn't miss the edge cases.

          Show
          Thomas Graves added a comment - Mostly looks good. Couple minor things: you misspelled queue in javadocs in RpcEngine.java for getServer. (+ * @param queueSizePerHandler the size of the queu per hander thread) Can you also indent the second line of the description of portRangeConfig It would also be nice to have a test for the RangeNumberIterator, to make sure it hits entire range and doesn't miss the edge cases.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521962/HADOOP-8227-trunk.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/841//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/841//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521962/HADOOP-8227-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/841//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/841//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          Updated patch to address review comments. Corresponding comments can be seen in the 0.23 version at

          https://issues.apache.org/jira/secure/attachment/12521960/MR-4079-full-branch-0.23.txt

          Show
          Robert Joseph Evans added a comment - Updated patch to address review comments. Corresponding comments can be seen in the 0.23 version at https://issues.apache.org/jira/secure/attachment/12521960/MR-4079-full-branch-0.23.txt
          Hide
          Robert Joseph Evans added a comment -

          Canceling patch to address Suresh's comments so far.

          Show
          Robert Joseph Evans added a comment - Canceling patch to address Suresh's comments so far.
          Hide
          Suresh Srinivas added a comment -

          Other minor comment - follow coding conventions - space after if, before "{" etc.

          Show
          Suresh Srinivas added a comment - Other minor comment - follow coding conventions - space after if, before "{" etc.
          Hide
          Suresh Srinivas added a comment -

          Still going through the patch. Early comments:

          1. Please add the Apache License header to TestServer.java
          2. Add brief description to the test on what it is testing. Otherwise tests at a later point in time become hard to maintain.

          Will continue the patch review.

          Show
          Suresh Srinivas added a comment - Still going through the patch. Early comments: Please add the Apache License header to TestServer.java Add brief description to the test on what it is testing. Otherwise tests at a later point in time become hard to maintain. Will continue the patch review.
          Hide
          Robert Joseph Evans added a comment -

          Yes, that is the exact issue. As part of this change I modified the RpcEngine interface. In trunk and branch-2 all implementations of RpcEngine are in common, but in 0.23 ProtoOverHadoopRpcEngine is in MapReduce so the fix has to be split. I think it is simpler to get the code changes approved and into trunk/branch-2 first and then put in the branch-23 patch at once, instead of splitting them up.

          Show
          Robert Joseph Evans added a comment - Yes, that is the exact issue. As part of this change I modified the RpcEngine interface. In trunk and branch-2 all implementations of RpcEngine are in common, but in 0.23 ProtoOverHadoopRpcEngine is in MapReduce so the fix has to be split. I think it is simpler to get the code changes approved and into trunk/branch-2 first and then put in the branch-23 patch at once, instead of splitting them up.
          Hide
          Suresh Srinivas added a comment -

          Unfortunately for 0.23 the RPC engine code is split among the projects so the patches for MAPREDUCE-4079 and HADOOP-8227 need to go in at the same time

          What is the RPC engine code split issue here. Does MAPREDUCE-3955 address this?

          Show
          Suresh Srinivas added a comment - Unfortunately for 0.23 the RPC engine code is split among the projects so the patches for MAPREDUCE-4079 and HADOOP-8227 need to go in at the same time What is the RPC engine code split issue here. Does MAPREDUCE-3955 address this?
          Hide
          Robert Joseph Evans added a comment -

          Unfortunately for 0.23 the RPC engine code is split among the projects so the patches for MAPREDUCE-4079 and HADOOP-8227 need to go in at the same time. I have attached the full patch for branch-0.23 at https://issues.apache.org/jira/secure/attachment/12520617/MR-4079-full-branch-0.23.txt for both of these JIRAs. Once both JIRAs are approved for trunk and branch-2.0, I can put the full patch into branch-0.23.

          Show
          Robert Joseph Evans added a comment - Unfortunately for 0.23 the RPC engine code is split among the projects so the patches for MAPREDUCE-4079 and HADOOP-8227 need to go in at the same time. I have attached the full patch for branch-0.23 at https://issues.apache.org/jira/secure/attachment/12520617/MR-4079-full-branch-0.23.txt for both of these JIRAs. Once both JIRAs are approved for trunk and branch-2.0, I can put the full patch into branch-0.23.
          Hide
          Robert Joseph Evans added a comment -

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.fs.viewfs.TestViewFsTrash

          TestViewFsTrash is known to fail intermittently (HADOOP-8110).

          Show
          Robert Joseph Evans added a comment - -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.viewfs.TestViewFsTrash TestViewFsTrash is known to fail intermittently ( HADOOP-8110 ).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520610/HADOOP-8227-trunk.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.fs.viewfs.TestViewFsTrash

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/802//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/802//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520610/HADOOP-8227-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/802//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/802//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          The current patch is not compatible with the 0.23 line. I am working on a patch for that as well, but I would like feedback on this before going too far down that route.

          Show
          Robert Joseph Evans added a comment - The current patch is not compatible with the 0.23 line. I am working on a patch for that as well, but I would like feedback on this before going too far down that route.
          Hide
          Robert Joseph Evans added a comment -

          This is the first go at the patch. I am not super familiar with all of the IPC code, so I am very open to suggestions about how to make this better.

          Show
          Robert Joseph Evans added a comment - This is the first go at the patch. I am not super familiar with all of the IPC code, so I am very open to suggestions about how to make this better.
          Hide
          Robert Joseph Evans added a comment -

          Yes "security reasons" is very vague. It more accurately is because a very zealous security minded individual that I work with does not want to open up all of the ACLs for all of the ports between every compute node in the cluster and all possible clients . They would much rather limit the set of ports, and I tend to agree with them. Limiting the ports also has the added benefit of being able to monitor that traffic and know with a greater degree of certainty that this given connection is an RPC connection to an AM.

          Show
          Robert Joseph Evans added a comment - Yes "security reasons" is very vague. It more accurately is because a very zealous security minded individual that I work with does not want to open up all of the ACLs for all of the ports between every compute node in the cluster and all possible clients . They would much rather limit the set of ports, and I tend to agree with them. Limiting the ports also has the added benefit of being able to monitor that traffic and know with a greater degree of certainty that this given connection is an RPC connection to an AM.
          Hide
          Aaron T. Myers added a comment -

          Hey Bobby, can you elaborate on how this issue is for "security reasons" ? Thanks a lot.

          Show
          Aaron T. Myers added a comment - Hey Bobby, can you elaborate on how this issue is for "security reasons" ? Thanks a lot.

            People

            • Assignee:
              Robert Joseph Evans
              Reporter:
              Robert Joseph Evans
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development