Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6338

MR AppMaster does not honor ephemeral port range

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: mr-am, mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The MR AppMaster should only use port ranges defined in the yarn.app.mapreduce.am.job.client.port-range property. On initial startup of the MRAppMaster, it does use the port range defined in the property. However, it also opens up a listener on a random ephemeral port. This is not the Jetty listener. It is another listener opened by the MRAppMaster via another thread and is recognized by the RM. Other nodes will try to communicate to it via that random port. With firewall settings on, the MR job will fail because the random port is not opened. This problem has caused others to have all OS ephemeral ports opened to have MR jobs run.

      This is related to MAPREDUCE-4079

      1. MAPREDUCE-6338.002.patch
        3 kB
        Frank Nguyen
      2. MAPREDUCE-6338.003.patch
        1 kB
        John

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11211 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11211/)
          MAPREDUCE-6338. MR AppMaster does not honor ephemeral port range. (junping_du: rev d401e63b6c3695d1f8f3f4958b8d592b15342b17)

          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11211 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11211/ ) MAPREDUCE-6338 . MR AppMaster does not honor ephemeral port range. (junping_du: rev d401e63b6c3695d1f8f3f4958b8d592b15342b17) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java
          Hide
          djp Junping Du added a comment -

          I have commit the patch to trunk and branch-2 with fixing a checkstyle issue. Thanks Frank Nguyen for patch contribution and all for review and comments!

          Show
          djp Junping Du added a comment - I have commit the patch to trunk and branch-2 with fixing a checkstyle issue. Thanks Frank Nguyen for patch contribution and all for review and comments!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 13m 1s trunk passed
          +1 compile 0m 22s trunk passed
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 24s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 34s trunk passed
          +1 javadoc 0m 15s trunk passed
          +1 mvninstall 0m 20s the patch passed
          +1 compile 0m 19s the patch passed
          +1 javac 0m 19s the patch passed
          -1 checkstyle 0m 12s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23)
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 40s the patch passed
          +1 javadoc 0m 12s the patch passed
          +1 unit 8m 57s hadoop-mapreduce-client-app in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          27m 29s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782545/MAPREDUCE-6338.003.patch
          JIRA Issue MAPREDUCE-6338
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0dba66c3f32d 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3ea6d35
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6887/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6887/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6887/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 1s trunk passed +1 compile 0m 22s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 34s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 20s the patch passed +1 compile 0m 19s the patch passed +1 javac 0m 19s the patch passed -1 checkstyle 0m 12s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 40s the patch passed +1 javadoc 0m 12s the patch passed +1 unit 8m 57s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 27m 29s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782545/MAPREDUCE-6338.003.patch JIRA Issue MAPREDUCE-6338 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0dba66c3f32d 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3ea6d35 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6887/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6887/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6887/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Forget my comments above, I double check that we don't involve any new configuration here so this approach looks fine to me. Set to patch available again and will review from there.

          Show
          djp Junping Du added a comment - Forget my comments above, I double check that we don't involve any new configuration here so this approach looks fine to me. Set to patch available again and will review from there.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 42s trunk passed
          +1 compile 0m 24s trunk passed
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 30s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 0m 35s trunk passed
          +1 javadoc 0m 16s trunk passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          -1 checkstyle 0m 13s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 1 new + 22 unchanged - 0 fixed = 23 total (was 22)
          +1 mvnsite 0m 26s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 40s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 9m 4s hadoop-mapreduce-client-app in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          22m 39s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782545/MAPREDUCE-6338.003.patch
          JIRA Issue MAPREDUCE-6338
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e4e53185c4e0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b3befc0
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6822/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6822/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6822/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 42s trunk passed +1 compile 0m 24s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 35s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed -1 checkstyle 0m 13s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 1 new + 22 unchanged - 0 fixed = 23 total (was 22) +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 40s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 9m 4s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 22m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782545/MAPREDUCE-6338.003.patch JIRA Issue MAPREDUCE-6338 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e4e53185c4e0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b3befc0 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6822/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6822/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6822/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Also, we should have a central static IP pool instead of setting port range for specific RPC, that would be nightmare in cluster administration prospective.
          Cancel the patch as I think this goes to wrong direction.

          Show
          djp Junping Du added a comment - Also, we should have a central static IP pool instead of setting port range for specific RPC, that would be nightmare in cluster administration prospective. Cancel the patch as I think this goes to wrong direction.
          Hide
          djp Junping Du added a comment -

          It sounds like many ports for RPC client has this issue - this could against firewall rules with source port checking. I think we should file a umbrella JIRA to track all these issues.

          Show
          djp Junping Du added a comment - It sounds like many ports for RPC client has this issue - this could against firewall rules with source port checking. I think we should file a umbrella JIRA to track all these issues.
          Hide
          xgong Xuan Gong added a comment -

          John Bridge

          Could you please explain why ContainerManagementProtocolProxy opens proxy and what it actually does ?

          AM needs to connect the NM to launch the containers.

          ContainerLauncher #0 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : host:45353
          ContainerLauncher #1 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : host:45358
          

          The "host:45353" is your node address which is randomly provided.

          Is this proxy configurable ? Can I disable proxy ?

          Do not think so.

          Show
          xgong Xuan Gong added a comment - John Bridge Could you please explain why ContainerManagementProtocolProxy opens proxy and what it actually does ? AM needs to connect the NM to launch the containers. ContainerLauncher #0 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : host:45353 ContainerLauncher #1 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : host:45358 The "host:45353" is your node address which is randomly provided. Is this proxy configurable ? Can I disable proxy ? Do not think so.
          Hide
          abalitsky John added a comment -

          Thank you, Varun Saxena, unfortunately these JIRAs' don't solve my problem.

          Let me explain my issue.

          We can see a few strings like this in AM logs when application completes successfully :
          ContainerLauncher #0 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : host:45353
          ContainerLauncher #1 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : host:45358

          These ports are random and, of course, job will hang with firewall enabled :

          ContainerLauncher #4 org.apache.hadoop.ipc.Client: Retrying connect to server : host/192.168.121.94:45353. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

          Could you please explain why ContainerManagementProtocolProxy opens proxy and what it actually does ? Is this proxy configurable ? Can I disable proxy ?

          Thanks.

          Show
          abalitsky John added a comment - Thank you, Varun Saxena , unfortunately these JIRAs' don't solve my problem. Let me explain my issue. We can see a few strings like this in AM logs when application completes successfully : ContainerLauncher #0 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : host:45353 ContainerLauncher #1 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : host:45358 These ports are random and, of course, job will hang with firewall enabled : ContainerLauncher #4 org.apache.hadoop.ipc.Client: Retrying connect to server : host/192.168.121.94:45353. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) Could you please explain why ContainerManagementProtocolProxy opens proxy and what it actually does ? Is this proxy configurable ? Can I disable proxy ? Thanks.
          Hide
          varun_saxena Varun Saxena added a comment -

          HADOOP-12097 and MAPREDUCE-6404 are the JIRAs' which address issue with random web ports.

          Show
          varun_saxena Varun Saxena added a comment - HADOOP-12097 and MAPREDUCE-6404 are the JIRAs' which address issue with random web ports.
          Hide
          abalitsky John added a comment -

          Hello, Frank Nguyen, you said : "Rohith, you are correct, the web port that opens up temporarily while the job is in progress is also random. I think a separate JIRA can be created for it since it does not stop the Job from completing successfully.".

          So, do we have the solution for random web ports ? I'm trying to configure hadoop with 2 nodes and with firewall enable. I can not do this because CantainerLauncher opens random port:

          2015-12-16 16:00:14,775 INFO ContainerLauncher #4
          org.apache.hadoop.ipc.Client: Retrying connect to server:
          ub2simpl500/192.168.121.94:45353. Already tried 9 time(s); retry policy is
          RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
          MILLISECONDS)

          Show
          abalitsky John added a comment - Hello, Frank Nguyen , you said : "Rohith, you are correct, the web port that opens up temporarily while the job is in progress is also random. I think a separate JIRA can be created for it since it does not stop the Job from completing successfully.". So, do we have the solution for random web ports ? I'm trying to configure hadoop with 2 nodes and with firewall enable. I can not do this because CantainerLauncher opens random port: 2015-12-16 16:00:14,775 INFO ContainerLauncher #4 org.apache.hadoop.ipc.Client: Retrying connect to server: ub2simpl500/192.168.121.94:45353. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 55s trunk passed
          +1 compile 0m 20s trunk passed with JDK v1.8.0_66
          +1 compile 0m 23s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 12s trunk passed
          +1 mvnsite 0m 29s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 45s trunk passed
          +1 javadoc 0m 15s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 18s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 19s the patch passed with JDK v1.8.0_66
          +1 javac 0m 19s the patch passed
          +1 compile 0m 21s the patch passed with JDK v1.7.0_91
          +1 javac 0m 21s the patch passed
          -1 checkstyle 0m 11s Patch generated 1 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app (total was 27, now 28).
          +1 mvnsite 0m 26s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 47s the patch passed
          +1 javadoc 0m 13s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 17s the patch passed with JDK v1.7.0_91
          +1 unit 9m 43s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_66.
          +1 unit 9m 54s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          35m 2s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782545/MAPREDUCE-6338.003.patch
          JIRA Issue MAPREDUCE-6338
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6ef076597e6b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1da762c
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6272/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6272/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6272/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 55s trunk passed +1 compile 0m 20s trunk passed with JDK v1.8.0_66 +1 compile 0m 23s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 45s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 18s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 22s the patch passed +1 compile 0m 19s the patch passed with JDK v1.8.0_66 +1 javac 0m 19s the patch passed +1 compile 0m 21s the patch passed with JDK v1.7.0_91 +1 javac 0m 21s the patch passed -1 checkstyle 0m 11s Patch generated 1 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app (total was 27, now 28). +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 47s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 17s the patch passed with JDK v1.7.0_91 +1 unit 9m 43s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_66. +1 unit 9m 54s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 35m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782545/MAPREDUCE-6338.003.patch JIRA Issue MAPREDUCE-6338 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6ef076597e6b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1da762c Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6272/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6272/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6272/console This message was automatically generated.
          Hide
          abalitsky John added a comment -

          cleared

          Show
          abalitsky John added a comment - cleared
          Hide
          bleuleon Frank Nguyen added a comment -

          The patch only requires a small change in:
          hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java
          @@ -127,7 +127,7 @@ protected void startRpcServer() {
          server =
          new RPC.Builder(conf).setProtocol(TaskUmbilicalProtocol.class)
          .setInstance(this).setBindAddress("0.0.0.0")

          • .setPort(0).setNumHandlers(
            + .setPortRangeConfig(MRJobConfig.MR_AM_JOB_CLIENT_PORT_RANGE).setNumHandlers(
            conf.getInt(MRJobConfig.MR_AM_TASK_LISTENER_THREAD_COUNT,
            MRJobConfig.DEFAULT_MR_AM_TASK_LISTENER_THREAD_COUNT))
            .setVerbose(false).setSecretManager(jobTokenSecretManager)

          Other files were not modified. Not sure why it is included in the patch file. For some reason, this small change never passed the Hadoop QA process since last year. For those familiar with this process, feel free to take over and submit a proper patch file. Thanks

          Show
          bleuleon Frank Nguyen added a comment - The patch only requires a small change in: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java @@ -127,7 +127,7 @@ protected void startRpcServer() { server = new RPC.Builder(conf).setProtocol(TaskUmbilicalProtocol.class) .setInstance(this).setBindAddress("0.0.0.0") .setPort(0).setNumHandlers( + .setPortRangeConfig(MRJobConfig.MR_AM_JOB_CLIENT_PORT_RANGE).setNumHandlers( conf.getInt(MRJobConfig.MR_AM_TASK_LISTENER_THREAD_COUNT, MRJobConfig.DEFAULT_MR_AM_TASK_LISTENER_THREAD_COUNT)) .setVerbose(false).setSecretManager(jobTokenSecretManager) Other files were not modified. Not sure why it is included in the patch file. For some reason, this small change never passed the Hadoop QA process since last year. For those familiar with this process, feel free to take over and submit a proper patch file. Thanks
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 46s trunk passed
          +1 compile 8m 40s trunk passed with JDK v1.8.0_66
          +1 compile 8m 30s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 59s trunk passed
          +1 mvnsite 6m 17s trunk passed
          +1 mvneclipse 1m 7s trunk passed
          -1 findbugs 6m 13s branch/hadoop-yarn-project/hadoop-yarn no findbugs output file (hadoop-yarn-project/hadoop-yarn/target/findbugsXml.xml)
          -1 findbugs 3m 33s branch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml)
          +1 javadoc 4m 32s trunk passed with JDK v1.8.0_66
          +1 javadoc 8m 19s trunk passed with JDK v1.7.0_91
          +1 mvninstall 6m 1s the patch passed
          +1 compile 8m 29s the patch passed with JDK v1.8.0_66
          +1 javac 8m 29s the patch passed
          +1 compile 8m 33s the patch passed with JDK v1.7.0_91
          +1 javac 8m 33s the patch passed
          -1 checkstyle 0m 58s Patch generated 1 new checkstyle issues in root (total was 27, now 28).
          +1 mvnsite 6m 9s the patch passed
          +1 mvneclipse 1m 8s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 6m 14s patch/hadoop-yarn-project/hadoop-yarn no findbugs output file (hadoop-yarn-project/hadoop-yarn/target/findbugsXml.xml)
          -1 findbugs 3m 33s patch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml)
          +1 javadoc 4m 29s the patch passed with JDK v1.8.0_66
          +1 javadoc 8m 13s the patch passed with JDK v1.7.0_91
          -1 unit 6m 57s hadoop-common in the patch failed with JDK v1.8.0_66.
          -1 unit 53m 28s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 76m 14s hadoop-yarn in the patch failed with JDK v1.8.0_66.
          -1 unit 133m 3s hadoop-mapreduce-project in the patch failed with JDK v1.8.0_66.
          +1 unit 6m 57s hadoop-common in the patch passed with JDK v1.7.0_91.
          -1 unit 50m 11s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 unit 76m 24s hadoop-yarn in the patch failed with JDK v1.7.0_91.
          -1 unit 139m 10s hadoop-mapreduce-project in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 51s Patch generated 33 ASF License warnings.
          662m 11s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.test.TestTimedOutTestsListener
            hadoop.ha.TestZKFailoverController
            hadoop.ipc.TestRPCWaitForProxy
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
            hadoop.mapred.TestNetworkedJob
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.mapred.TestNetworkedJob



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12728283/MAPREDUCE-6338.002.patch
          JIRA Issue MAPREDUCE-6338
          Optional Tests asflicense mvnsite unit compile javac javadoc mvninstall findbugs checkstyle
          uname Linux 85e7fe3ecddf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1715864
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 46s trunk passed +1 compile 8m 40s trunk passed with JDK v1.8.0_66 +1 compile 8m 30s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 59s trunk passed +1 mvnsite 6m 17s trunk passed +1 mvneclipse 1m 7s trunk passed -1 findbugs 6m 13s branch/hadoop-yarn-project/hadoop-yarn no findbugs output file (hadoop-yarn-project/hadoop-yarn/target/findbugsXml.xml) -1 findbugs 3m 33s branch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml) +1 javadoc 4m 32s trunk passed with JDK v1.8.0_66 +1 javadoc 8m 19s trunk passed with JDK v1.7.0_91 +1 mvninstall 6m 1s the patch passed +1 compile 8m 29s the patch passed with JDK v1.8.0_66 +1 javac 8m 29s the patch passed +1 compile 8m 33s the patch passed with JDK v1.7.0_91 +1 javac 8m 33s the patch passed -1 checkstyle 0m 58s Patch generated 1 new checkstyle issues in root (total was 27, now 28). +1 mvnsite 6m 9s the patch passed +1 mvneclipse 1m 8s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 6m 14s patch/hadoop-yarn-project/hadoop-yarn no findbugs output file (hadoop-yarn-project/hadoop-yarn/target/findbugsXml.xml) -1 findbugs 3m 33s patch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml) +1 javadoc 4m 29s the patch passed with JDK v1.8.0_66 +1 javadoc 8m 13s the patch passed with JDK v1.7.0_91 -1 unit 6m 57s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 53m 28s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 76m 14s hadoop-yarn in the patch failed with JDK v1.8.0_66. -1 unit 133m 3s hadoop-mapreduce-project in the patch failed with JDK v1.8.0_66. +1 unit 6m 57s hadoop-common in the patch passed with JDK v1.7.0_91. -1 unit 50m 11s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 unit 76m 24s hadoop-yarn in the patch failed with JDK v1.7.0_91. -1 unit 139m 10s hadoop-mapreduce-project in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 51s Patch generated 33 ASF License warnings. 662m 11s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.test.TestTimedOutTestsListener   hadoop.ha.TestZKFailoverController   hadoop.ipc.TestRPCWaitForProxy   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.mapreduce.v2.TestMRJobsWithProfiler   hadoop.mapred.TestNetworkedJob JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.TestWriteReadStripedFile   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12728283/MAPREDUCE-6338.002.patch JIRA Issue MAPREDUCE-6338 Optional Tests asflicense mvnsite unit compile javac javadoc mvninstall findbugs checkstyle uname Linux 85e7fe3ecddf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1715864 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6266/console This message was automatically generated.
          Hide
          abalitsky John added a comment -

          Hello,
          do we have any updates about random web app port ?
          Do we have a separate JIRA for this ?

          Show
          abalitsky John added a comment - Hello, do we have any updates about random web app port ? Do we have a separate JIRA for this ?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Frank Nguyen would you mind updating the patch?
          Some comments

          1. The patch has so many other file changes which is not really modified.
          2. For the web app port, earlier version used to support port range, but now it has been modified. I will re look into the code and let you know
          Show
          rohithsharma Rohith Sharma K S added a comment - Frank Nguyen would you mind updating the patch? Some comments The patch has so many other file changes which is not really modified. For the web app port, earlier version used to support port range, but now it has been modified. I will re look into the code and let you know
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 27s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 56s There were no new javac warning messages.
          +1 javadoc 10m 11s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 35s The applied patch generated 1 new checkstyle issues (total was 27, now 28).
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 8s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 9m 17s Tests passed in hadoop-mapreduce-client-app.
              48m 1s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728283/MAPREDUCE-6338.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4620767
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5963/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5963/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5963/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5963/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5963/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 27s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 56s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 35s The applied patch generated 1 new checkstyle issues (total was 27, now 28). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 8s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 9m 17s Tests passed in hadoop-mapreduce-client-app.     48m 1s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728283/MAPREDUCE-6338.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4620767 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5963/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5963/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5963/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5963/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5963/console This message was automatically generated.
          Hide
          michaeledmonds Michael Edmonds added a comment -

          Is there a target version for this patch to make into a distribution release?

          Show
          michaeledmonds Michael Edmonds added a comment - Is there a target version for this patch to make into a distribution release?
          Hide
          bleuleon Frank Nguyen added a comment -

          This patch is working for me. Just got to figure out the checkstyle issue to get this patch accepted.
          Rohith, you are correct, the web port that opens up temporarily while the job is in progress is also random. I think a separate JIRA can be created for it since it does not stop the Job from completing successfully. The problem might be in MRClientService.java at the following line:
          webApp =
          WebApps.$for("mapreduce", AppContext.class, appContext, "ws")
          .withHttpPolicy(conf, Policy.HTTP_ONLY).start(new AMWebApp());

          Show
          bleuleon Frank Nguyen added a comment - This patch is working for me. Just got to figure out the checkstyle issue to get this patch accepted. Rohith, you are correct, the web port that opens up temporarily while the job is in progress is also random. I think a separate JIRA can be created for it since it does not stop the Job from completing successfully. The problem might be in MRClientService.java at the following line: webApp = WebApps.$for("mapreduce", AppContext.class, appContext, "ws") .withHttpPolicy(conf, Policy.HTTP_ONLY).start(new AMWebApp());
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          The patch looks good to me..
          ApplicationMaster web port also random, I think it is good to make it also in port range.. Any thoughts?

          Show
          rohithsharma Rohith Sharma K S added a comment - The patch looks good to me.. ApplicationMaster web port also random, I think it is good to make it also in port range.. Any thoughts?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 34s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 5m 22s The applied patch generated 1 additional checkstyle issues.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 0m 56s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 10m 13s Tests passed in hadoop-mapreduce-client-app.
              50m 40s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728283/MAPREDUCE-6338.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 618ba70
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5453/artifact/patchprocess/checkstyle-result-diff.txt
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5453/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5453/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5453/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 34s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 22s The applied patch generated 1 additional checkstyle issues. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 56s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 10m 13s Tests passed in hadoop-mapreduce-client-app.     50m 40s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728283/MAPREDUCE-6338.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 618ba70 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5453/artifact/patchprocess/checkstyle-result-diff.txt hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5453/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5453/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5453/console This message was automatically generated.
          Hide
          bleuleon Frank Nguyen added a comment -

          Minor change to previous patch file.

          Show
          bleuleon Frank Nguyen added a comment - Minor change to previous patch file.
          Hide
          bleuleon Frank Nguyen added a comment -

          Patch was tested with 3 node cluster. After an MR job was submitted, netstat was ran on the node running MRAppMaster to verify that the IPC ports opened were within the range defined in the MRJobConfig.MR_AM_JOB_CLIENT_PORT_RANGE property.

          Show
          bleuleon Frank Nguyen added a comment - Patch was tested with 3 node cluster. After an MR job was submitted, netstat was ran on the node running MRAppMaster to verify that the IPC ports opened were within the range defined in the MRJobConfig.MR_AM_JOB_CLIENT_PORT_RANGE property.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 33s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          -1 javac 4m 20s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728197/MAPREDUCE-6338.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a00e001
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5451/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 javac 4m 20s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728197/MAPREDUCE-6338.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a00e001 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5451/console This message was automatically generated.
          Hide
          bleuleon Frank Nguyen added a comment -

          The fix for this is to update org/apache/hadoop/mapred/TaskAttemptListenerImpl.java (2.6.0)
          Change line 124:
          from
          .setPort(0).setNumHandlers(
          to:
          setPortRangeConfig(MRJobConfig.MR_AM_JOB_CLIENT_PORT_RANGE).setNumHandlers(

          Preliminary testing shows that the MRAppMaster will only open up RPC listeners on ports defined within the range property.

          Show
          bleuleon Frank Nguyen added a comment - The fix for this is to update org/apache/hadoop/mapred/TaskAttemptListenerImpl.java (2.6.0) Change line 124: from .setPort(0).setNumHandlers( to: setPortRangeConfig(MRJobConfig.MR_AM_JOB_CLIENT_PORT_RANGE).setNumHandlers( Preliminary testing shows that the MRAppMaster will only open up RPC listeners on ports defined within the range property.

            People

            • Assignee:
              bleuleon Frank Nguyen
              Reporter:
              bleuleon Frank Nguyen
            • Votes:
              5 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development