Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6514

Job hangs as ask is not updated after ramping down of all reducers

    Details

      Description

      In RMContainerAllocator#preemptReducesIfNeeded, we simply clear the scheduled reduces map and put these reducers to pending. This is not updated in ask. So RM keeps on assigning and AM is not able to assign as no reducer is scheduled(check logs below the code).
      If this is updated immediately, RM will be able to schedule mappers immediately which anyways is the intention when we ramp down reducers.
      Scheduler need not allocate for ramped down reducers
      This if not handled can lead to map starvation as pointed out in MAPREDUCE-6513

       LOG.info("Ramping down all scheduled reduces:"
                  + scheduledRequests.reduces.size());
              for (ContainerRequest req : scheduledRequests.reduces.values()) {
                pendingReduces.add(req);
              }
              scheduledRequests.reduces.clear();
      
      2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container not assigned : container_1437451211867_1485_01_000215
      2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Cannot assign container Container: [ContainerId: container_1437451211867_1485_01_000216, NodeId: hdszzdcxdat6g06u04p:26009, NodeHttpAddress: hdszzdcxdat6g06u04p:26010, Resource: <memory:4096, vCores:1>, Priority: 10, Token: Token { kind: ContainerToken, service: 10.2.33.236:26009 }, ] for a reduce as either  container memory less than required 4096 or no pending reduce tasks - reduces.isEmpty=true
      2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container not assigned : container_1437451211867_1485_01_000216
      2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Cannot assign container Container: [ContainerId: container_1437451211867_1485_01_000217, NodeId: hdszzdcxdat6g06u06p:26009, NodeHttpAddress: hdszzdcxdat6g06u06p:26010, Resource: <memory:4096, vCores:1>, Priority: 10, Token: Token { kind: ContainerToken, service: 10.2.33.239:26009 }, ] for a reduce as either  container memory less than required 4096 or no pending reduce tasks - reduces.isEmpty=true
      
      1. MAPREDUCE-6514.01.patch
        10 kB
        Varun Saxena
      2. MAPREDUCE-6514.02.patch
        11 kB
        Wangda Tan

        Issue Links

          Activity

          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Thanks Varun Saxena for creating issue.

          I too have seen many times this particular problem because of not updating ResourceRequest when Ramping down the reducers. As a result, number of containers count in the ask is increased as explained in the below

          Consider MR job is running with 3 mapper and 100 reducers. All mappers were finished, and 3 reducers are running in the cluster.
          Kill the task attempt of mapper. So Ramping down will happen.

          2015-10-17 12:09:10,836 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all scheduled reduces:97
          

          Since resourcerequest is not decremented from remoteRequestsTable. Next time when ask for 97 reducers i.e pendingReducers --> scheduledReducers, which starts adding number of containers to existing RR. So 97(previous outstanding reducers)+ 100(total reducers)=197

          2015-10-17 12:10:49,617 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps assigned. Ramping up all remaining reduces:100
          2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=98 #asks=2
          2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=99 #asks=2
          2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=100 #asks=2
          ......
          ......
          2015-10-17 12:10:49,647 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=195 #asks=2
          2015-10-17 12:10:49,648 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=196 #asks=2
          2015-10-17 12:10:49,648 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=197 #asks=2
          

          In ResourceManager.log
          In the log, observe that Containers: 100 before ramping down all the reducers. Afterwords, MR ask for reducers requests, Containers: 197

          2015-10-17 12:07:40,613 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: showRequests: application=application_1445063712158_0001 request={Priority: 10, Capability: <memory:2048, vCores:1>, # Containers: 100, Location: *, Relax Locality: true, Node Label Expression: }
          

          After ramping down all the reducers and request for ask.

          2015-10-17 12:10:50,651 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: showRequests: application=application_1445063712158_0001 request={Priority: 10, Capability: <memory:2048, vCores:1>, # Containers: 197, Location: *, Relax Locality: true, Node Label Expression: }
          

          Basically it causes unnecessary overhead on ResourceManager that for assignment and also effect other applications container assignment. These allocated containers are rejected by MR.

          Show
          rohithsharma Rohith Sharma K S added a comment - Thanks Varun Saxena for creating issue. I too have seen many times this particular problem because of not updating ResourceRequest when Ramping down the reducers. As a result, number of containers count in the ask is increased as explained in the below Consider MR job is running with 3 mapper and 100 reducers. All mappers were finished, and 3 reducers are running in the cluster. Kill the task attempt of mapper. So Ramping down will happen. 2015-10-17 12:09:10,836 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all scheduled reduces:97 Since resourcerequest is not decremented from remoteRequestsTable. Next time when ask for 97 reducers i.e pendingReducers --> scheduledReducers , which starts adding number of containers to existing RR. So 97(previous outstanding reducers)+ 100(total reducers)=197 2015-10-17 12:10:49,617 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps assigned. Ramping up all remaining reduces:100 2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=98 #asks=2 2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=99 #asks=2 2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=100 #asks=2 ...... ...... 2015-10-17 12:10:49,647 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=195 #asks=2 2015-10-17 12:10:49,648 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=196 #asks=2 2015-10-17 12:10:49,648 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=197 #asks=2 In ResourceManager.log In the log, observe that Containers: 100 before ramping down all the reducers. Afterwords, MR ask for reducers requests, Containers: 197 2015-10-17 12:07:40,613 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: showRequests: application=application_1445063712158_0001 request={Priority: 10, Capability: <memory:2048, vCores:1>, # Containers: 100, Location: *, Relax Locality: true, Node Label Expression: } After ramping down all the reducers and request for ask. 2015-10-17 12:10:50,651 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: showRequests: application=application_1445063712158_0001 request={Priority: 10, Capability: <memory:2048, vCores:1>, # Containers: 197, Location: *, Relax Locality: true, Node Label Expression: } Basically it causes unnecessary overhead on ResourceManager that for assignment and also effect other applications container assignment. These allocated containers are rejected by MR.
          Hide
          sunilg Sunil G added a comment -

          Yes Rohith Sharma K S, thanks for the detailed analysis. This improvement can definitely avoid many unnecessary allocations from RM side.
          And if any extra container allocation happened from RM side within a heartbeat interval (AM requested less number of reducers, however more containers were already allocated by RM), such containers has to be rejected by AM and we already do that. I hope this number wont be big as it may happen only within a heartbeat interval.

          Show
          sunilg Sunil G added a comment - Yes Rohith Sharma K S , thanks for the detailed analysis. This improvement can definitely avoid many unnecessary allocations from RM side. And if any extra container allocation happened from RM side within a heartbeat interval (AM requested less number of reducers, however more containers were already allocated by RM), such containers has to be rejected by AM and we already do that. I hope this number wont be big as it may happen only within a heartbeat interval.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 30s trunk passed
          +1 compile 0m 17s trunk passed with JDK v1.8.0_60
          +1 compile 0m 19s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 12s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 45s trunk passed
          +1 javadoc 0m 16s trunk passed with JDK v1.8.0_60
          +1 javadoc 0m 17s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.8.0_60
          +1 javac 0m 15s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.7.0_79
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 50s the patch passed
          +1 javadoc 0m 14s the patch passed with JDK v1.8.0_60
          +1 javadoc 0m 18s the patch passed with JDK v1.7.0_79
          -1 unit 9m 42s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_60.
          -1 unit 10m 9s hadoop-mapreduce-client-app in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 22s Patch generated 7 ASF License warnings.
          29m 45s



          Reason Tests
          JDK v1.7.0_79 Failed junit tests hadoop.mapreduce.v2.app.TestRecovery
            hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
            hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-29
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769614/MAPREDUCE-6514.01.patch
          JIRA Issue MAPREDUCE-6514
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 1f4486852554 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-c3a2069/precommit/personality/hadoop.sh
          git revision trunk / 58d1df5
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 30s trunk passed +1 compile 0m 17s trunk passed with JDK v1.8.0_60 +1 compile 0m 19s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 12s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 45s trunk passed +1 javadoc 0m 16s trunk passed with JDK v1.8.0_60 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 21s the patch passed +1 compile 0m 15s the patch passed with JDK v1.8.0_60 +1 javac 0m 15s the patch passed +1 compile 0m 17s the patch passed with JDK v1.7.0_79 +1 javac 0m 17s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 50s the patch passed +1 javadoc 0m 14s the patch passed with JDK v1.8.0_60 +1 javadoc 0m 18s the patch passed with JDK v1.7.0_79 -1 unit 9m 42s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_60. -1 unit 10m 9s hadoop-mapreduce-client-app in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 22s Patch generated 7 ASF License warnings. 29m 45s Reason Tests JDK v1.7.0_79 Failed junit tests hadoop.mapreduce.v2.app.TestRecovery   hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler   hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-29 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769614/MAPREDUCE-6514.01.patch JIRA Issue MAPREDUCE-6514 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 1f4486852554 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-c3a2069/precommit/personality/hadoop.sh git revision trunk / 58d1df5 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6099/console This message was automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Test failure to be handled by YARN-4320

          Show
          varun_saxena Varun Saxena added a comment - Test failure to be handled by YARN-4320
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Comment on current patch

          You should look at rampDownReduces() API and use it instead of hand-rolling decContainerReq. I actually think once we do this, you should remove clearAllPendingReduceRequests() altogether.

          I am looking at branch-2 and I think the current patch is better served on top of MAPREDUCE-6302 (and this only in 2.8+) given the numerous changes made there. The patch obviously doesn't apply on branch-2.7 which you set the target-version as (2.7.2). Canceling the patch.

          Meta thought

          If MAPREDUCE-6513 goes through per my latest proposal there, there is no need for canceling all the reduce asks and thus this patch, no?

          Release

          IAC, this has been a long-standing problem (though I'm very surprised nobody caught this till now), so I'd propose we move this out into 2.7.3 or better 2.8+ so I can make progress on the 2.7.2 release. Thoughts?

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Comment on current patch You should look at rampDownReduces() API and use it instead of hand-rolling decContainerReq . I actually think once we do this, you should remove clearAllPendingReduceRequests() altogether. I am looking at branch-2 and I think the current patch is better served on top of MAPREDUCE-6302 (and this only in 2.8+) given the numerous changes made there. The patch obviously doesn't apply on branch-2.7 which you set the target-version as (2.7.2). Canceling the patch. Meta thought If MAPREDUCE-6513 goes through per my latest proposal there, there is no need for canceling all the reduce asks and thus this patch, no? Release IAC, this has been a long-standing problem (though I'm very surprised nobody caught this till now), so I'd propose we move this out into 2.7.3 or better 2.8+ so I can make progress on the 2.7.2 release. Thoughts?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Rebased patch to latest trunk, and addressed comments from Vinod Kumar Vavilapalli:

          Use rampDownReduces to cancel all reducer requests.

          Show
          leftnoteasy Wangda Tan added a comment - Rebased patch to latest trunk, and addressed comments from Vinod Kumar Vavilapalli : Use rampDownReduces to cancel all reducer requests.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 7s trunk passed
          +1 compile 0m 20s trunk passed with JDK v1.8.0_91
          +1 compile 0m 22s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 46s trunk passed
          +1 javadoc 0m 15s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 18s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.8.0_91
          +1 javac 0m 17s the patch passed
          +1 compile 0m 20s the patch passed with JDK v1.7.0_95
          +1 javac 0m 20s the patch passed
          +1 checkstyle 0m 14s the patch passed
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 53s the patch passed
          +1 javadoc 0m 13s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95
          +1 unit 8m 13s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_91.
          +1 unit 8m 49s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          31m 47s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802482/MAPREDUCE-6514.02.patch
          JIRA Issue MAPREDUCE-6514
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 134bb84d3e93 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d0da132
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6483/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6483/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 7s trunk passed +1 compile 0m 20s trunk passed with JDK v1.8.0_91 +1 compile 0m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 46s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 18s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_91 +1 javac 0m 17s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_95 +1 javac 0m 20s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95 +1 unit 8m 13s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_91. +1 unit 8m 49s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 31m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802482/MAPREDUCE-6514.02.patch JIRA Issue MAPREDUCE-6514 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 134bb84d3e93 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d0da132 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6483/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6483/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          The patch looks good to me now. Tx Wangda Tan for finishing the patch on behalf of Varun Saxena!

          BTW, why is this a blocker on the 2.6 / 2.7 maint lines?

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - The patch looks good to me now. Tx Wangda Tan for finishing the patch on behalf of Varun Saxena ! BTW, why is this a blocker on the 2.6 / 2.7 maint lines?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Vinod Kumar Vavilapalli,

          This behavior is commented by Rohith Sharma K S above in above comment,

          As a result, number of containers count in the ask is increased as explained in the below...

          And because of MAPREDUCE-6302, it is possible that MR AM cancel all reducer requests and re-add all reducer requests at the same heartbeat, so the #containers increases fast in RM side and finally it becomes a enormous number. Since MAPREDUCE-6302 is included by branch-2.6/2.7, we need to back port this patch to branch-2.6/2.7 as well.

          Show
          leftnoteasy Wangda Tan added a comment - Vinod Kumar Vavilapalli , This behavior is commented by Rohith Sharma K S above in above comment, As a result, number of containers count in the ask is increased as explained in the below... And because of MAPREDUCE-6302 , it is possible that MR AM cancel all reducer requests and re-add all reducer requests at the same heartbeat, so the #containers increases fast in RM side and finally it becomes a enormous number. Since MAPREDUCE-6302 is included by branch-2.6/2.7, we need to back port this patch to branch-2.6/2.7 as well.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9728 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9728/)
          MAPREDUCE-6514. Fixed MapReduce ApplicationMaster to properly updated (vinodkv: rev 8d48266720dcf0e71cfd87fef18b60a53aa1bef9)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9728 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9728/ ) MAPREDUCE-6514 . Fixed MapReduce ApplicationMaster to properly updated (vinodkv: rev 8d48266720dcf0e71cfd87fef18b60a53aa1bef9) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Okay, makes sense.

          The 2.7 and 2.6 commits ran into conflicts and needed trivial fixes - I did them during commit time.

          Committed this to trunk, branch-2, branch-2.8, branch-2.7 and branch-2.6. Thanks Varun Saxena and Wangda Tan!

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Okay, makes sense. The 2.7 and 2.6 commits ran into conflicts and needed trivial fixes - I did them during commit time. Committed this to trunk, branch-2, branch-2.8, branch-2.7 and branch-2.6. Thanks Varun Saxena and Wangda Tan !
          Hide
          varun_saxena Varun Saxena added a comment -

          Thanks Wangda Tan.
          Sorry could not address the review comment from Vinod yesterday as it was late night here and waiting for QA report would have meant an even longer wait.

          Show
          varun_saxena Varun Saxena added a comment - Thanks Wangda Tan . Sorry could not address the review comment from Vinod yesterday as it was late night here and waiting for QA report would have meant an even longer wait.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          vinishavyasa vinisha added a comment -

          can you please add affect version to the Jira.

          Show
          vinishavyasa vinisha added a comment - can you please add affect version to the Jira.

            People

            • Assignee:
              varun_saxena Varun Saxena
              Reporter:
              varun_saxena Varun Saxena
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development