Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2947

Sort fails on YARN+MR with lots of task failures

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      Karam Singh(the great man the world hardly knows about) found lots of failing tasks while running sort on a 350 node cluster. The failed tasks eventually failed the job and this happening consistently on the big cluster.

      Container launch failed for container_1315410418107_0002_01_002511 : RemoteTrace: java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:218) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:129) at java.nio.ByteBuffer.get(ByteBuffer.java:675) at com.google.protobuf.ByteString.copyFrom(ByteString.java:108) at com.google.protobuf.ByteString.copyFrom(ByteString.java:117) at org.apache.hadoop.yarn.util.ProtoUtils.convertToProtoFormat(ProtoUtils.java:97) at org.apache.hadoop.yarn.api.records.ProtoBase.convertToProtoFormat(ProtoBase.java:59) at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl.access$100(StartContainerResponsePBImpl.java:35) at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl$1$1.next(StartContainerResponsePBImpl.java:134) at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl$1$1.next(StartContainerResponsePBImpl.java:122) at com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:319) at org.apache.hadoop.yarn.proto.YarnServiceProtos$StartContainerResponseProto$Builder.addAllServiceResponse(YarnServiceProtos.java:12620) at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl.addServiceResponseToProto(StartContainerResponsePBImpl.java:144) at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl.mergeLocalToBuilder(StartContainerResponsePBImpl.java:60) at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl.mergeLocalToProto(StartContainerResponsePBImpl.java:68) at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl.getProto(StartContainerResponsePBImpl.java:52) at org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:69) at org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83) at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:337) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1496) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1492) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1490) at LocalTrace: org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:151) at $Proxy20.startContainer(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:81) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:215) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)

      1. MAPREDUCE-2947-20110907.txt
        6 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

            People

            • Assignee:
              Vinod Kumar Vavilapalli
              Reporter:
              Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development