Details

      Description

      It looks like 4 threads in the AM died with OOM but not the one pinging the RM.

      stderr for this AM

      WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
      May 30, 2012 4:49:55 AM com.google.inject.servlet.InternalServletModule$BackwardsCompatibleServletContextProvider get
      WARNING: You are attempting to use a deprecated API (specifically, attempting to @Inject ServletContext inside an eagerly created singleton. While we allow this for backwards compatibility, be warned that this MAY have unexpected behavior if you have more than one injector (with ServletModule) running in the same JVM. Please consult the Guice documentation at http://code.google.com/p/google-guice/wiki/Servlets for more information.
      May 30, 2012 4:49:55 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
      INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class
      May 30, 2012 4:49:55 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
      INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
      May 30, 2012 4:49:55 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
      INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class
      May 30, 2012 4:49:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
      INFO: Initiating Jersey application, version 'Jersey: 1.8 06/24/2011 12:17 PM'
      May 30, 2012 4:49:55 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
      INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"
      May 30, 2012 4:49:56 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
      INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
      May 30, 2012 4:49:56 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
      INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest"
      Exception in thread "ResponseProcessor for block BP-1114822160-<IP>-1322528669066:blk_-6528896407411719649_34227308" java.lang.OutOfMemoryError: Java heap space
      	at com.google.protobuf.CodedInputStream.(CodedInputStream.java:538)
      	at com.google.protobuf.CodedInputStream.newInstance(CodedInputStream.java:55)
      	at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:201)
      	at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:738)
      	at org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos$PipelineAckProto.parseFrom(DataTransferProtos.java:7287)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:95)
      	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:656)
      Exception in thread "DefaultSpeculator background processing" java.lang.OutOfMemoryError: Java heap space
      	at java.util.HashMap.resize(HashMap.java:462)
      	at java.util.HashMap.addEntry(HashMap.java:755)
      	at java.util.HashMap.put(HashMap.java:385)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.getTasks(JobImpl.java:632)
      	at org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.maybeScheduleASpeculation(DefaultSpeculator.java:465)
      	at org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.maybeScheduleAMapSpeculation(DefaultSpeculator.java:433)
      	at org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.computeSpeculations(DefaultSpeculator.java:509)
      	at org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.access$100(DefaultSpeculator.java:56)
      	at org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator$1.run(DefaultSpeculator.java:176)
      	at java.lang.Thread.run(Thread.java:619)
      Exception in thread "Timer for 'MRAppMaster' metrics system" java.lang.OutOfMemoryError: Java heap space
      Exception in thread "Socket Reader #4 for port 50500" java.lang.OutOfMemoryError: Java heap space
      
      1. StackDump.txt
        34 kB
        Robert Joseph Evans
      2. MR-4300.txt
        14 kB
        Robert Joseph Evans
      3. MR-4300.txt
        15 kB
        Robert Joseph Evans
      4. MR-4300.txt
        14 kB
        Robert Joseph Evans

        Activity

        Robert Joseph Evans created issue -
        Robert Joseph Evans made changes -
        Field Original Value New Value
        Attachment StackDump.txt [ 12533082 ]
        Robert Joseph Evans made changes -
        Assignee Robert Joseph Evans [ revans2 ]
        Robert Joseph Evans made changes -
        Attachment MR-4300.txt [ 12533094 ]
        Robert Joseph Evans made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Robert Joseph Evans made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Robert Joseph Evans made changes -
        Attachment MR-4300.txt [ 12533113 ]
        Robert Joseph Evans made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Robert Joseph Evans made changes -
        Attachment MR-4300.txt [ 12533365 ]
        Thomas Graves made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.23.3 [ 12320060 ]
        Fix Version/s 2.0.1-alpha [ 12321442 ]
        Fix Version/s 3.0.0 [ 12320355 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322471 ]
        Fix Version/s 3.0.0 [ 12320355 ]
        Fix Version/s 2.1.0-alpha [ 12321442 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Robert Joseph Evans
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development