Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2857

ConcurrentModificationException in ContainerLogAppender

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Context:

      • Hadoop-2.3.0
      • Using Oozie 4.0.1
      • Pig version 0.11.x

      The job is submitted by Oozie to launch Pig script.

      The following exception traces were found on MR task log:

      In syslog:

      2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException
      java.util.ConcurrentModificationException
      	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
      	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
      	at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
      	at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
      	at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
      	at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
      	at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
      	at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
      	at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
      	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
      2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
      

      in stderr:

      java.util.ConcurrentModificationException
      	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
      	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
      	at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
      	at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
      	at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
      	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
      	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
      	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
      	at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
      	at org.apache.pig.Main.configureLog4J(Main.java:740)
      	at org.apache.pig.Main.run(Main.java:384)
      	at org.apache.pig.PigRunner.run(PigRunner.java:49)
      	at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
      	at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
      	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
      	at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
      

        Attachments

        1. YARN-2857.3.patch
          3 kB
          Mohammad Islam
        2. MAPREDUCE-6139-test.01.patch
          2 kB
          Gera Shegalov
        3. MAPREDUCE-6139.3.patch
          1 kB
          Mohammad Islam
        4. MAPREDUCE-6139.2.patch
          1 kB
          Mohammad Islam
        5. MAPREDUCE-6139.1.patch
          1 kB
          Mohammad Islam
        6. ContainerLogAppender.java
          5 kB
          Mohammad Islam

          Activity

            People

            • Assignee:
              kamrul Mohammad Islam
              Reporter:
              kamrul Mohammad Islam
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: