Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4145

ConcurrentModificationException in TT if any one of the local dirs is invalid

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.1
    • Fix Version/s: None
    • Component/s: tasktracker
    • Labels:
      None
    • Environment:

      Ubuntu Linux 10.04 w/ Hadoop installed via Bigtop 0.3.0

      Description

      TT won't come up, stack trace instead. Looks like the deletion of invalid dirs breaks the iterator going through the list of dirs.

        Activity

        Hide
        Steve Loughran added a comment -

        stack
        2012-04-12 18:56:16,187 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
        /************************************************************
        STARTUP_MSG: Starting TaskTracker
        STARTUP_MSG: host = ranier/192.168.1.92
        STARTUP_MSG: args = []
        STARTUP_MSG: version = 1.0.1
        STARTUP_MSG: build = -r ; compiled by 'jenkins' on Tue Mar 20 16:09:05 UTC 2012
        ************************************************************/
        2012-04-12 18:56:16,453 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
        2012-04-12 18:56:16,531 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
        2012-04-12 18:56:16,532 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
        2012-04-12 18:56:16,532 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
        2012-04-12 18:56:16,865 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
        2012-04-12 18:56:16,870 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
        2012-04-12 18:56:22,135 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
        2012-04-12 18:56:22,215 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
        2012-04-12 18:56:22,238 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker local dir /var/lib/hadoop/cache/mapred/mapred/local error can not create directory: /var/lib/hadoop/cache/mapred/mapred/local, removing from local dirs
        2012-04-12 18:56:22,239 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.util.ConcurrentModificationException
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:819)
        at java.util.ArrayList$Itr.next(ArrayList.java:791)
        at org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:211)
        at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1447)
        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3716)

        Show
        Steve Loughran added a comment - stack 2012-04-12 18:56:16,187 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting TaskTracker STARTUP_MSG: host = ranier/192.168.1.92 STARTUP_MSG: args = [] STARTUP_MSG: version = 1.0.1 STARTUP_MSG: build = -r ; compiled by 'jenkins' on Tue Mar 20 16:09:05 UTC 2012 ************************************************************/ 2012-04-12 18:56:16,453 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2012-04-12 18:56:16,531 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2012-04-12 18:56:16,532 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2012-04-12 18:56:16,532 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started 2012-04-12 18:56:16,865 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2012-04-12 18:56:16,870 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2012-04-12 18:56:22,135 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2012-04-12 18:56:22,215 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2012-04-12 18:56:22,238 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker local dir /var/lib/hadoop/cache/mapred/mapred/local error can not create directory: /var/lib/hadoop/cache/mapred/mapred/local, removing from local dirs 2012-04-12 18:56:22,239 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:819) at java.util.ArrayList$Itr.next(ArrayList.java:791) at org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:211) at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1447) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3716)

          People

          • Assignee:
            Unassigned
            Reporter:
            Steve Loughran
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development