Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2846

a small % of all tasks fail with DefaultTaskController

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed a race condition in writing the log index file that caused tasks to 'fail'.

      Description

      After upgrading our test 0.20.203 grid to 0.20.204-rc2, we ran terasort to verify operation. While the job completed successfully, approx 10% of the tasks failed with task runner execution errors and the inability to create symlinks for attempt logs.

      1. sync.patch
        0.9 kB
        Owen O'Malley
      2. sync-trunk.patch
        1 kB
        Owen O'Malley

        Issue Links

          Activity

          Hide
          Allen Wittenauer added a comment -

          Some other details:

          a) we do not have security on
          b) we are not using the LinuxTaskController

          Show
          Allen Wittenauer added a comment - Some other details: a) we do not have security on b) we are not using the LinuxTaskController
          Hide
          Eli Collins added a comment -

          MR-2413 (disk failure handling) and MR-2415 (distribute userlogs onto multiple disks) were only implemented for LinuxTaskController but change some of the relevant code, might be worth looking into.

          Show
          Eli Collins added a comment - MR-2413 (disk failure handling) and MR-2415 (distribute userlogs onto multiple disks) were only implemented for LinuxTaskController but change some of the relevant code, might be worth looking into.
          Hide
          Allen Wittenauer added a comment -

          nods I'm mostly convinced it is a race condition in MR-2415. I haven't had enough time to start playing in the source to track it down more. I did talk to Owen about already, but thought it might be useful to at least get the JIRA filed to put more eyes on it since race conditions are usually pretty awful to track down.

          Show
          Allen Wittenauer added a comment - nods I'm mostly convinced it is a race condition in MR-2415. I haven't had enough time to start playing in the source to track it down more. I did talk to Owen about already, but thought it might be useful to at least get the JIRA filed to put more eyes on it since race conditions are usually pretty awful to track down.
          Hide
          Bharath Mundlapudi added a comment -

          Hi Allen,

          Can you post how you are configuring mapred.local.dir values? We have not seen this problem in our cluster since we run with Linux task controller. But, Eli is right, we did change Default task controller to make it consistent. Giving more information will help us to understand better like how many disks you have, mapred.local.dir value etc. or even mapred-site.xml. I am asking this information to get an idea of how we can reproduce in our test cluster?

          Show
          Bharath Mundlapudi added a comment - Hi Allen, Can you post how you are configuring mapred.local.dir values? We have not seen this problem in our cluster since we run with Linux task controller. But, Eli is right, we did change Default task controller to make it consistent. Giving more information will help us to understand better like how many disks you have, mapred.local.dir value etc. or even mapred-site.xml. I am asking this information to get an idea of how we can reproduce in our test cluster?
          Hide
          Allen Wittenauer added a comment -

          Some relevant properties:

          <property>
          <name>mapred.local.dir</name>
          <value>/grid/a/mapred/local,/grid/b/mapred/local,/grid/c/mapred/local,/grid/d/mapred/local,/grid/e/mapred/local,/grid/f/mapred/local</value>
          </property>

          <property>
          <name>hadoop.job.history.user.location</name>
          <value>none</value>
          <final>true</final>
          </property>

          <property>
          <name>hadoop.tmp.dir</name>
          <value>/grid/a/mapred/tmp/hadoop-$

          {user.name}

          </value>
          <description>A base for other temporary directories.</description>
          <final>true</final>
          </property>

          The permissions on these dirs are 775. User and group match the user we run the tasktracker as. (So, with DefaultTaskController, this should work just fine.)

          Some other questions I've been asked over IM:

          • Nodes can show failures with one run, be perfectly clean the next, then show failures during a third run. Some nodes will throw failures during all three.
          • This problem is reflected in both map tasks and reduce tasks.
          • The dir permissions really are the same across all dirs and all nodes.
          • I have not tried LTC because my test grid is not configured to support it yet.
          • I've been testing the Apache releases with no custom patches other than including the LZO bits.
          • The number of failures per run is wildly inconsistent.
          • Running 203 on the same gear with the same config shows zero failures. So this is clearly a result of something added in 204.
          • Yes, enough tasks have failed during certain runs that tasktrackers are getting blacklisted from the job.

          I'm currently playing with a debug jar from Owen to try and gather more information. Part of the problem is that there clearly isn't enough information on why tasks are failing. The tasktracker logs throw the symlink error but see MAPREDUCE-2804. The child error stack trace:

          java.lang.Throwable: Child Error
          	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
          Caused by: java.io.IOException: Task process exit with nonzero status of -1.
          	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
          

          is equally unhelpful.

          Show
          Allen Wittenauer added a comment - Some relevant properties: <property> <name>mapred.local.dir</name> <value>/grid/a/mapred/local,/grid/b/mapred/local,/grid/c/mapred/local,/grid/d/mapred/local,/grid/e/mapred/local,/grid/f/mapred/local</value> </property> <property> <name>hadoop.job.history.user.location</name> <value>none</value> <final>true</final> </property> <property> <name>hadoop.tmp.dir</name> <value>/grid/a/mapred/tmp/hadoop-$ {user.name} </value> <description>A base for other temporary directories.</description> <final>true</final> </property> The permissions on these dirs are 775. User and group match the user we run the tasktracker as. (So, with DefaultTaskController, this should work just fine.) Some other questions I've been asked over IM: Nodes can show failures with one run, be perfectly clean the next, then show failures during a third run. Some nodes will throw failures during all three. This problem is reflected in both map tasks and reduce tasks. The dir permissions really are the same across all dirs and all nodes. I have not tried LTC because my test grid is not configured to support it yet. I've been testing the Apache releases with no custom patches other than including the LZO bits. The number of failures per run is wildly inconsistent. Running 203 on the same gear with the same config shows zero failures. So this is clearly a result of something added in 204. Yes, enough tasks have failed during certain runs that tasktrackers are getting blacklisted from the job. I'm currently playing with a debug jar from Owen to try and gather more information. Part of the problem is that there clearly isn't enough information on why tasks are failing. The tasktracker logs throw the symlink error but see MAPREDUCE-2804 . The child error stack trace: java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of -1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) is equally unhelpful.
          Hide
          Owen O'Malley added a comment -

          Offline, Allen gave me a stack trace:

          java.io.FileNotFoundException: File /export/apps/hadoop/hadoop-0.20.204.0/logs/userlogs/job_201108100052_0008/attempt_201108100052_0008_r_000145_0/log.tmp does not exist.
          at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
          at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:210)
          at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:160)
          at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:261)
          at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:406)
          at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:345)
          at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:391)
          at org.apache.hadoop.mapred.Child.main(Child.java:235)

          Based on this, I discovered that there is a missing synchronization in writeToIndexFile. This seems to reduce the failures that Allen is seeing.

          Show
          Owen O'Malley added a comment - Offline, Allen gave me a stack trace: java.io.FileNotFoundException: File /export/apps/hadoop/hadoop-0.20.204.0/logs/userlogs/job_201108100052_0008/attempt_201108100052_0008_r_000145_0/log.tmp does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:210) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:160) at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:261) at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:406) at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:345) at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:391) at org.apache.hadoop.mapred.Child.main(Child.java:235) Based on this, I discovered that there is a missing synchronization in writeToIndexFile. This seems to reduce the failures that Allen is seeing.
          Hide
          Owen O'Malley added a comment -

          This adds synchronization to writeToIndexFile on the 2xx branch.

          Since the TT is going away in trunk, there isn't a need to fix it there.

          Show
          Owen O'Malley added a comment - This adds synchronization to writeToIndexFile on the 2xx branch. Since the TT is going away in trunk, there isn't a need to fix it there.
          Hide
          Owen O'Malley added a comment -

          Actually, this is in the task, so here is the relevant patch.

          Show
          Owen O'Malley added a comment - Actually, this is in the task, so here is the relevant patch.
          Hide
          Allen Wittenauer added a comment -

          +1

          Show
          Allen Wittenauer added a comment - +1
          Hide
          Devaraj Das added a comment -

          Looks good to me as well

          Show
          Devaraj Das added a comment - Looks good to me as well
          Hide
          Owen O'Malley added a comment -

          I just committed this.

          Show
          Owen O'Malley added a comment - I just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #780 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/780/)
          MAPREDUCE-2846. Fix missing synchronization in the task log management.
          (omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #780 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/780/ ) MAPREDUCE-2846 . Fix missing synchronization in the task log management. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324 Files : /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #855 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/855/)
          MAPREDUCE-2846. Fix missing synchronization in the task log management.
          (omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #855 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/855/ ) MAPREDUCE-2846 . Fix missing synchronization in the task log management. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324 Files : /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #777 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/777/)
          MAPREDUCE-2846. Fix missing synchronization in the task log management.
          (omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #777 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/777/ ) MAPREDUCE-2846 . Fix missing synchronization in the task log management. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324 Files : /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #787 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/787/)
          MAPREDUCE-2846. Fix missing synchronization in the task log management.
          (omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #787 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/787/ ) MAPREDUCE-2846 . Fix missing synchronization in the task log management. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324 Files : /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #765 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/765/)
          MAPREDUCE-2846. Fix missing synchronization in the task log management.
          (omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #765 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/765/ ) MAPREDUCE-2846 . Fix missing synchronization in the task log management. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324 Files : /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Hide
          Owen O'Malley added a comment -

          Hadoop 0.20.204.0 was just released.

          Show
          Owen O'Malley added a comment - Hadoop 0.20.204.0 was just released.
          Hide
          Devaraj K added a comment -

          I know it is too late to comment here but just want to inform.

          The trunk patch is not required because writeToIndexFile() which is private and calling from only one place i.e syncLogs() which is already synchronized method.

          Show
          Devaraj K added a comment - I know it is too late to comment here but just want to inform. The trunk patch is not required because writeToIndexFile() which is private and calling from only one place i.e syncLogs() which is already synchronized method.

            People

            • Assignee:
              Owen O'Malley
              Reporter:
              Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development