Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2846

a small % of all tasks fail with DefaultTaskController

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed a race condition in writing the log index file that caused tasks to 'fail'.

      Description

      After upgrading our test 0.20.203 grid to 0.20.204-rc2, we ran terasort to verify operation. While the job completed successfully, approx 10% of the tasks failed with task runner execution errors and the inability to create symlinks for attempt logs.

      1. sync.patch
        0.9 kB
        Owen O'Malley
      2. sync-trunk.patch
        1 kB
        Owen O'Malley

        Issue Links

          Activity

          Allen Wittenauer created issue -
          Allen Wittenauer made changes -
          Field Original Value New Value
          Link This issue relates to MAPREDUCE-2415 [ MAPREDUCE-2415 ]
          Allen Wittenauer made changes -
          Description After upgrading our test 0.20.203 grid to 0.20.204, we ran terasort to verify operation. While the job completed successfully, approx 10% of the tasks failed with task runner execution errors and the inability to create symlinks for attempt logs. After upgrading our test 0.20.203 grid to 0.20.204-rc2, we ran terasort to verify operation. While the job completed successfully, approx 10% of the tasks failed with task runner execution errors and the inability to create symlinks for attempt logs.
          Hide
          Allen Wittenauer added a comment -

          Some other details:

          a) we do not have security on
          b) we are not using the LinuxTaskController

          Show
          Allen Wittenauer added a comment - Some other details: a) we do not have security on b) we are not using the LinuxTaskController
          Allen Wittenauer made changes -
          Link This issue is related to MAPREDUCE-2804 [ MAPREDUCE-2804 ]
          Hide
          Eli Collins added a comment -

          MR-2413 (disk failure handling) and MR-2415 (distribute userlogs onto multiple disks) were only implemented for LinuxTaskController but change some of the relevant code, might be worth looking into.

          Show
          Eli Collins added a comment - MR-2413 (disk failure handling) and MR-2415 (distribute userlogs onto multiple disks) were only implemented for LinuxTaskController but change some of the relevant code, might be worth looking into.
          Hide
          Allen Wittenauer added a comment -

          nods I'm mostly convinced it is a race condition in MR-2415. I haven't had enough time to start playing in the source to track it down more. I did talk to Owen about already, but thought it might be useful to at least get the JIRA filed to put more eyes on it since race conditions are usually pretty awful to track down.

          Show
          Allen Wittenauer added a comment - nods I'm mostly convinced it is a race condition in MR-2415. I haven't had enough time to start playing in the source to track it down more. I did talk to Owen about already, but thought it might be useful to at least get the JIRA filed to put more eyes on it since race conditions are usually pretty awful to track down.
          Hide
          Bharath Mundlapudi added a comment -

          Hi Allen,

          Can you post how you are configuring mapred.local.dir values? We have not seen this problem in our cluster since we run with Linux task controller. But, Eli is right, we did change Default task controller to make it consistent. Giving more information will help us to understand better like how many disks you have, mapred.local.dir value etc. or even mapred-site.xml. I am asking this information to get an idea of how we can reproduce in our test cluster?

          Show
          Bharath Mundlapudi added a comment - Hi Allen, Can you post how you are configuring mapred.local.dir values? We have not seen this problem in our cluster since we run with Linux task controller. But, Eli is right, we did change Default task controller to make it consistent. Giving more information will help us to understand better like how many disks you have, mapred.local.dir value etc. or even mapred-site.xml. I am asking this information to get an idea of how we can reproduce in our test cluster?
          Hide
          Allen Wittenauer added a comment -

          Some relevant properties:

          <property>
          <name>mapred.local.dir</name>
          <value>/grid/a/mapred/local,/grid/b/mapred/local,/grid/c/mapred/local,/grid/d/mapred/local,/grid/e/mapred/local,/grid/f/mapred/local</value>
          </property>

          <property>
          <name>hadoop.job.history.user.location</name>
          <value>none</value>
          <final>true</final>
          </property>

          <property>
          <name>hadoop.tmp.dir</name>
          <value>/grid/a/mapred/tmp/hadoop-$

          {user.name}

          </value>
          <description>A base for other temporary directories.</description>
          <final>true</final>
          </property>

          The permissions on these dirs are 775. User and group match the user we run the tasktracker as. (So, with DefaultTaskController, this should work just fine.)

          Some other questions I've been asked over IM:

          • Nodes can show failures with one run, be perfectly clean the next, then show failures during a third run. Some nodes will throw failures during all three.
          • This problem is reflected in both map tasks and reduce tasks.
          • The dir permissions really are the same across all dirs and all nodes.
          • I have not tried LTC because my test grid is not configured to support it yet.
          • I've been testing the Apache releases with no custom patches other than including the LZO bits.
          • The number of failures per run is wildly inconsistent.
          • Running 203 on the same gear with the same config shows zero failures. So this is clearly a result of something added in 204.
          • Yes, enough tasks have failed during certain runs that tasktrackers are getting blacklisted from the job.

          I'm currently playing with a debug jar from Owen to try and gather more information. Part of the problem is that there clearly isn't enough information on why tasks are failing. The tasktracker logs throw the symlink error but see MAPREDUCE-2804. The child error stack trace:

          java.lang.Throwable: Child Error
          	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
          Caused by: java.io.IOException: Task process exit with nonzero status of -1.
          	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
          

          is equally unhelpful.

          Show
          Allen Wittenauer added a comment - Some relevant properties: <property> <name>mapred.local.dir</name> <value>/grid/a/mapred/local,/grid/b/mapred/local,/grid/c/mapred/local,/grid/d/mapred/local,/grid/e/mapred/local,/grid/f/mapred/local</value> </property> <property> <name>hadoop.job.history.user.location</name> <value>none</value> <final>true</final> </property> <property> <name>hadoop.tmp.dir</name> <value>/grid/a/mapred/tmp/hadoop-$ {user.name} </value> <description>A base for other temporary directories.</description> <final>true</final> </property> The permissions on these dirs are 775. User and group match the user we run the tasktracker as. (So, with DefaultTaskController, this should work just fine.) Some other questions I've been asked over IM: Nodes can show failures with one run, be perfectly clean the next, then show failures during a third run. Some nodes will throw failures during all three. This problem is reflected in both map tasks and reduce tasks. The dir permissions really are the same across all dirs and all nodes. I have not tried LTC because my test grid is not configured to support it yet. I've been testing the Apache releases with no custom patches other than including the LZO bits. The number of failures per run is wildly inconsistent. Running 203 on the same gear with the same config shows zero failures. So this is clearly a result of something added in 204. Yes, enough tasks have failed during certain runs that tasktrackers are getting blacklisted from the job. I'm currently playing with a debug jar from Owen to try and gather more information. Part of the problem is that there clearly isn't enough information on why tasks are failing. The tasktracker logs throw the symlink error but see MAPREDUCE-2804 . The child error stack trace: java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of -1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) is equally unhelpful.
          Allen Wittenauer made changes -
          Summary approx 10% of all tasks fail with DefaultTaskController a small % of all tasks fail with DefaultTaskController
          Hide
          Owen O'Malley added a comment -

          Offline, Allen gave me a stack trace:

          java.io.FileNotFoundException: File /export/apps/hadoop/hadoop-0.20.204.0/logs/userlogs/job_201108100052_0008/attempt_201108100052_0008_r_000145_0/log.tmp does not exist.
          at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
          at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:210)
          at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:160)
          at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:261)
          at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:406)
          at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:345)
          at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:391)
          at org.apache.hadoop.mapred.Child.main(Child.java:235)

          Based on this, I discovered that there is a missing synchronization in writeToIndexFile. This seems to reduce the failures that Allen is seeing.

          Show
          Owen O'Malley added a comment - Offline, Allen gave me a stack trace: java.io.FileNotFoundException: File /export/apps/hadoop/hadoop-0.20.204.0/logs/userlogs/job_201108100052_0008/attempt_201108100052_0008_r_000145_0/log.tmp does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:210) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:160) at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:261) at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:406) at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:345) at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:391) at org.apache.hadoop.mapred.Child.main(Child.java:235) Based on this, I discovered that there is a missing synchronization in writeToIndexFile. This seems to reduce the failures that Allen is seeing.
          Owen O'Malley made changes -
          Assignee Owen O'Malley [ owen.omalley ]
          Hide
          Owen O'Malley added a comment -

          This adds synchronization to writeToIndexFile on the 2xx branch.

          Since the TT is going away in trunk, there isn't a need to fix it there.

          Show
          Owen O'Malley added a comment - This adds synchronization to writeToIndexFile on the 2xx branch. Since the TT is going away in trunk, there isn't a need to fix it there.
          Owen O'Malley made changes -
          Attachment sync.patch [ 12490979 ]
          Hide
          Owen O'Malley added a comment -

          Actually, this is in the task, so here is the relevant patch.

          Show
          Owen O'Malley added a comment - Actually, this is in the task, so here is the relevant patch.
          Owen O'Malley made changes -
          Attachment sync-trunk.patch [ 12490980 ]
          Owen O'Malley made changes -
          Release Note Fixed a race condition in writing the log index file that caused tasks to fail.
          Hide
          Allen Wittenauer added a comment -

          +1

          Show
          Allen Wittenauer added a comment - +1
          Hide
          Devaraj Das added a comment -

          Looks good to me as well

          Show
          Devaraj Das added a comment - Looks good to me as well
          Owen O'Malley made changes -
          Release Note Fixed a race condition in writing the log index file that caused tasks to fail. Fixed a race condition in writing the log index file that caused tasks to 'fail'.
          Hide
          Owen O'Malley added a comment -

          I just committed this.

          Show
          Owen O'Malley added a comment - I just committed this.
          Owen O'Malley made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.20.204.0 [ 12316318 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #780 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/780/)
          MAPREDUCE-2846. Fix missing synchronization in the task log management.
          (omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #780 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/780/ ) MAPREDUCE-2846 . Fix missing synchronization in the task log management. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324 Files : /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #855 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/855/)
          MAPREDUCE-2846. Fix missing synchronization in the task log management.
          (omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #855 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/855/ ) MAPREDUCE-2846 . Fix missing synchronization in the task log management. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324 Files : /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #777 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/777/)
          MAPREDUCE-2846. Fix missing synchronization in the task log management.
          (omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #777 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/777/ ) MAPREDUCE-2846 . Fix missing synchronization in the task log management. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324 Files : /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #787 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/787/)
          MAPREDUCE-2846. Fix missing synchronization in the task log management.
          (omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #787 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/787/ ) MAPREDUCE-2846 . Fix missing synchronization in the task log management. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324 Files : /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #765 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/765/)
          MAPREDUCE-2846. Fix missing synchronization in the task log management.
          (omalley)

          omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #765 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/765/ ) MAPREDUCE-2846 . Fix missing synchronization in the task log management. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324 Files : /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
          Hide
          Owen O'Malley added a comment -

          Hadoop 0.20.204.0 was just released.

          Show
          Owen O'Malley added a comment - Hadoop 0.20.204.0 was just released.
          Owen O'Malley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Devaraj K added a comment -

          I know it is too late to comment here but just want to inform.

          The trunk patch is not required because writeToIndexFile() which is private and calling from only one place i.e syncLogs() which is already synchronized method.

          Show
          Devaraj K added a comment - I know it is too late to comment here but just want to inform. The trunk patch is not required because writeToIndexFile() which is private and calling from only one place i.e syncLogs() which is already synchronized method.
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          8d 7h 19m 1 Owen O'Malley 25/Aug/11 00:57
          Resolved Resolved Closed Closed
          8d 22h 15m 1 Owen O'Malley 02/Sep/11 23:13

            People

            • Assignee:
              Owen O'Malley
              Reporter:
              Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development