Some other details:

a) we do not have security on
b) we are not using the LinuxTaskController

MR-2413 (disk failure handling) and MR-2415 (distribute userlogs onto multiple disks) were only implemented for LinuxTaskController but change some of the relevant code, might be worth looking into.

nods I'm mostly convinced it is a race condition in MR-2415. I haven't had enough time to start playing in the source to track it down more. I did talk to Owen about already, but thought it might be useful to at least get the JIRA filed to put more eyes on it since race conditions are usually pretty awful to track down.

Hi Allen,

Can you post how you are configuring mapred.local.dir values? We have not seen this problem in our cluster since we run with Linux task controller. But, Eli is right, we did change Default task controller to make it consistent. Giving more information will help us to understand better like how many disks you have, mapred.local.dir value etc. or even mapred-site.xml. I am asking this information to get an idea of how we can reproduce in our test cluster?

Some relevant properties:

<property>
<name>mapred.local.dir</name>
<value>/grid/a/mapred/local,/grid/b/mapred/local,/grid/c/mapred/local,/grid/d/mapred/local,/grid/e/mapred/local,/grid/f/mapred/local</value>
</property>

<property>
<name>hadoop.job.history.user.location</name>
<value>none</value>
<final>true</final>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/grid/a/mapred/tmp/hadoop-$

</value>
<description>A base for other temporary directories.</description>
<final>true</final>
</property>

The permissions on these dirs are 775. User and group match the user we run the tasktracker as. (So, with DefaultTaskController, this should work just fine.)

Some other questions I've been asked over IM:

• Nodes can show failures with one run, be perfectly clean the next, then show failures during a third run. Some nodes will throw failures during all three.
• This problem is reflected in both map tasks and reduce tasks.
• The dir permissions really are the same across all dirs and all nodes.
• I have not tried LTC because my test grid is not configured to support it yet.
• I've been testing the Apache releases with no custom patches other than including the LZO bits.
• The number of failures per run is wildly inconsistent.
• Running 203 on the same gear with the same config shows zero failures. So this is clearly a result of something added in 204.
• Yes, enough tasks have failed during certain runs that tasktrackers are getting blacklisted from the job.

I'm currently playing with a debug jar from Owen to try and gather more information. Part of the problem is that there clearly isn't enough information on why tasks are failing. The tasktracker logs throw the symlink error but see MAPREDUCE-2804. The child error stack trace:

java.lang.Throwable: Child Error
	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of -1.
	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

is equally unhelpful.

Offline, Allen gave me a stack trace:

java.io.FileNotFoundException: File /export/apps/hadoop/hadoop-0.20.204.0/logs/userlogs/job_201108100052_0008/attempt_201108100052_0008_r_000145_0/log.tmp does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:210)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:160)
at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:261)
at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:406)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:345)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:391)
at org.apache.hadoop.mapred.Child.main(Child.java:235)

Based on this, I discovered that there is a missing synchronization in writeToIndexFile. This seems to reduce the failures that Allen is seeing.

This adds synchronization to writeToIndexFile on the 2xx branch.

Since the TT is going away in trunk, there isn't a need to fix it there.

Actually, this is in the task, so here is the relevant patch.

+1

Looks good to me as well

I just committed this.

Integrated in Hadoop-Mapreduce-trunk-Commit #780 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/780/)
MAPREDUCE-2846. Fix missing synchronization in the task log management.
(omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
Files :

• /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java

Integrated in Hadoop-Hdfs-trunk-Commit #855 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/855/)
MAPREDUCE-2846. Fix missing synchronization in the task log management.
(omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
Files :

• /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java

Integrated in Hadoop-Common-trunk-Commit #777 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/777/)
MAPREDUCE-2846. Fix missing synchronization in the task log management.
(omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
Files :

• /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java

Integrated in Hadoop-Mapreduce-trunk #787 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/787/)
MAPREDUCE-2846. Fix missing synchronization in the task log management.
(omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
Files :

• /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java

Integrated in Hadoop-Hdfs-trunk #765 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/765/)
MAPREDUCE-2846. Fix missing synchronization in the task log management.
(omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161324
Files :

• /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java

Hadoop 0.20.204.0 was just released.

I know it is too late to comment here but just want to inform.

The trunk patch is not required because writeToIndexFile() which is private and calling from only one place i.e syncLogs() which is already synchronized method.