[HADOOP-984] Exception while retrieving a map output index file is not reported to JobTracker

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.11.0
• Fix Version/s: 0.11.1
• Assignee: Arun Murthy
• Reporter: Nigel Daley
• Created: 06/Feb/07 19:41
• Updated: 08/Jul/09 16:52
• Resolved: 07/Feb/07 18:36

Description

The index file of a map output reported a checksum error and was moved to "bad_file" dir. Reduces, however, continued to attempt retrieval of the file forever. It seem the checksum exception did NOT result in the map's TaskTracker reporting to the JobTracker that the map output was lost. This was likely introduced by HADOOP-331.

TaskTracker log:

2007-02-06 09:06:19,200 WARN org.apache.hadoop.dfs.DistributedFileSystem: Moving bad file /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index to /foo/bad_files/file.out.index.-48633003
2007-02-06 09:06:19,203 INFO org.apache.hadoop.fs.DataInputStream: Found checksum error: org.apache.hadoop.fs.ChecksumException: Checksum error: /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index at 363792
at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:167)
at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:125)
at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:218)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:281)
at org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:268)
at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:331)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1533)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)