Hadoop Common
  1. Hadoop Common
  2. HADOOP-1179

task Tracker should be restarted if its jetty http server cannot serve get-map-output files

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.3
    • Component/s: None
    • Labels:
      None

      Description

      Due to some errors (mem leak?), the jetty http server throws outOfMemory exception when serving get-map-output requests:

      2007-03-28 20:42:39,608 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
      /mapOutput?map=task_0334_m_013127_0&reduce=591
      2007-03-28 20:46:42,788 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
      /mapOutput?map=task_0334_m_013127_0&reduce=591
      2007-03-28 20:49:38,064 WARN org.mortbay.jetty.servlet.ServletHandler: Error for
      java.lang.OutOfMemoryError
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:199)
      at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(R
      awLocalFileSystem.java:119)
      at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
      Stream.java:41)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
      at java.io.DataInputStream.read(DataInputStream.java:132)
      at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(Che
      cksumFileSystem.java:182)
      at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumF
      ileSystem.java:167)
      at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInput
      Stream.java:41)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
      at java.io.DataInputStream.readFully(DataInputStream.java:178)
      at java.io.DataInputStream.readLong(DataInputStream.java:399)
      at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTrack
      er.java:1643)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
      at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427
      )
      at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicati
      onHandler.java:475)
      at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:5
      67)
      at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
      at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplication
      Context.java:635)
      at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
      at org.mortbay.http.HttpServer.service(HttpServer.java:954)
      at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
      at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
      at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
      at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:
      244)
      at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
      at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)

      In this case, the task tracker cannot send out the map outut files on that machine, rendering it useless.
      Moreover, all the reduces depending on those map output files are just stuck there.
      If the task tracker reports fail to the job tracker, the map/reduce job can recover.
      If the task tracker restarted, it can continue to join the cluster as a new mamber.

      1. 1179.patch
        3 kB
        Devaraj Das

        Activity

        Hide
        Hadoop QA added a comment -
        Show
        Hadoop QA added a comment - Integrated in Hadoop-Nightly #48 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/48/ )
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Devaraj!

        Show
        Tom White added a comment - I've just committed this. Thanks Devaraj!
        Hide
        Owen O'Malley added a comment -

        +1

        Show
        Owen O'Malley added a comment - +1
        Hide
        Koji Noguchi added a comment -

        For all the nodes with org.mortbay.jetty.servlet.ServletHandler: java.lang.OutOfMemoryError, they each had one instance of OutOfMemoryError from inmemory file.

        2007-03-28 16:30:58,204 WARN org.apache.hadoop.mapred.TaskRunner: task_0334_r_000606_0 Intermediate Merge of the inmemory files threw an exception: java.lang.OutOfMemoryError
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:260)
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:166)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:38)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.write(ChecksumFileSystem.java:391)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:38)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.io.SequenceFile$CompressedBytes.writeCompressedBytes(SequenceFile.java:492)
        at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.appendRaw(SequenceFile.java:903)
        at org.apache.hadoop.io.SequenceFile$Sorter.writeFile(SequenceFile.java:2227)
        at org.apache.hadoop.mapred.ReduceTaskRunner$InMemFSMergeThread.run(ReduceTaskRunner.java:838)

        Show
        Koji Noguchi added a comment - For all the nodes with org.mortbay.jetty.servlet.ServletHandler: java.lang.OutOfMemoryError, they each had one instance of OutOfMemoryError from inmemory file. 2007-03-28 16:30:58,204 WARN org.apache.hadoop.mapred.TaskRunner: task_0334_r_000606_0 Intermediate Merge of the inmemory files threw an exception: java.lang.OutOfMemoryError at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:260) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:166) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:38) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.write(ChecksumFileSystem.java:391) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:38) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.io.SequenceFile$CompressedBytes.writeCompressedBytes(SequenceFile.java:492) at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.appendRaw(SequenceFile.java:903) at org.apache.hadoop.io.SequenceFile$Sorter.writeFile(SequenceFile.java:2227) at org.apache.hadoop.mapred.ReduceTaskRunner$InMemFSMergeThread.run(ReduceTaskRunner.java:838)
        Hide
        Doug Cutting added a comment -

        Note that running out of file handles can cause OutOfMemoryExceptions, even when there's lots of memory left. So this could just be a file-handle leak. It'd be great to log the heap size somehow (e.g. simply by turning on verbosegc, which has little performance impact) so we had an idea of whether this is really a memory issue or a file handle issue.

        Show
        Doug Cutting added a comment - Note that running out of file handles can cause OutOfMemoryExceptions, even when there's lots of memory left. So this could just be a file-handle leak. It'd be great to log the heap size somehow (e.g. simply by turning on verbosegc, which has little performance impact) so we had an idea of whether this is really a memory issue or a file handle issue.
        Hide
        Runping Qi added a comment -

        This issue is related to HADOOP-1158 but not a dup.
        Two things need to be investigated/fixed: why out of memory in Jetty (or in TT). Early comments and the attached patch may address that partially. However, I am not sure that accounts all.

        The second thing is proper handling of outOfMemory. Normally, when this happens, it may be desirable to kill the TT and hope to be restarted.

        Show
        Runping Qi added a comment - This issue is related to HADOOP-1158 but not a dup. Two things need to be investigated/fixed: why out of memory in Jetty (or in TT). Early comments and the attached patch may address that partially. However, I am not sure that accounts all. The second thing is proper handling of outOfMemory. Normally, when this happens, it may be desirable to kill the TT and hope to be restarted.
        Hide
        Devaraj Das added a comment -

        I have attached a patch containing the part to do with closing index file as soon as the read is done. Regarding the crc thing, I think what needs to change is the method ChecksumFileSystem.getSumBufferSize, but was not sure whether it would affect the other parts of the code. So didn't touch that part.

        Show
        Devaraj Das added a comment - I have attached a patch containing the part to do with closing index file as soon as the read is done. Regarding the crc thing, I think what needs to change is the method ChecksumFileSystem.getSumBufferSize, but was not sure whether it would affect the other parts of the code. So didn't touch that part.
        Hide
        Devaraj Das added a comment -

        Yeah Owen, i totally missed the open-file issues.

        Show
        Devaraj Das added a comment - Yeah Owen, i totally missed the open-file issues.
        Hide
        Owen O'Malley added a comment -

        Actually, it isn't really a duplicate of HADOOP-1158. Part of what is going on is that the MapOutputServlet is using a lot of memory. In particular, each thread is using:
        1. A file to read the index file
        2. A file to read the index file crcs
        3. A file to read the map outputs
        4. A file to read the map output crcs
        All 4 files use the io.file.buffer.size, which we usually set to 64k. Since the servlet is lazy about closing them, all 4 files stay open during the entire call. Since we have ~40 threads, that is 10MB of file buffers in the TaskTracker's JVM.

        A couple of steps would make this much better:
        1. Use smaller buffers for reading crc files (io.file.buffer.size * sizeof(crc) / sizeof(crc block) ?)
        2. Close the index file before opening the data files

        Show
        Owen O'Malley added a comment - Actually, it isn't really a duplicate of HADOOP-1158 . Part of what is going on is that the MapOutputServlet is using a lot of memory. In particular, each thread is using: 1. A file to read the index file 2. A file to read the index file crcs 3. A file to read the map outputs 4. A file to read the map output crcs All 4 files use the io.file.buffer.size, which we usually set to 64k. Since the servlet is lazy about closing them, all 4 files stay open during the entire call. Since we have ~40 threads, that is 10MB of file buffers in the TaskTracker's JVM. A couple of steps would make this much better: 1. Use smaller buffers for reading crc files (io.file.buffer.size * sizeof(crc) / sizeof(crc block) ?) 2. Close the index file before opening the data files
        Hide
        Devaraj Das added a comment -

        This is a duplicate of HADOOP-1158

        Show
        Devaraj Das added a comment - This is a duplicate of HADOOP-1158
        Hide
        Devaraj Das added a comment -

        This is related to HADOOP-1158

        Show
        Devaraj Das added a comment - This is related to HADOOP-1158

          People

          • Assignee:
            Devaraj Das
            Reporter:
            Runping Qi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development