Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-258

Update MapOutputServlet to use NIO channels

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The TaskTracker can serve the map output segments using RandomAccessFileBuffer, added in JETTY-748.

        Issue Links

          Activity

          Hide
          Chris Douglas added a comment -

          The checksum validation needs to be moved to the reduce before we can use FileChannel::transferTo.

          Show
          Chris Douglas added a comment - The checksum validation needs to be moved to the reduce before we can use FileChannel::transferTo.
          Hide
          David Yu added a comment -

          Attaching a sample servlet that demonstrates serving portions of a file using RandomAccessFileBuffer

          Show
          David Yu added a comment - Attaching a sample servlet that demonstrates serving portions of a file using RandomAccessFileBuffer
          Hide
          Chris Douglas added a comment -

          Thanks, David; the example servlet you attached was exceedingly helpful. I applied it to 6.1.14.1 and am trying to work it into our use case.

          There are a few issues to resolve. Among them, the Buffer interface assumes ints, where our use case calls for longs. We can't work around this one, unfortunately. I'm trying to add an HttpContent type that wraps RandomAccessFileBuffer (as in David's example), but letting RandomAccessFileBuffer simply implement Buffer (rather than extend AbstractBuffer), and use longs internally.

          Unfortunately, simply letting this use ints doesn't work in testing, either. There's a bug that shows up on Linux, where EAGAIN in sendfile isn't handled correctly. The bug fix, like others in transferTo, may not be fixed in 1.6 (while the openjdk source seems to have resolved the specific issue I'm hitting in in solaris/native/sun/nio/ch/FileChannelImpl.c in the 6-b05 "bundle", I couldn't find a mention of this issue or a transferTo fix in any of the Java SE 6 release notes). It looks like most are fixed in JDK7. Even if we resort to blocking I/O- which would require more surgery to Jetty- it doesn't look like we can take meaningful advantage of this feature right now. I'll spend some more time with it, but this looks like it may be a dead end.

          Given 6253145 (also fixed in JDK7), we'll need a workaround for large files anyway.

          Show
          Chris Douglas added a comment - Thanks, David; the example servlet you attached was exceedingly helpful. I applied it to 6.1.14.1 and am trying to work it into our use case. There are a few issues to resolve. Among them, the Buffer interface assumes ints, where our use case calls for longs. We can't work around this one, unfortunately. I'm trying to add an HttpContent type that wraps RandomAccessFileBuffer (as in David's example), but letting RandomAccessFileBuffer simply implement Buffer (rather than extend AbstractBuffer), and use longs internally. Unfortunately, simply letting this use ints doesn't work in testing, either. There's a bug that shows up on Linux, where EAGAIN in sendfile isn't handled correctly. The bug fix, like others in transferTo, may not be fixed in 1.6 (while the openjdk source seems to have resolved the specific issue I'm hitting in in solaris/native/sun/nio/ch/FileChannelImpl.c in the 6-b05 "bundle", I couldn't find a mention of this issue or a transferTo fix in any of the Java SE 6 release notes). It looks like most are fixed in JDK7. Even if we resort to blocking I/O- which would require more surgery to Jetty- it doesn't look like we can take meaningful advantage of this feature right now. I'll spend some more time with it, but this looks like it may be a dead end. Given 6253145 (also fixed in JDK7), we'll need a workaround for large files anyway.
          Hide
          Greg Wilkins added a comment -

          The Buffer mechanism of jetty would need major surgery to convert to longs from ints. So this is not likely to happen.
          So we would need to come up with some other to send such large content.

          Please comment on http://jira.codehaus.org/browse/JETTY-748 when you are interested in looking at this again and we
          can discuss the possibilities.

          Show
          Greg Wilkins added a comment - The Buffer mechanism of jetty would need major surgery to convert to longs from ints. So this is not likely to happen. So we would need to come up with some other to send such large content. Please comment on http://jira.codehaus.org/browse/JETTY-748 when you are interested in looking at this again and we can discuss the possibilities.
          Hide
          Chris Douglas added a comment -

          Thanks for updating us, Greg. Per my last comment, this issue seems dead for now, at least as long as we keep the transfer in the servlet. Our measurements don't suggest that we're bottlenecked here at the moment, but we'll be sure to check back when we revisit this. Thanks for your help.

          Show
          Chris Douglas added a comment - Thanks for updating us, Greg. Per my last comment, this issue seems dead for now, at least as long as we keep the transfer in the servlet. Our measurements don't suggest that we're bottlenecked here at the moment, but we'll be sure to check back when we revisit this. Thanks for your help.
          Hide
          Allen Wittenauer added a comment -

          He said dead, not me.

          Show
          Allen Wittenauer added a comment - He said dead, not me.

            People

            • Assignee:
              Chris Douglas
              Reporter:
              Chris Douglas
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development