Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-916

Rewrite DFSOutputStream to use a single thread with NIO

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.22.0
    • None
    • hdfs-client
    • None

    Description

      The DFS write pipeline code has some really hairy multi-threaded synchronization. There have been a lot of bugs produced by this (HDFS-101, HDFS-793, HDFS-915, tens of others) since it's very hard to understand the message passing, lock sharing, and interruption properties. The reason for the multiple threads is to be able to simultaneously send and receive. If instead of using multiple threads, it used nonblocking IO, I think the whole thing would be a lot less error prone.

      I think we could do this in two halves: one half is the DFSOutputStream. The other half is BlockReceiver. I opened this JIRA first as I think it's simpler (only one TCP connection to deal with, rather than an up and downstream)

      Opinions? Am I crazy? I would like to see some agreement on the idea before I spend time writing code.

      Attachments

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. HDFS-223 Asynchronous IO Handling in Hadoop and HDFS

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. AVRO-405 Netty-based Java RPC server

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-281 Explore usage of the sendfile api via java.nio.channels.FileChannel.transfer{To|From} for i/o in datanodes

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-258 Update MapOutputServlet to use NIO channels

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-319 Use Grizzly for Fetching Map Output in Shuffle

          • Major - Major loss of function.
          • Resolved

          Wish - General wishlist item. HADOOP-3518 Want NIO.2 (JSR 203) file system provider for Hadoop FileSystem

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-14790 Implement a new DFSOutputStream for logging WAL only

          • Major - Major loss of function.
          • Closed

          Umbrella - An overarching type made of sub-tasks HBASE-2182 rpc/ipc refactor; x-version compatibility, nio, async, enveloping, sane timeouts, etc.

          • Major - Major loss of function.
          • Closed
          requires

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-914 Refactor DFSOutputStream and DFSInputStream out of DFSClient

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link Asynchronous DFS Output Proposal

          Activity

            People

              Unassigned Unassigned
              tlipcon Todd Lipcon
              Votes:
              1 Vote for this issue
              Watchers:
              30 Start watching this issue

              Dates

                Created:
                Updated: