Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-826

Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

    Details

    • Hadoop Flags:
      Reviewed
    • Tags:
      hbase

      Description

      HDFS does not replicate the last block of the file that is being currently written to by an application. Every datanode death in the write pipeline decreases the reliability of the last block of the currently-being-written block. This situation can be improved if the application can be notified of a datanode death in the write pipeline. Then, the application can decide what is the right course of action to be taken on this event.

      In our use-case, the application can close the file on the first datanode death, and start writing to a newly created file. This ensures that the reliability guarantee of a block is close to 3 at all time.

      One idea is to make DFSOutoutStream. write() throw an exception if the number of datanodes in the write pipeline fall below minimum.replication.factor that is set on the client (this is backward compatible).

      1. ReplicableHdfs.txt
        3 kB
        dhruba borthakur
      2. ReplicableHdfs2.txt
        5 kB
        dhruba borthakur
      3. ReplicableHdfs3.txt
        5 kB
        dhruba borthakur
      4. Replicable4.txt
        5 kB
        dhruba borthakur
      5. HDFS-826-0.20.patch
        7 kB
        Nicolas Spiegelberg
      6. HDFS-826-0.20-v2.patch
        5 kB
        stack
      7. HDFS-826.20-security.1.patch
        5 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development