Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3702

Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client

    XMLWordPrintableJSON

Details

    • Hide
      This patch will attempt to allocate all replicas to remote DataNodes, by adding local DataNode to the excluded DataNodes. If no sufficient replicas can be obtained, it will fall back to default block placement policy, which writes one replica to local DataNode.
      Show
      This patch will attempt to allocate all replicas to remote DataNodes, by adding local DataNode to the excluded DataNodes. If no sufficient replicas can be obtained, it will fall back to default block placement policy, which writes one replica to local DataNode.

    Description

      This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are not read when there are no failures.

      Taking HBase as an example, these files will be read only if the process that wrote them (the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only 2 of them are really useful.

      Attachments

        1. HDFS-3702_Design.pdf
          63 kB
          Lei (Eddy) Xu
        2. HDFS-3702.000.patch
          20 kB
          Lei (Eddy) Xu
        3. HDFS-3702.001.patch
          19 kB
          Lei (Eddy) Xu
        4. HDFS-3702.002.patch
          25 kB
          Lei (Eddy) Xu
        5. HDFS-3702.003.patch
          23 kB
          Lei (Eddy) Xu
        6. HDFS-3702.004.patch
          75 kB
          Lei (Eddy) Xu
        7. HDFS-3702.005.patch
          75 kB
          Lei (Eddy) Xu
        8. HDFS-3702.006.patch
          75 kB
          Lei (Eddy) Xu
        9. HDFS-3702.007.patch
          74 kB
          Lei (Eddy) Xu
        10. HDFS-3702.008.patch
          75 kB
          Lei (Eddy) Xu
        11. HDFS-3702.009.patch
          75 kB
          Lei (Eddy) Xu
        12. HDFS-3702.010.patch
          78 kB
          Lei (Eddy) Xu
        13. HDFS-3702.011.patch
          79 kB
          Lei (Eddy) Xu
        14. HDFS-3702.012.patch
          79 kB
          Lei (Eddy) Xu

        Issue Links

          Activity

            People

              eddyxu Lei (Eddy) Xu
              nkeywal Nicolas Liochon
              Votes:
              0 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: