Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5318

Support read-only and read-write paths to shared replicas



    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.4.0
    • namenode
    • None
    • Reviewed


      There are several use cases for using shared-storage for datanode block storage in an HDFS environment (storing cold blocks on a NAS device, Amazon S3, etc.).

      With shared-storage, there is a distinction between:

      1. a distinct physical copy of a block
      2. an access-path to that block via a datanode.

      A single 'replication count' metric cannot accurately capture both aspects. However, for most of the current uses of 'replication count' in the Namenode, the "number of physical copies" aspect seems to be the appropriate semantic.

      I propose altering the replication counting algorithm in the Namenode to accurately infer distinct physical copies in a shared storage environment. With HDFS-5115, a StorageID is a UUID. I propose associating some minor additional semantics to the StorageID - namely that multiple datanodes attaching to the same physical shared storage pool should report the same StorageID for that pool. A minor modification would be required in the DataNode to enable the generation of StorageID s to be pluggable behind the FsDatasetSpi interface.

      With those semantics in place, the number of physical copies of a block in a shared storage environment can be calculated as the number of distinct StorageID s associated with that block.

      Consider the following combinations for two (DataNode ID, Storage ID) pairs (DN_A, S_A) (DN_B, S_B) for a given block B:

      • DN_A != DN_B && S_A != S_B - different access paths to different physical replicas (i.e. the traditional HDFS case with local disks)
        • → Block B has ReplicationCount == 2
      • DN_A != DN_B && S_A == S_B - different access paths to the same physical replica (e.g. HDFS datanodes mounting the same NAS share)
        • → Block B has ReplicationCount == 1

      For example, if block B has the following location tuples:

      • DN_1, STORAGE_A
      • DN_2, STORAGE_A
      • DN_3, STORAGE_B
      • DN_4, STORAGE_B,

      the effect of this proposed change would be to calculate the replication factor in the namenode as 2 instead of 4.


        1. HDFS-5318.patch
          44 kB
          Eric Sirianni
        2. hdfs-5318.pdf
          391 kB
          Eric Sirianni
        3. HDFS-5318a-branch-2.patch
          27 kB
          Eric Sirianni
        4. HDFS-5318b-branch-2.patch
          29 kB
          Eric Sirianni
        5. HDFS-5318c-branch-2.patch
          36 kB
          Eric Sirianni
        6. HDFS-5318-trunk.patch
          36 kB
          Eric Sirianni
        7. HDFS-5318-trunkb.patch
          37 kB
          Eric Sirianni
        8. HDFS-5318-trunk-c.patch
          37 kB
          Eric Sirianni



            Unassigned Unassigned
            sirianni Eric Sirianni
            1 Vote for this issue
            16 Start watching this issue