Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10285 Storage Policy Satisfier in HDFS
  3. HDFS-11248

[SPS]: Handle partial block location movements



    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • HDFS-10285
    • HDFS-10285, 3.2.0
    • datanode, namenode
    • None
    • Reviewed


      This jira is to handle partial block location movements due to unavailability of target nodes for the matching storage type.

      For example, We have only A(disk,archive), B(disk) and C(disk,archive) are live nodes with A & C have archive storage type. Say, we have a block with locations A(disk), B(disk), C(disk). Again assume, user changed the storage policy to COLD. Now, SPS internally starts preparing the src-target pairing like, src=> (A, B, C) and target=> (A, C) and sends BLOCK_STORAGE_MOVEMENT to the coordinator. SPS is skipping B as it doesn't have archive media to indicate that it should do retries to satisfy all block locations after some time. On receiving the movement command, coordinator will pair the src-target node to schedule actual physical movements like, movetask=> (A, A), (B, C). Here ideally it should do (C, C) instead of (B, C) but mistakenly choosing the source C and creates problem.

      IMHO, the implicit assumptions of retry needed is creating confusions and leads to coding mistakes. One idea to fix this problem is to create a new flag retryNeeded flag to make it more readable. With this, SPS will prepare only the matching pair and dummy source slots will be avoided like, src=> (A, C) and target=> (A, C) and mark retryNeeded=true to convey the message that this trackId has only partial blocks movements.


        1. HDFS-11248-HDFS-10285-05.patch
          41 kB
          Rakesh Radhakrishnan
        2. HDFS-11248-HDFS-10285-04.patch
          40 kB
          Rakesh Radhakrishnan
        3. HDFS-11248-HDFS-10285-03.patch
          47 kB
          Rakesh Radhakrishnan
        4. HDFS-11248-HDFS-10285-02.patch
          46 kB
          Rakesh Radhakrishnan
        5. HDFS-11248-HDFS-10285-01.patch
          35 kB
          Rakesh Radhakrishnan
        6. HDFS-11248-HDFS-10285-00.patch
          40 kB
          Rakesh Radhakrishnan

        Issue Links



              rakeshr Rakesh Radhakrishnan
              rakeshr Rakesh Radhakrishnan
              0 Vote for this issue
              3 Start watching this issue