Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10285 Storage Policy Satisfier in HDFS
  3. HDFS-11965

[SPS]: Should give chance to satisfy the low redundant blocks before removing the xattr

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HDFS-10285
    • Fix Version/s: HDFS-10285, 3.2.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The test case is failing because all the required replicas are not moved in expected storage. This is happened because of delay in datanode registration after cluster restart.

      Scenario :
      1. Start cluster with 3 DataNodes.
      2. Create file and set storage policy to WARM.
      3. Restart the cluster.
      4. Now Namenode and two DataNodes started first and got registered with NameNode. (one datanode not yet registered)
      5. SPS scheduled block movement based on available DataNodes (It will move one replica in ARCHIVE based on policy).
      6. Block movement also success and Xattr removed from the file because this condition is true itemInfo.isAllBlockLocsAttemptedToSatisfy().

      if (itemInfo != null
                      && !itemInfo.isAllBlockLocsAttemptedToSatisfy()) {
                    blockStorageMovementNeeded
                        .add(storageMovementAttemptedResult.getTrackId());
                  ....................
                  ......................
                  } else {
                  ....................
                  ......................
                    this.sps.postBlkStorageMovementCleanup(
                        storageMovementAttemptedResult.getTrackId());
                  }
      

      7. Now third DN registered with namenode and its reported one more DISK replica. Now Namenode has two DISK and one ARCHIVE replica.

      In test case we have condition to check the number of DISK replica..

       DFSTestUtil.waitExpectedStorageType(testFileName, StorageType.DISK, 1, timeout, fs);

      This condition never became true and test case will be timed out.

        Attachments

        1. HDFS-11965-HDFS-10285.008.patch
          15 kB
          Surendra Singh Lilhore
        2. HDFS-11965-HDFS-10285.007.patch
          15 kB
          Surendra Singh Lilhore
        3. HDFS-11965-HDFS-10285.006.patch
          15 kB
          Surendra Singh Lilhore
        4. HDFS-11965-HDFS-10285.005.patch
          24 kB
          Surendra Singh Lilhore
        5. HDFS-11965-HDFS-10285.004.patch
          24 kB
          Surendra Singh Lilhore
        6. HDFS-11965-HDFS-10285.003.patch
          24 kB
          Surendra Singh Lilhore
        7. HDFS-11965-HDFS-10285.002.patch
          17 kB
          Surendra Singh Lilhore
        8. HDFS-11965-HDFS-10285.001.patch
          16 kB
          Surendra Singh Lilhore

          Activity

            People

            • Assignee:
              surendrasingh Surendra Singh Lilhore
              Reporter:
              surendrasingh Surendra Singh Lilhore
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: