Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8361

Choose SSD over DISK in block placement

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      BlockPlacementPolicyDefault chooses the StorageType by iterating the given StorageType EnumMap in its natural order (the order in which the enum constants are declared). So DISK will be chosen over SSD in One-SSD policy since DISK is declared before SSD as shown below. We should choose SSD first.

      public enum StorageType {
        DISK(false),
        SSD(false),
        ARCHIVE(false),
        RAM_DISK(true);
      
        ...
      }
      
      1. h8361_20150612.patch
        5 kB
        Tsz Wo Nicholas Sze
      2. h8361_20150508.patch
        2 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8361_20150508.patch: reorders the storage types.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8361_20150508.patch: reorders the storage types.
          Hide
          jpallas Joe Pallas added a comment -

          There's a typo in the test name.

          The test verifies the ordering of the enum entries, but it doesn't actually verify that the policy prefers SSD over DISK. A change to the implementation could break the behavior without failing the test.

          Eventually, storage types might be characterized by more than one dimension, so a single ordering may not serve the purposes of all placement policies.

          Show
          jpallas Joe Pallas added a comment - There's a typo in the test name. The test verifies the ordering of the enum entries, but it doesn't actually verify that the policy prefers SSD over DISK. A change to the implementation could break the behavior without failing the test. Eventually, storage types might be characterized by more than one dimension, so a single ordering may not serve the purposes of all placement policies.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Joe, thanks for looking at the patch and the comments. Will fix the typo and add a new test.

          > Eventually, storage types might be characterized by more than one dimension, so a single ordering may not serve the purposes of all placement policies.

          Let's deal with multiple dimensional characteristic when there is a need.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Joe, thanks for looking at the patch and the comments. Will fix the typo and add a new test. > Eventually, storage types might be characterized by more than one dimension, so a single ordering may not serve the purposes of all placement policies. Let's deal with multiple dimensional characteristic when there is a need.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8361_20150612.patch: addresses Joe's comments.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8361_20150612.patch: addresses Joe's comments.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 35s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 3m 22s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 5m 5s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 33s Tests failed in hadoop-common.
          +1 hdfs tests 160m 35s Tests passed in hadoop-hdfs.
              231m 2s  



          Reason Tests
          Failed unit tests hadoop.fs.shell.TestCount



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739390/h8361_20150612.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / eef7b50
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11336/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11336/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11336/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11336/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 3m 22s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 5m 5s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 33s Tests failed in hadoop-common. +1 hdfs tests 160m 35s Tests passed in hadoop-hdfs.     231m 2s   Reason Tests Failed unit tests hadoop.fs.shell.TestCount Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739390/h8361_20150612.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / eef7b50 hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11336/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11336/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11336/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11336/console This message was automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for working on this, Nicholas! It may be easier to follow if exposing the storage type selection logic in a more explicit way in BlockPlacementPolicyDefault. But the current fix also looks good to me. +1.

          Show
          jingzhao Jing Zhao added a comment - Thanks for working on this, Nicholas! It may be easier to follow if exposing the storage type selection logic in a more explicit way in BlockPlacementPolicyDefault. But the current fix also looks good to me. +1.
          Hide
          jpallas Joe Pallas added a comment -

          Yes, thanks.

          Show
          jpallas Joe Pallas added a comment - Yes, thanks.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8022 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8022/)
          HDFS-8361. Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8022 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8022/ ) HDFS-8361 . Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Joe and Jing for the reviewing.

          I have committed this.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Joe and Jing for the reviewing. I have committed this.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #960 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/960/)
          HDFS-8361. Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #960 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/960/ ) HDFS-8361 . Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/230/)
          HDFS-8361. Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/230/ ) HDFS-8361 . Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2158/)
          HDFS-8361. Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2158/ ) HDFS-8361 . Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/219/)
          HDFS-8361. Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/219/ ) HDFS-8361 . Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/228/)
          HDFS-8361. Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/228/ ) HDFS-8361 . Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2176/)
          HDFS-8361. Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2176/ ) HDFS-8361 . Choose SSD over DISK in block placement. (szetszwo: rev 175e6d120fc34710048c2e9512dd270410172803) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

            People

            • Assignee:
              szetszwo Tsz Wo Nicholas Sze
              Reporter:
              szetszwo Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development