Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7770

Need document for storage type label of data node storage locations under dfs.datanode.data.dir

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: documentation
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-2832 enables support for heterogeneous storages in HDFS, which allows DN as a collection of storages with different types. However, I can't find document on how to label different storage types from the following two documents. I found the information from the design spec. It will be good we document this for admins and users to use the related Archival storage and storage policy features.

      http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html

      http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

      This JIRA is opened to add document for the new storage type labels.

      1. Add an example under ArchivalStorage.html#Configuration section:

        <property>
          <name>dfs.datanode.data.dir</name>
          <value>[DISK]file:///hddata/dn/disk0,  [SSD]file:///hddata/dn/ssd0,[ARCHIVE]file:///hddata/dn/archive0</value>
        </property>
      

      2. Add a short description of [DISK/SSD/ARCHIVE/RAM_DISK] options in hdfs-default.xml#dfs.data.dir and document DISK as storage type if no storage type is labeled in the data node storage location configuration.

      1. HDFS-7770.00.patch
        2 kB
        Xiaoyu Yao
      2. HDFS-7700.01.patch
        8 kB
        Xiaoyu Yao
      3. HDFS-7770.02.patch
        2 kB
        Xiaoyu Yao

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2129 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2129/)
          HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2129 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2129/ ) HDFS-7770 . Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/180/)
          HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/180/ ) HDFS-7770 . Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #913 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/913/)
          HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed)

          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #913 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/913/ ) HDFS-7770 . Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #170 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/170/)
          HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #170 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/170/ ) HDFS-7770 . Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #179 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/179/)
          HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #179 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/179/ ) HDFS-7770 . Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2111 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2111/)
          HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed)

          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2111 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2111/ ) HDFS-7770 . Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7705 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7705/)
          HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7705 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7705/ ) HDFS-7770 . Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao. (aajisaka: rev de9404f02f36bf9a1100c67f41db907d494bb9ed) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk, branch-2, and branch-2.7. Thanks Xiaoyu Yao for the contribution.

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, and branch-2.7. Thanks Xiaoyu Yao for the contribution.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Xiaoyu Yao for updating the patch. LGTM, +1.

          I think we can address that in a separate JIRA.

          Agree, let's create a jira for this.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Xiaoyu Yao for updating the patch. LGTM, +1. I think we can address that in a separate JIRA. Agree, let's create a jira for this.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 32s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 32s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 54s Site still builds.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 native 3m 12s Pre-build of native portion
          -1 hdfs tests 163m 21s Tests failed in hadoop-hdfs.
              206m 31s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728926/HDFS-7770.02.patch
          Optional Tests javadoc javac unit site
          git revision trunk / 5190923
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10437/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10437/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10437/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 site 2m 54s Site still builds. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 163m 21s Tests failed in hadoop-hdfs.     206m 31s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728926/HDFS-7770.02.patch Optional Tests javadoc javac unit site git revision trunk / 5190923 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10437/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10437/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10437/console This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Patch v01 applied on my local machine without problem. Rebase the patch anyway to get a clean Jenkins run.

          Show
          xyao Xiaoyu Yao added a comment - Patch v01 applied on my local machine without problem. Rebase the patch anyway to get a clean Jenkins run.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728118/HDFS-7700.01.patch
          Optional Tests site
          git revision trunk / 78c6b46
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10386/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728118/HDFS-7700.01.patch Optional Tests site git revision trunk / 78c6b46 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10386/console This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Akira Ajisaka for reviewing and providing feedback. I've updated the patch to address 1.

          2. I'm thinking it's better to document how to tag a storage type explicitly.

          Agree and I think we can address that in a separate JIRA.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Akira Ajisaka for reviewing and providing feedback. I've updated the patch to address 1. 2. I'm thinking it's better to document how to tag a storage type explicitly. Agree and I think we can address that in a separate JIRA.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Xiaoyu Yao for the patch. Two comments:

          +    4.  A datanode storage location /grid/dn/ram0 on ARCHIVE should be configured with `[RAM_DISK]file:///grid/dn/ram0`
          

          1. ARCHIVE should be RAM_DISK.

          +    The default storage type of a datanode storage location will be DISK if it does not have a storage type tagged explicitly.
          

          2. I'm thinking it's better to document how to tag a storage type explicitly.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Xiaoyu Yao for the patch. Two comments: + 4. A datanode storage location /grid/dn/ram0 on ARCHIVE should be configured with `[RAM_DISK]file: ///grid/dn/ram0` 1. ARCHIVE should be RAM_DISK. + The default storage type of a datanode storage location will be DISK if it does not have a storage type tagged explicitly. 2. I'm thinking it's better to document how to tag a storage type explicitly.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12699828/HDFS-7770.00.patch
          against trunk revision c0d9b93.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
          org.apache.hadoop.hdfs.TestLeaseRecovery2

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.fs.TestSymlinkHdfsFileContext

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9627//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9627//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699828/HDFS-7770.00.patch against trunk revision c0d9b93. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster org.apache.hadoop.hdfs.TestLeaseRecovery2 The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.TestSymlinkHdfsFileContext Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9627//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9627//console This message is automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach a trunk patch that covers document of storage type tag in dfs.datanode.data.dir for storage policies. I will add a branch-2 patch once the trunk patch is reviewed.

          Show
          xyao Xiaoyu Yao added a comment - Attach a trunk patch that covers document of storage type tag in dfs.datanode.data.dir for storage policies. I will add a branch-2 patch once the trunk patch is reviewed.

            People

            • Assignee:
              xyao Xiaoyu Yao
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development