Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      Data-node directories dfs.datanode.data.dir should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

      1. dataDirURI.patch
        13 kB
        Konstantin Shvachko
      2. dataDirURI.patch
        13 kB
        Konstantin Shvachko
      3. dataDirURI-0-21.patch
        13 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #96 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/96/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #96 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/96/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #198 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/198/)
          Remove duplicate lines introduced by .
          . Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #198 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/198/ ) Remove duplicate lines introduced by . . Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #183 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/183/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #183 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/183/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #166 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/166/)
          Remove duplicate lines introduced by .
          . Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #166 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/166/ ) Remove duplicate lines introduced by . . Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.

          Show
          Konstantin Shvachko added a comment - I just committed this.
          Hide
          Konstantin Shvachko added a comment -

          Here is the patch for 0.21.

          Show
          Konstantin Shvachko added a comment - Here is the patch for 0.21.
          Hide
          Suresh Srinivas added a comment -

          +1 for applying this to 0.21. This should have been part of HDFS-396 to begin with.

          Show
          Suresh Srinivas added a comment - +1 for applying this to 0.21. This should have been part of HDFS-396 to begin with.
          Hide
          Konstantin Shvachko added a comment -

          Should we apply this to 0.21?
          This is an incompatible change. But this makes configuration of name-node and data-node directories consistent. I am in doubt.

          Show
          Konstantin Shvachko added a comment - Should we apply this to 0.21? This is an incompatible change. But this makes configuration of name-node and data-node directories consistent. I am in doubt.
          Hide
          Konstantin Shvachko added a comment -

          Makes sense. I added a test case where URI has an authority part in it.
          Updated patch to the latest trunk.
          Checked the test failure: TestReadWhileWriting failed because of some lease recovery issues. This is not related to this patch.

          Show
          Konstantin Shvachko added a comment - Makes sense. I added a test case where URI has an authority part in it. Updated patch to the latest trunk. Checked the test failure: TestReadWhileWriting failed because of some lease recovery issues. This is not related to this patch.
          Hide
          Eli Collins added a comment -

          +1 Patch looks nice. Probably worth adding a third case in testDataDirectories for a URI that includes an authority (file://localhost/path).

          Show
          Eli Collins added a comment - +1 Patch looks nice. Probably worth adding a third case in testDataDirectories for a URI that includes an authority ( file://localhost/path ).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429801/dataDirURI.patch
          against trunk revision 897068.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429801/dataDirURI.patch against trunk revision 897068. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          With this patch data-node directories are specified as URIs.

          • It is backward compatible, that is if config specifies a path data-node will print a warning but will keep working with the directory.
          • We support only "file:" URIs. If another scheme is specified the directory will be ignored.
          • The patch includes a test to test this behavior.
          • Changed hdfs-default.xml to set an example.
          • MiniDFSCluster is changed to set directories as URIs in order to avoid warnings.
          • TestStorageDirectories was failing on windows because the drive letter "c:" in directory paths was capitalized in one case and not in an another, so the string comparison of the paths was failing. Fixed it.
          Show
          Konstantin Shvachko added a comment - With this patch data-node directories are specified as URIs. It is backward compatible, that is if config specifies a path data-node will print a warning but will keep working with the directory. We support only "file:" URIs. If another scheme is specified the directory will be ignored. The patch includes a test to test this behavior. Changed hdfs-default.xml to set an example. MiniDFSCluster is changed to set directories as URIs in order to avoid warnings. TestStorageDirectories was failing on windows because the drive letter "c:" in directory paths was capitalized in one case and not in an another, so the string comparison of the paths was failing. Fixed it.
          Hide
          Konstantin Shvachko added a comment -

          Currently it is rather confusing to have name-node and data-node storage directories configured in different formats.

          Show
          Konstantin Shvachko added a comment - Currently it is rather confusing to have name-node and data-node storage directories configured in different formats.

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development