Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      Data-node directories dfs.datanode.data.dir should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

      1. dataDirURI-0-21.patch
        13 kB
        Konstantin Shvachko
      2. dataDirURI.patch
        13 kB
        Konstantin Shvachko
      3. dataDirURI.patch
        13 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          2d 4h 40m 1 Konstantin Shvachko 09/Jan/10 00:49
          Patch Available Patch Available Resolved Resolved
          2d 23h 44m 1 Konstantin Shvachko 12/Jan/10 00:34
          Resolved Resolved Closed Closed
          224d 20h 16m 1 Tom White 24/Aug/10 21:50
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #96 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/96/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #96 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/96/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #198 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/198/)
          Remove duplicate lines introduced by .
          . Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #198 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/198/ ) Remove duplicate lines introduced by . . Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #183 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/183/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #183 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/183/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #166 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/166/)
          Remove duplicate lines introduced by .
          . Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #166 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/166/ ) Remove duplicate lines introduced by . . Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.
          Konstantin Shvachko made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change, Reviewed]
          Fix Version/s 0.21.0 [ 12314046 ]
          Resolution Fixed [ 1 ]
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.

          Show
          Konstantin Shvachko added a comment - I just committed this.
          Konstantin Shvachko made changes -
          Attachment dataDirURI-0-21.patch [ 12429955 ]
          Hide
          Konstantin Shvachko added a comment -

          Here is the patch for 0.21.

          Show
          Konstantin Shvachko added a comment - Here is the patch for 0.21.
          Hide
          Suresh Srinivas added a comment -

          +1 for applying this to 0.21. This should have been part of HDFS-396 to begin with.

          Show
          Suresh Srinivas added a comment - +1 for applying this to 0.21. This should have been part of HDFS-396 to begin with.
          Hide
          Konstantin Shvachko added a comment -

          Should we apply this to 0.21?
          This is an incompatible change. But this makes configuration of name-node and data-node directories consistent. I am in doubt.

          Show
          Konstantin Shvachko added a comment - Should we apply this to 0.21? This is an incompatible change. But this makes configuration of name-node and data-node directories consistent. I am in doubt.
          Konstantin Shvachko made changes -
          Attachment dataDirURI.patch [ 12429945 ]
          Hide
          Konstantin Shvachko added a comment -

          Makes sense. I added a test case where URI has an authority part in it.
          Updated patch to the latest trunk.
          Checked the test failure: TestReadWhileWriting failed because of some lease recovery issues. This is not related to this patch.

          Show
          Konstantin Shvachko added a comment - Makes sense. I added a test case where URI has an authority part in it. Updated patch to the latest trunk. Checked the test failure: TestReadWhileWriting failed because of some lease recovery issues. This is not related to this patch.
          Hide
          Eli Collins added a comment -

          +1 Patch looks nice. Probably worth adding a third case in testDataDirectories for a URI that includes an authority (file://localhost/path).

          Show
          Eli Collins added a comment - +1 Patch looks nice. Probably worth adding a third case in testDataDirectories for a URI that includes an authority ( file://localhost/path ).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429801/dataDirURI.patch
          against trunk revision 897068.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429801/dataDirURI.patch against trunk revision 897068. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/console This message is automatically generated.
          Konstantin Shvachko made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Konstantin Shvachko made changes -
          Attachment dataDirURI.patch [ 12429801 ]
          Hide
          Konstantin Shvachko added a comment -

          With this patch data-node directories are specified as URIs.

          • It is backward compatible, that is if config specifies a path data-node will print a warning but will keep working with the directory.
          • We support only "file:" URIs. If another scheme is specified the directory will be ignored.
          • The patch includes a test to test this behavior.
          • Changed hdfs-default.xml to set an example.
          • MiniDFSCluster is changed to set directories as URIs in order to avoid warnings.
          • TestStorageDirectories was failing on windows because the drive letter "c:" in directory paths was capitalized in one case and not in an another, so the string comparison of the paths was failing. Fixed it.
          Show
          Konstantin Shvachko added a comment - With this patch data-node directories are specified as URIs. It is backward compatible, that is if config specifies a path data-node will print a warning but will keep working with the directory. We support only "file:" URIs. If another scheme is specified the directory will be ignored. The patch includes a test to test this behavior. Changed hdfs-default.xml to set an example. MiniDFSCluster is changed to set directories as URIs in order to avoid warnings. TestStorageDirectories was failing on windows because the drive letter "c:" in directory paths was capitalized in one case and not in an another, so the string comparison of the paths was failing. Fixed it.
          Hide
          Konstantin Shvachko added a comment -

          Currently it is rather confusing to have name-node and data-node storage directories configured in different formats.

          Show
          Konstantin Shvachko added a comment - Currently it is rather confusing to have name-node and data-node storage directories configured in different formats.
          Konstantin Shvachko made changes -
          Link This issue is related to HDFS-456 [ HDFS-456 ]
          Konstantin Shvachko made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-396 [ HDFS-396 ]
          Konstantin Shvachko created issue -

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development