Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-463

CreateEditsLog utility broken due to FSImage URL scheme check

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      CreateEditLog FSImage

      Description

      CreateEditsLog creates editslog using FSImage class. With new URI scheme test added, the utility fails.

      1. HDFS-463.patch
        1 kB
        Suresh Srinivas
      2. HDFS-463.patch
        1 kB
        Suresh Srinivas
      3. HDFS-463.patch
        0.9 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #25 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/25/)
          . CreateEditLog utility broken after HDFS-396 (URI for
          FSImage). (Suresh Srinivas via rangadi)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #25 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/25/ ) . CreateEditLog utility broken after HDFS-396 (URI for FSImage). (Suresh Srinivas via rangadi)
          Hide
          Raghu Angadi added a comment -

          I just committed this. Thanks Suresh.

          Show
          Raghu Angadi added a comment - I just committed this. Thanks Suresh.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12413484/HDFS-463.patch
          against trunk revision 794054.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/21/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/21/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/21/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/21/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413484/HDFS-463.patch against trunk revision 794054. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/21/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/21/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/21/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/21/console This message is automatically generated.
          Hide
          Raghu Angadi added a comment -

          +1.

          Show
          Raghu Angadi added a comment - +1.
          Hide
          Suresh Srinivas added a comment -

          Updated patch incorporates the comments...

          Show
          Suresh Srinivas added a comment - Updated patch incorporates the comments...
          Hide
          Raghu Angadi added a comment -

          Right, you need the first hunk in your patch as well..

          Show
          Raghu Angadi added a comment - Right, you need the first hunk in your patch as well..
          Hide
          Raghu Angadi added a comment -

          Does it imply CreateEditLog works with non-file URIs?...

          Show
          Raghu Angadi added a comment - Does it imply CreateEditLog works with non-file URIs?...
          Hide
          Suresh Srinivas added a comment -

          That chagne will not be sufficient, edits_dir needs to be change to URI instead of String.

          CreaetEditLog stored directory (as String) and not URL. Since FSImage changed to use URI, I feel only the code that is constructing the FSImage should change. If you agree, I can incorporate what Konstantin has suggested to create URL from the String.

          Show
          Suresh Srinivas added a comment - That chagne will not be sufficient, edits_dir needs to be change to URI instead of String. CreaetEditLog stored directory (as String) and not URL. Since FSImage changed to use URI, I feel only the code that is constructing the FSImage should change. If you agree, I can incorporate what Konstantin has suggested to create URL from the String.
          Hide
          Raghu Angadi added a comment -

          Looks like the fix might just be replacing 'new URI(edits_dir)' with 'new File(edits_dir).getAbsoluteFile().toURI()'.

          Show
          Raghu Angadi added a comment - Looks like the fix might just be replacing 'new URI(edits_dir)' with 'new File(edits_dir).getAbsoluteFile().toURI()'.
          Hide
          Raghu Angadi added a comment -

          The fix may not work for windows (see this comment).

          CreateEditLog seems treat edits_dir as file in rest of the CrateEditLog.java. Shouldn't it just convert it to a file URI?

          Show
          Raghu Angadi added a comment - The fix may not work for windows ( see this comment ). CreateEditLog seems treat edits_dir as file in rest of the CrateEditLog.java. Shouldn't it just convert it to a file URI?
          Hide
          Suresh Srinivas added a comment -

          This bug is caused by changed from HDFS-396

          Show
          Suresh Srinivas added a comment - This bug is caused by changed from HDFS-396
          Hide
          Suresh Srinivas added a comment -

          Manual test - ensure following commands run successfully and create a file current/edits under /tmp/EditsLog directory.

          bin/hadoop org.apache.hadoop.hdfs.server.namenode.CreateEditsLog -f 1 0 1 -r 3 -d /tmp/EditsLog
          bin/hadoop org.apache.hadoop.hdfs.server.namenode.CreateEditsLog -f 1 0 1 -r 3
          
          Show
          Suresh Srinivas added a comment - Manual test - ensure following commands run successfully and create a file current/edits under /tmp/EditsLog directory. bin/hadoop org.apache.hadoop.hdfs.server.namenode.CreateEditsLog -f 1 0 1 -r 3 -d /tmp/EditsLog bin/hadoop org.apache.hadoop.hdfs.server.namenode.CreateEditsLog -f 1 0 1 -r 3
          Hide
          Suresh Srinivas added a comment -

          Passing expected file:// URL to the FSImage constuctor to fix the problem.

          Show
          Suresh Srinivas added a comment - Passing expected file:// URL to the FSImage constuctor to fix the problem.
          Hide
          Suresh Srinivas added a comment -

          Command fails with an exception shown below:

          bin/hadoop org.apache.hadoop.hdfs.server.namenode.CreateEditsLog -f 1 0 1 -r 3 -d /tmp
          Exception in thread "main" java.io.IOException: Undefined scheme for /private/tmp
                  at org.apache.hadoop.hdfs.server.namenode.FSImage.checkSchemeConsistency(FSImage.java:254)
                  at org.apache.hadoop.hdfs.server.namenode.FSImage.setStorageDirectories(FSImage.java:213)
                  at org.apache.hadoop.hdfs.server.namenode.FSImage.<init>(FSImage.java:186)
                  at org.apache.hadoop.hdfs.server.namenode.CreateEditsLog.main(CreateEditsLog.java:203)
          
          Show
          Suresh Srinivas added a comment - Command fails with an exception shown below: bin/hadoop org.apache.hadoop.hdfs.server.namenode.CreateEditsLog -f 1 0 1 -r 3 -d /tmp Exception in thread "main" java.io.IOException: Undefined scheme for /private/tmp at org.apache.hadoop.hdfs.server.namenode.FSImage.checkSchemeConsistency(FSImage.java:254) at org.apache.hadoop.hdfs.server.namenode.FSImage.setStorageDirectories(FSImage.java:213) at org.apache.hadoop.hdfs.server.namenode.FSImage.<init>(FSImage.java:186) at org.apache.hadoop.hdfs.server.namenode.CreateEditsLog.main(CreateEditsLog.java:203)

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development