Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: datanode, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The last portion of the dfs.datanode.data.dir is incorrectly formatted.

      2015-06-14 09:44:37,133 INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:startDataNodes(1413)) - Starting DataNode 0 with dfs.datanode.data.dir: file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
      2015-06-14 09:44:37,141 ERROR common.Util (Util.java:stringAsURI(50)) - Syntax error in URI file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data. Please check hdfs configuration.
      java.net.URISyntaxException: Illegal character in authority at index 7: file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
      
      1. HDFS-8597.00.patch
        1 kB
        Xiaoyu Yao
      2. HDFS-8597.01.patch
        1 kB
        Xiaoyu Yao

        Activity

        Hide
        xyao Xiaoyu Yao added a comment -

        Attach a patch that fixes the datanode storage location parsing for windows.

        Show
        xyao Xiaoyu Yao added a comment - Attach a patch that fixes the datanode storage location parsing for windows.
        Hide
        xyao Xiaoyu Yao added a comment -

        The test takes dfs.datanode.data.dir from "hdfs-default.xml"

          <name>dfs.datanode.data.dir</name>
          <value>file://${hadoop.tmp.dir}/dfs/data</value>
        

        The location ends up as file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data before passing to Util.stringAsURI(). However, the Java URI class cannot handle mix '/' and '\' separator in the path, which causes the exception.

        The fix is to use Java Path class to normalize the path separator and then convert it to URI.

        Show
        xyao Xiaoyu Yao added a comment - The test takes dfs.datanode.data.dir from "hdfs-default.xml" <name>dfs.datanode.data.dir</name> <value>file: //${hadoop.tmp.dir}/dfs/data</value> The location ends up as file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data before passing to Util.stringAsURI(). However, the Java URI class cannot handle mix '/' and '\' separator in the path, which causes the exception. The fix is to use Java Path class to normalize the path separator and then convert it to URI.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 54s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 33s There were no new javac warning messages.
        +1 javadoc 9m 38s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 16s The applied patch generated 1 new checkstyle issues (total was 4, now 5).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 13s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 15s Pre-build of native portion
        +1 hdfs tests 162m 21s Tests passed in hadoop-hdfs.
            208m 44s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12739690/HDFS-8597.00.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 75a2560
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11363/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11363/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11363/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11363/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 16s The applied patch generated 1 new checkstyle issues (total was 4, now 5). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 13s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion +1 hdfs tests 162m 21s Tests passed in hadoop-hdfs.     208m 44s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739690/HDFS-8597.00.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 75a2560 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11363/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11363/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11363/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11363/console This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        Update patch that addresses the checkstyle issue by removing the unused import.

        Show
        xyao Xiaoyu Yao added a comment - Update patch that addresses the checkstyle issue by removing the unused import.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 20s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 37s There were no new javac warning messages.
        +1 javadoc 9m 51s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 2m 16s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 3m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 23s Pre-build of native portion
        +1 hdfs tests 162m 18s Tests passed in hadoop-hdfs.
            209m 48s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12739778/HDFS-8597.01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 32e39d8
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11365/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11365/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11365/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 20s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 51s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 16s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 3m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 23s Pre-build of native portion +1 hdfs tests 162m 18s Tests passed in hadoop-hdfs.     209m 48s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739778/HDFS-8597.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 32e39d8 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11365/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11365/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11365/console This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for the patch. I verified it on Windows. Thank you, Xiaoyu.

        Show
        cnauroth Chris Nauroth added a comment - +1 for the patch. I verified it on Windows. Thank you, Xiaoyu.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Chris Nauroth for the review. I've committed the patch to trunk, branch-2 and branch-2.7.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Chris Nauroth for the review. I've committed the patch to trunk, branch-2 and branch-2.7.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8027 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8027/)
        HDFS-8597. Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8027 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8027/ ) HDFS-8597 . Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #961 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/961/)
        HDFS-8597. Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #961 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/961/ ) HDFS-8597 . Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #231 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/231/)
        HDFS-8597. Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #231 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/231/ ) HDFS-8597 . Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/220/)
        HDFS-8597. Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/220/ ) HDFS-8597 . Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #229 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/229/)
        HDFS-8597. Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #229 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/229/ ) HDFS-8597 . Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2177 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2177/)
        HDFS-8597. Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2177 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2177/ ) HDFS-8597 . Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2159/)
        HDFS-8597. Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2159/ ) HDFS-8597 . Fix TestFSImage#testZeroBlockSize on Windows. Contributed by Xiaoyu Yao. (xyao: rev 4e88ff5b27cc33d311ab7a7248c3cf6303997ddd) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java

          People

          • Assignee:
            xyao Xiaoyu Yao
            Reporter:
            xyao Xiaoyu Yao
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development