Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6234

TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.4.0
    • Fix Version/s: 2.4.1
    • Component/s: datanode, test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      TestDatanodeConfig#testMemlockLimit fails to initialize a DataNode due to an invalid URI configured in dfs.datanode.data.dir.

      1. HDFS-6234.1.patch
        2 kB
        Chris Nauroth

        Activity

        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1755 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1755/)
        HDFS-6234. TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586682)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeConfig.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1755 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1755/ ) HDFS-6234 . TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586682 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeConfig.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1730 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1730/)
        HDFS-6234. TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586682)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeConfig.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1730 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1730/ ) HDFS-6234 . TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586682 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeConfig.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #538 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/538/)
        HDFS-6234. TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586682)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeConfig.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #538 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/538/ ) HDFS-6234 . TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586682 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeConfig.java
        Hide
        Chris Nauroth added a comment -

        I committed this to trunk, branch-2 and branch-2.4. Thanks for the review, Jing!

        Show
        Chris Nauroth added a comment - I committed this to trunk, branch-2 and branch-2.4. Thanks for the review, Jing!
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5500 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5500/)
        HDFS-6234. TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586682)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeConfig.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5500 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5500/ ) HDFS-6234 . TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586682 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeConfig.java
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12639740/HDFS-6234.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6650//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6650//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639740/HDFS-6234.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6650//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6650//console This message is automatically generated.
        Hide
        Jing Zhao added a comment -

        Patch looks good to me. +1. Also thanks for cleaning the test code, Chris Nauroth!

        Show
        Jing Zhao added a comment - Patch looks good to me. +1. Also thanks for cleaning the test code, Chris Nauroth !
        Hide
        Chris Nauroth added a comment -

        I'm attaching a patch that sets a valid URI in dfs.datanode.data.dir. While I was in here, I also made some minor changes to make sure every created DataNode gets shut down. I ran the test successfully on Mac and Windows with this patch.

        Show
        Chris Nauroth added a comment - I'm attaching a patch that sets a valid URI in dfs.datanode.data.dir . While I was in here, I also made some minor changes to make sure every created DataNode gets shut down. I ran the test successfully on Mac and Windows with this patch.
        Hide
        Chris Nauroth added a comment -

        Here is the default for dfs.datanode.data.dir:

        file://${hadoop.tmp.dir}/dfs/data
        

        hadoop.tmp.dir will be a Windows file system path with back slashes. The problem is that prepending "file://" in front of a Windows file system path does not necessarily yield a valid URI because of the back slashes. The test fails with this exception:

        java.lang.IllegalArgumentException: Failed to parse conf property dfs.datanode.data.dir: file://D:\w\hbk\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
        	at java.io.WinNTFileSystem.canonicalize0(Native Method)
        	at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:414)
        	at java.io.File.getCanonicalPath(File.java:589)
        	at java.io.File.getCanonicalFile(File.java:614)
        	at org.apache.hadoop.hdfs.server.common.Util.fileAsURI(Util.java:73)
        	at org.apache.hadoop.hdfs.server.common.Util.stringAsURI(Util.java:58)
        	at org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:94)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:1784)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1768)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1812)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1802)
        	at org.apache.hadoop.hdfs.TestDatanodeConfig.testMemlockLimit(TestDatanodeConfig.java:133)
        
        Show
        Chris Nauroth added a comment - Here is the default for dfs.datanode.data.dir : file: //${hadoop.tmp.dir}/dfs/data hadoop.tmp.dir will be a Windows file system path with back slashes. The problem is that prepending "file://" in front of a Windows file system path does not necessarily yield a valid URI because of the back slashes. The test fails with this exception: java.lang.IllegalArgumentException: Failed to parse conf property dfs.datanode.data.dir: file: //D:\w\hbk\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data at java.io.WinNTFileSystem.canonicalize0(Native Method) at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:414) at java.io.File.getCanonicalPath(File.java:589) at java.io.File.getCanonicalFile(File.java:614) at org.apache.hadoop.hdfs.server.common.Util.fileAsURI(Util.java:73) at org.apache.hadoop.hdfs.server.common.Util.stringAsURI(Util.java:58) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:94) at org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:1784) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1768) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1812) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1802) at org.apache.hadoop.hdfs.TestDatanodeConfig.testMemlockLimit(TestDatanodeConfig.java:133)

          People

          • Assignee:
            Chris Nauroth
            Reporter:
            Chris Nauroth
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development