Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5513

CacheAdmin commands fail when using . as the path

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: caching, tools
    • Labels:
      None

      Description

      The hdfs CLI commands generally accept "." as a path argument.
      e.g.

      hdfs dfs -rm .
      hdfs dfsadmin -allowSnapshot .
      

      I don't think it's very common to use the path "." but the CacheAdmin commands will fail saying that it cannot create a Path from an empty string.

      [schu@hdfs-c5-nfs ~]$ hdfs cacheadmin -removeDirectives -path .
      Exception in thread "main" java.lang.IllegalArgumentException: Can not create a Path from an empty string
      	at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
      	at org.apache.hadoop.fs.Path.<init>(Path.java:184)
      	at org.apache.hadoop.hdfs.protocol.PathBasedCacheDirective$Builder.<init>(PathBasedCacheDirective.java:66)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.listPathBasedCacheDirectives(DistributedFileSystem.java:1639)
      	at org.apache.hadoop.hdfs.tools.CacheAdmin$RemovePathBasedCacheDirectivesCommand.run(CacheAdmin.java:365)
      	at org.apache.hadoop.hdfs.tools.CacheAdmin.run(CacheAdmin.java:82)
      	at org.apache.hadoop.hdfs.tools.CacheAdmin.main(CacheAdmin.java:87)
      [schu@hdfs-c5-nfs ~]$ hdfs cacheadmin -addDirective -path . -pool schu
      Exception in thread "main" java.lang.IllegalArgumentException: Can not create a Path from an empty string
      	at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
      	at org.apache.hadoop.fs.Path.<init>(Path.java:184)
      	at org.apache.hadoop.hdfs.protocol.PathBasedCacheDirective$Builder.<init>(PathBasedCacheDirective.java:66)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.addPathBasedCacheDirective(DistributedFileSystem.java:1598)
      	at org.apache.hadoop.hdfs.tools.CacheAdmin$AddPathBasedCacheDirectiveCommand.run(CacheAdmin.java:180)
      	at org.apache.hadoop.hdfs.tools.CacheAdmin.run(CacheAdmin.java:82)
      	at org.apache.hadoop.hdfs.tools.CacheAdmin.main(CacheAdmin.java:87)
      
      1. hdfs-5513-1.patch
        4 kB
        Andrew Wang
      2. hdfs-5513-2.patch
        4 kB
        Andrew Wang
      3. hdfs-5513-3.patch
        3 kB
        Andrew Wang

        Activity

        Hide
        Andrew Wang added a comment -

        This was caused by the attempt to deep-copy in the PBCD Builder. Paths normalize the URI upon creation, so the single . simply gets thrown away. There doesn't seem to be a way to deep-copy a Path, but at the same time it doesn't look like you can mutate a Path either.

        This didn't crop up on the other relative path tests because they aren't a single ".", so I added a unit test for that. Also tested manually via CacheAdmin.

        Show
        Andrew Wang added a comment - This was caused by the attempt to deep-copy in the PBCD Builder. Paths normalize the URI upon creation, so the single . simply gets thrown away. There doesn't seem to be a way to deep-copy a Path, but at the same time it doesn't look like you can mutate a Path either. This didn't crop up on the other relative path tests because they aren't a single ".", so I added a unit test for that. Also tested manually via CacheAdmin.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12614533/hdfs-5513-1.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5475//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614533/hdfs-5513-1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5475//console This message is automatically generated.
        Hide
        Andrew Wang added a comment -

        Rebase patch

        Show
        Andrew Wang added a comment - Rebase patch
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12614547/hdfs-5513-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5476//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5476//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614547/hdfs-5513-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5476//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5476//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        This was caused by the attempt to deep-copy in the PBCD Builder. Paths normalize the URI upon creation, so the single . simply gets thrown away. There doesn't seem to be a way to deep-copy a Path, but at the same time it doesn't look like you can mutate a Path either.

        I took another look and you are right. Although the Path does make its URI accessible to the outside world, the URI has no methods that could be used to mutate it.

        Can we merge testSingleDotPath into another junit test? It just seems kind of like overkill to set up a whole DFSCluster just to see if a PBCE with "." as the path can be added and then removed. It would be nice to keep test execution time down.

        +1 once that's addressed

        Show
        Colin Patrick McCabe added a comment - This was caused by the attempt to deep-copy in the PBCD Builder. Paths normalize the URI upon creation, so the single . simply gets thrown away. There doesn't seem to be a way to deep-copy a Path, but at the same time it doesn't look like you can mutate a Path either. I took another look and you are right. Although the Path does make its URI accessible to the outside world, the URI has no methods that could be used to mutate it. Can we merge testSingleDotPath into another junit test? It just seems kind of like overkill to set up a whole DFSCluster just to see if a PBCE with "." as the path can be added and then removed. It would be nice to keep test execution time down. +1 once that's addressed
        Hide
        Andrew Wang added a comment -

        Thanks Colin. I combined the test into one of the other ones in the file. Let's see what Jenkins thinks.

        Show
        Andrew Wang added a comment - Thanks Colin. I combined the test into one of the other ones in the file. Let's see what Jenkins thinks.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12614695/hdfs-5513-3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5492//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5492//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614695/hdfs-5513-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5492//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5492//console This message is automatically generated.
        Hide
        Andrew Wang added a comment -

        With Jenkins clean, will commit this shortly based on Colin's earlier +1. Thanks again for the review.

        Show
        Andrew Wang added a comment - With Jenkins clean, will commit this shortly based on Colin's earlier +1. Thanks again for the review.
        Hide
        Andrew Wang added a comment -

        Committed to trunk. Thanks again Colin for reviews.

        Show
        Andrew Wang added a comment - Committed to trunk. Thanks again Colin for reviews.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #4763 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4763/)
        HDFS-5513. CacheAdmin commands fail when using . as the path. Contributed by Andrew Wang. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543670)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/PathBasedCacheDirective.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPathBasedCacheRequests.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4763 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4763/ ) HDFS-5513 . CacheAdmin commands fail when using . as the path. Contributed by Andrew Wang. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543670 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/PathBasedCacheDirective.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPathBasedCacheRequests.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #397 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/397/)
        HDFS-5513. CacheAdmin commands fail when using . as the path. Contributed by Andrew Wang. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543670)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/PathBasedCacheDirective.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPathBasedCacheRequests.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #397 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/397/ ) HDFS-5513 . CacheAdmin commands fail when using . as the path. Contributed by Andrew Wang. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543670 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/PathBasedCacheDirective.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPathBasedCacheRequests.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #1588 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1588/)
        HDFS-5513. CacheAdmin commands fail when using . as the path. Contributed by Andrew Wang. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543670)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/PathBasedCacheDirective.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPathBasedCacheRequests.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1588 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1588/ ) HDFS-5513 . CacheAdmin commands fail when using . as the path. Contributed by Andrew Wang. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543670 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/PathBasedCacheDirective.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPathBasedCacheRequests.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1614 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1614/)
        HDFS-5513. CacheAdmin commands fail when using . as the path. Contributed by Andrew Wang. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543670)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/PathBasedCacheDirective.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPathBasedCacheRequests.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1614 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1614/ ) HDFS-5513 . CacheAdmin commands fail when using . as the path. Contributed by Andrew Wang. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1543670 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/PathBasedCacheDirective.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestPathBasedCacheRequests.java

          People

          • Assignee:
            Andrew Wang
            Reporter:
            Stephen Chu
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development