Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4104

dfs -test -d prints inappropriate error on nonexistent directory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha
    • Fix Version/s: 2.0.3-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Running hdfs dfs -test -d foo should return 0 or 1 as appropriate. It should not generate any output due to missing files. Alas, it prints an error message when foo does not exist.

      $ hdfs dfs -test -d foo; echo $?
      test: `foo': No such file or directory
      1
      
      1. hdfs-4104.txt
        0.7 kB
        Andy Isaacson
      2. hdfs4104-2.txt
        2 kB
        Andy Isaacson
      3. hdfs4104-3.txt
        5 kB
        Andy Isaacson
      4. hdfs4104-4.txt
        5 kB
        Andy Isaacson

        Issue Links

          Activity

          Andy Isaacson created issue -
          Andy Isaacson made changes -
          Field Original Value New Value
          Summary dfs -test -d prints inappropriate error on nonexistent directory dfs -test -d prints inappropriate error on nonexistent directory, and -z should be -s
          Andy Isaacson made changes -
          Summary dfs -test -d prints inappropriate error on nonexistent directory, and -z should be -s dfs -test -d prints inappropriate error on nonexistent directory
          Hide
          Andy Isaacson added a comment -

          Remove code to print errors on file not found from dfs -test, for feature parity with /usr/bin/test.

          The existing test frameworks don't provide a way to verify that stderr is clean, so no tests updated.

          Show
          Andy Isaacson added a comment - Remove code to print errors on file not found from dfs -test , for feature parity with /usr/bin/test. The existing test frameworks don't provide a way to verify that stderr is clean, so no tests updated.
          Andy Isaacson made changes -
          Attachment hdfs-4104.txt [ 12550383 ]
          Andy Isaacson made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Andy Isaacson made changes -
          Link This issue relates to HDFS-1331 [ HDFS-1331 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550383/hdfs-4104.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3381//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3381//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550383/hdfs-4104.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3381//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3381//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          +1 I wanted to make this change while redesigned the shell but was attempting to preserve backwards compat at the time. If noone else objects, I'll commit tomorrow.

          Show
          Daryn Sharp added a comment - +1 I wanted to make this change while redesigned the shell but was attempting to preserve backwards compat at the time. If noone else objects, I'll commit tomorrow.
          Hide
          Daryn Sharp added a comment -

          I actually just noticed there are no tests included. The precommit build was fooled since the file changed was fs/shell/Test.java. Please add some tests and I'll commit.

          Show
          Daryn Sharp added a comment - I actually just noticed there are no tests included. The precommit build was fooled since the file changed was fs/shell/Test.java . Please add some tests and I'll commit.
          Hide
          Andy Isaacson added a comment -

          I actually just noticed there are no tests included.

          As I mentioned, I don't see any way in JUnit to verify that stderr is clean. TestDFSShell passes with this patch applied, and manual testing verifies that the message no longer appears.

          Show
          Andy Isaacson added a comment - I actually just noticed there are no tests included. As I mentioned, I don't see any way in JUnit to verify that stderr is clean. TestDFSShell passes with this patch applied, and manual testing verifies that the message no longer appears.
          Hide
          Daryn Sharp added a comment -

          You can add tests to the existing TestDFSShell#testErrOutPut. It's already checking stderr for a number of commands.

          Show
          Daryn Sharp added a comment - You can add tests to the existing TestDFSShell#testErrOutPut . It's already checking stderr for a number of commands.
          Hide
          Andy Isaacson added a comment -

          Thanks for the pointer to testErrOutPut, Daryn. It successfully unit tests this change (test fails without the fix, passes with the fix).

          Show
          Andy Isaacson added a comment - Thanks for the pointer to testErrOutPut, Daryn. It successfully unit tests this change (test fails without the fix, passes with the fix).
          Andy Isaacson made changes -
          Attachment hdfs4104-2.txt [ 12551299 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551299/hdfs4104-2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.server.balancer.TestBalancer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3423//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3423//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551299/hdfs4104-2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.balancer.TestBalancer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3423//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3423//console This message is automatically generated.
          Hide
          Andy Isaacson added a comment -

          Update TestHDFSCLI for new -test output semantics.

          Show
          Andy Isaacson added a comment - Update TestHDFSCLI for new -test output semantics.
          Andy Isaacson made changes -
          Attachment hdfs4104-3.txt [ 12551406 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551406/hdfs4104-3.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3429//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3429//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551406/hdfs4104-3.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3429//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3429//console This message is automatically generated.
          Hide
          Colin Patrick McCabe added a comment -

          looks reasonable to me

          Show
          Colin Patrick McCabe added a comment - looks reasonable to me
          Hide
          Daryn Sharp added a comment -

          +1 Looks good to me. Just add an assert to check the return code in the new test and I'll commit.

          Show
          Daryn Sharp added a comment - +1 Looks good to me. Just add an assert to check the return code in the new test and I'll commit.
          Hide
          Andy Isaacson added a comment -

          add assertion of correct exit value for "dfs -test -d".

          @@ -459,6 +459,7 @@
                 argv[2] = "/no/such/dir";
                 ret = ToolRunner.run(shell, argv);
                 returned = out.toString();
          +      assertEquals(" -test -d wrong result ", 1, ret);
                 assertTrue(returned.isEmpty());
               } finally {
                 if (bak != null) {
          
          Show
          Andy Isaacson added a comment - add assertion of correct exit value for "dfs -test -d". @@ -459,6 +459,7 @@ argv[2] = "/no/such/dir" ; ret = ToolRunner.run(shell, argv); returned = out.toString(); + assertEquals( " -test -d wrong result " , 1, ret); assertTrue(returned.isEmpty()); } finally { if (bak != null ) {
          Andy Isaacson made changes -
          Attachment hdfs4104-4.txt [ 12553604 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12553604/hdfs4104-4.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestHDFSFileSystemContract

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3514//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3514//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553604/hdfs4104-4.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestHDFSFileSystemContract +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3514//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3514//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          +1 Looks good! I'll commit shortly.

          Show
          Daryn Sharp added a comment - +1 Looks good! I'll commit shortly.
          Hide
          Daryn Sharp added a comment -

          I've committed to trunk and branch-2. Thanks Andy!

          Show
          Daryn Sharp added a comment - I've committed to trunk and branch-2. Thanks Andy!
          Daryn Sharp made changes -
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 3.0.0 [ 12320356 ]
          Fix Version/s 2.0.3-alpha [ 12323274 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3024 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3024/)
          HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3024 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3024/ ) HDFS-4104 . dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/38/)
          HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/38/ ) HDFS-4104 . dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1228/)
          HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1228/ ) HDFS-4104 . dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1259 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1259/)
          HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848)

          Result = FAILURE
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1259 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1259/ ) HDFS-4104 . dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320356 ]

            People

            • Assignee:
              Andy Isaacson
              Reporter:
              Andy Isaacson
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development