Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4104

dfs -test -d prints inappropriate error on nonexistent directory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha
    • Fix Version/s: 2.0.3-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Running hdfs dfs -test -d foo should return 0 or 1 as appropriate. It should not generate any output due to missing files. Alas, it prints an error message when foo does not exist.

      $ hdfs dfs -test -d foo; echo $?
      test: `foo': No such file or directory
      1
      
      1. hdfs4104-4.txt
        5 kB
        Andy Isaacson
      2. hdfs4104-3.txt
        5 kB
        Andy Isaacson
      3. hdfs4104-2.txt
        2 kB
        Andy Isaacson
      4. hdfs-4104.txt
        0.7 kB
        Andy Isaacson

        Issue Links

          Activity

          Hide
          Andy Isaacson added a comment -

          Remove code to print errors on file not found from dfs -test, for feature parity with /usr/bin/test.

          The existing test frameworks don't provide a way to verify that stderr is clean, so no tests updated.

          Show
          Andy Isaacson added a comment - Remove code to print errors on file not found from dfs -test , for feature parity with /usr/bin/test. The existing test frameworks don't provide a way to verify that stderr is clean, so no tests updated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550383/hdfs-4104.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3381//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3381//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550383/hdfs-4104.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3381//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3381//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          +1 I wanted to make this change while redesigned the shell but was attempting to preserve backwards compat at the time. If noone else objects, I'll commit tomorrow.

          Show
          Daryn Sharp added a comment - +1 I wanted to make this change while redesigned the shell but was attempting to preserve backwards compat at the time. If noone else objects, I'll commit tomorrow.
          Hide
          Daryn Sharp added a comment -

          I actually just noticed there are no tests included. The precommit build was fooled since the file changed was fs/shell/Test.java. Please add some tests and I'll commit.

          Show
          Daryn Sharp added a comment - I actually just noticed there are no tests included. The precommit build was fooled since the file changed was fs/shell/Test.java . Please add some tests and I'll commit.
          Hide
          Andy Isaacson added a comment -

          I actually just noticed there are no tests included.

          As I mentioned, I don't see any way in JUnit to verify that stderr is clean. TestDFSShell passes with this patch applied, and manual testing verifies that the message no longer appears.

          Show
          Andy Isaacson added a comment - I actually just noticed there are no tests included. As I mentioned, I don't see any way in JUnit to verify that stderr is clean. TestDFSShell passes with this patch applied, and manual testing verifies that the message no longer appears.
          Hide
          Daryn Sharp added a comment -

          You can add tests to the existing TestDFSShell#testErrOutPut. It's already checking stderr for a number of commands.

          Show
          Daryn Sharp added a comment - You can add tests to the existing TestDFSShell#testErrOutPut . It's already checking stderr for a number of commands.
          Hide
          Andy Isaacson added a comment -

          Thanks for the pointer to testErrOutPut, Daryn. It successfully unit tests this change (test fails without the fix, passes with the fix).

          Show
          Andy Isaacson added a comment - Thanks for the pointer to testErrOutPut, Daryn. It successfully unit tests this change (test fails without the fix, passes with the fix).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551299/hdfs4104-2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.server.balancer.TestBalancer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3423//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3423//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551299/hdfs4104-2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.balancer.TestBalancer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3423//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3423//console This message is automatically generated.
          Hide
          Andy Isaacson added a comment -

          Update TestHDFSCLI for new -test output semantics.

          Show
          Andy Isaacson added a comment - Update TestHDFSCLI for new -test output semantics.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551406/hdfs4104-3.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3429//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3429//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551406/hdfs4104-3.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3429//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3429//console This message is automatically generated.
          Hide
          Colin Patrick McCabe added a comment -

          looks reasonable to me

          Show
          Colin Patrick McCabe added a comment - looks reasonable to me
          Hide
          Daryn Sharp added a comment -

          +1 Looks good to me. Just add an assert to check the return code in the new test and I'll commit.

          Show
          Daryn Sharp added a comment - +1 Looks good to me. Just add an assert to check the return code in the new test and I'll commit.
          Hide
          Andy Isaacson added a comment -

          add assertion of correct exit value for "dfs -test -d".

          @@ -459,6 +459,7 @@
                 argv[2] = "/no/such/dir";
                 ret = ToolRunner.run(shell, argv);
                 returned = out.toString();
          +      assertEquals(" -test -d wrong result ", 1, ret);
                 assertTrue(returned.isEmpty());
               } finally {
                 if (bak != null) {
          
          Show
          Andy Isaacson added a comment - add assertion of correct exit value for "dfs -test -d". @@ -459,6 +459,7 @@ argv[2] = "/no/such/dir" ; ret = ToolRunner.run(shell, argv); returned = out.toString(); + assertEquals( " -test -d wrong result " , 1, ret); assertTrue(returned.isEmpty()); } finally { if (bak != null ) {
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12553604/hdfs4104-4.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestHDFSFileSystemContract

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3514//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3514//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553604/hdfs4104-4.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestHDFSFileSystemContract +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3514//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3514//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          +1 Looks good! I'll commit shortly.

          Show
          Daryn Sharp added a comment - +1 Looks good! I'll commit shortly.
          Hide
          Daryn Sharp added a comment -

          I've committed to trunk and branch-2. Thanks Andy!

          Show
          Daryn Sharp added a comment - I've committed to trunk and branch-2. Thanks Andy!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3024 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3024/)
          HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3024 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3024/ ) HDFS-4104 . dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/38/)
          HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/38/ ) HDFS-4104 . dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1228/)
          HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1228/ ) HDFS-4104 . dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1259 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1259/)
          HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848)

          Result = FAILURE
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1259 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1259/ ) HDFS-4104 . dfs -test -d prints inappropriate error on nonexistent directory (Andy Isaacson via daryn) (Revision 1409848) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409848 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml

            People

            • Assignee:
              Andy Isaacson
              Reporter:
              Andy Isaacson
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development