Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1786

Some cli test cases expect a "null" message

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are a few tests cases specified in TestHDFSCLI and TestDFSShell expecting "null" messages.
      e.g. in testHDFSConf.xml,

                <expected-output>get: null</expected-output>
      
      1. HDFS-1786_21.patch
        5 kB
        Uma Maheswara Rao G
      2. HDFS-1786.1.patch
        5 kB
        Uma Maheswara Rao G
      3. HDFS-1786.patch
        3 kB
        Uma Maheswara Rao G

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Todd Lipcon made changes -
          Fix Version/s 0.22.0 [ 12314241 ]
          Owen O'Malley made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.23.0 [ 12315571 ]
          Resolution Fixed [ 1 ]
          Hide
          Owen O'Malley added a comment -

          This has been committed to trunk.

          Show
          Owen O'Malley added a comment - This has been committed to trunk.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #582 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/582/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #582 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/582/ )
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Right now I cannot commit to 0.21 because the test is passing. If I add the patch, the test starts to fail. ...

          Okay.

          If you are starting a poll for abandoning 0.21, we could wait for the results. Otherwise, we should just fix 0.21.

          Show
          Tsz Wo Nicholas Sze added a comment - > Right now I cannot commit to 0.21 because the test is passing. If I add the patch, the test starts to fail. ... Okay. If you are starting a poll for abandoning 0.21, we could wait for the results. Otherwise, we should just fix 0.21.
          Hide
          Todd Lipcon added a comment -

          Right now I cannot commit to 0.21 because the test is passing. If I add the patch, the test starts to fail. Committing something that knowingly breaks tests doesn't seem like the right move.

          Looking at 0.21 more closely, it's not even properly set up to pull the hadoop-common-0.21.1-SNAPSHOT from ivy - it's still building against an 0.21.0. So, even though HADOOP-7174 was committed to common 0.21, it's not affecting HDFS 0.21 yet. I flipped ivy/libraries.properties over to 0.21.1-SNAPSHOT and now hdfs doesn't build at all - some issues related to Jasper that I can't quite discern.

          I will start a poll on the list to see if anyone would like to step up to maintain the 0.21 branch. If no one does, we can vote to officially abandon it.

          Show
          Todd Lipcon added a comment - Right now I cannot commit to 0.21 because the test is passing. If I add the patch, the test starts to fail. Committing something that knowingly breaks tests doesn't seem like the right move. Looking at 0.21 more closely, it's not even properly set up to pull the hadoop-common-0.21.1-SNAPSHOT from ivy - it's still building against an 0.21.0. So, even though HADOOP-7174 was committed to common 0.21, it's not affecting HDFS 0.21 yet. I flipped ivy/libraries.properties over to 0.21.1-SNAPSHOT and now hdfs doesn't build at all - some issues related to Jasper that I can't quite discern. I will start a poll on the list to see if anyone would like to step up to maintain the 0.21 branch. If no one does, we can vote to officially abandon it.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > As far as most people I know are concerned, 0.21 is a dead branch ...

          Unless we declare it dead officially, it is not dead yet. I believe many people thought that 0.21 was dead before the 0.21.0 release.

          So, could you commit it to 0.21?

          Show
          Tsz Wo Nicholas Sze added a comment - > As far as most people I know are concerned, 0.21 is a dead branch ... Unless we declare it dead officially, it is not dead yet. I believe many people thought that 0.21 was dead before the 0.21.0 release. So, could you commit it to 0.21?
          Hide
          Todd Lipcon added a comment -

          Makes sense, but when I applied it, tests started failing that were previously passing.

          As far as most people I know are concerned, 0.21 is a dead branch, so I don't think we need to worry about it.

          Show
          Todd Lipcon added a comment - Makes sense, but when I applied it, tests started failing that were previously passing. As far as most people I know are concerned, 0.21 is a dead branch, so I don't think we need to worry about it.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have not tried but it totally makes sense to commit this to 0.21. No?

          Show
          Tsz Wo Nicholas Sze added a comment - I have not tried but it totally makes sense to commit this to 0.21. No?
          Hide
          Todd Lipcon added a comment -

          Oh, strange... so does TestHDFSCLI fail for you when you build HDFS 0.21?

          Show
          Todd Lipcon added a comment - Oh, strange... so does TestHDFSCLI fail for you when you build HDFS 0.21?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Todd, I just have double checked HADOOP-7174. It is indeed committed to 0.21. Below is copied from http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.21/CHANGES.txt?view=log

          Revision 1085045 - (view) (download) (annotate) - [select for diffs]
          Modified Thu Mar 24 17:24:05 2011 UTC (5 days, 4 hours ago) by szetszwo
          File length: 404919 byte(s)
          Diff to previous 1084771 (colored)
          
          HADOOP-7174. Null is displayed in the "fs -copyToLocal" command.  Contributed by Uma Maheswara Rao G 
          
          Show
          Tsz Wo Nicholas Sze added a comment - Hi Todd, I just have double checked HADOOP-7174 . It is indeed committed to 0.21. Below is copied from http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.21/CHANGES.txt?view=log Revision 1085045 - (view) (download) (annotate) - [select for diffs] Modified Thu Mar 24 17:24:05 2011 UTC (5 days, 4 hours ago) by szetszwo File length: 404919 byte(s) Diff to previous 1084771 (colored) HADOOP-7174. Null is displayed in the "fs -copyToLocal" command. Contributed by Uma Maheswara Rao G
          Hide
          Todd Lipcon added a comment -

          I went to commit this to the 0.21 branch, but it looks like TestHDFSCLI is already passing on a fresh 0.21 checkout without this patch. Looking closely, I think the metadata on HADOOP-7174 is incorrect, it wasn't committed to 0.21. I think that's OK, since I don't think anyone plans on an 0.21.1.

          Show
          Todd Lipcon added a comment - I went to commit this to the 0.21 branch, but it looks like TestHDFSCLI is already passing on a fresh 0.21 checkout without this patch. Looking closely, I think the metadata on HADOOP-7174 is incorrect, it wasn't committed to 0.21. I think that's OK, since I don't think anyone plans on an 0.21.1.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks, Todd. Could you also commit the 0.21 patch?

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks, Todd. Could you also commit the 0.21 patch?
          Hide
          Uma Maheswara Rao G added a comment -

          HDFS-1786_21.patch is for 0.21 version.

          Show
          Uma Maheswara Rao G added a comment - HDFS-1786 _21.patch is for 0.21 version.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12474891/HDFS-1786_21.patch
          against trunk revision 1086479.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 17 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/301//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12474891/HDFS-1786_21.patch against trunk revision 1086479. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 17 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/301//console This message is automatically generated.
          Uma Maheswara Rao G made changes -
          Attachment HDFS-1786_21.patch [ 12474891 ]
          Hide
          Todd Lipcon added a comment -

          I verified that this decreases the number of failing HDFSCLI tests by 9. I'm going to commit this to trunk and 0.22 while we wait for an 0.21 patch, just to get trunk back closer to green.

          Show
          Todd Lipcon added a comment - I verified that this decreases the number of failing HDFSCLI tests by 9. I'm going to commit this to trunk and 0.22 while we wait for an 0.21 patch, just to get trunk back closer to green.
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the patch look good (forgot to say it earlier)

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the patch look good (forgot to say it earlier)
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Uma, since HADOOP-7174 was also committed to 0.21.1, could you provide a corresponding patch? The current patch does not apply to 0.21.1.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Uma, since HADOOP-7174 was also committed to 0.21.1, could you provide a corresponding patch? The current patch does not apply to 0.21.1.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I checked the test log build #293. All failures are chmod/chown/chgrp related but not null messages.

          Show
          Tsz Wo Nicholas Sze added a comment - I checked the test log build #293 . All failures are chmod/chown/chgrp related but not null messages.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12474791/HDFS-1786.1.patch
          against trunk revision 1085509.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 17 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.TestDFSShell

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/293//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/293//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/293//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12474791/HDFS-1786.1.patch against trunk revision 1085509. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 17 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestDFSShell -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/293//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/293//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/293//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          I provided the patch for this issue.
          And raised the above mentioned issue separately as HADOOP-7210
          TestDFSShell tests will pass after committing HADOOP-7210

          Show
          Uma Maheswara Rao G added a comment - I provided the patch for this issue. And raised the above mentioned issue separately as HADOOP-7210 TestDFSShell tests will pass after committing HADOOP-7210
          Uma Maheswara Rao G made changes -
          Attachment HDFS-1786.1.patch [ 12474791 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I see. Thanks for checking it.

          Show
          Tsz Wo Nicholas Sze added a comment - I see. Thanks for checking it.
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Nicholas,
          I fixed null message releated testcases.

          TestDFSShell failures are not because of HADOOP-7174 patch.
          I debugged the test failures for TestDFSShell and found the root cause
          Looks this is a bug in FsShellPermissions.java. ( for -chown argument)

          Bease on arguments it is creating the Handlers. when argument is -chown, it is creating ChownHandler.

           else if (cmd.equals("-chown")) {
                handler = new ChownHandler(argv[startIndex++]);
          

          But here the problem is, it is not populating owner, group values. just it is calling supper.

            protected ChownHandler(String cmd) { //for chgrp
                super(cmd);
              }
          

          Since it is not pupulating setOwner will not be invoked on fileSystem.So permissions will not change.
          Because of this assertions are failing TestDFSShell.
          After correcting this tests are passing.

          Since code change required here, i will raise separate issue for that in HDFS and will give patch.

          Show
          Uma Maheswara Rao G added a comment - Hi Nicholas, I fixed null message releated testcases. TestDFSShell failures are not because of HADOOP-7174 patch. I debugged the test failures for TestDFSShell and found the root cause Looks this is a bug in FsShellPermissions.java. ( for -chown argument) Bease on arguments it is creating the Handlers. when argument is -chown, it is creating ChownHandler. else if (cmd.equals( "-chown" )) { handler = new ChownHandler(argv[startIndex++]); But here the problem is, it is not populating owner, group values. just it is calling supper. protected ChownHandler( String cmd) { // for chgrp super (cmd); } Since it is not pupulating setOwner will not be invoked on fileSystem.So permissions will not change. Because of this assertions are failing TestDFSShell. After correcting this tests are passing. Since code change required here, i will raise separate issue for that in HDFS and will give patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > I came to know, if we provide regexComparator, then * can work.
          That's great!

          Please also fix TestDFSShell.

          BTW, you may run the tests local by
          ant run-test-hdfs -Dtestcase=TestDFSShell
          ant run-test-hdfs -Dtestcase=TestHDFSCLI

          Show
          Tsz Wo Nicholas Sze added a comment - > I came to know, if we provide regexComparator, then * can work. That's great! Please also fix TestDFSShell . BTW, you may run the tests local by ant run-test-hdfs -Dtestcase=TestDFSShell ant run-test-hdfs -Dtestcase=TestHDFSCLI
          Hide
          Uma Maheswara Rao G added a comment -

          I came to know, if we provide regexComparator, then * can work.
          I will prepare patch with that and provide.

          Show
          Uma Maheswara Rao G added a comment - I came to know, if we provide regexComparator, then * can work. I will prepare patch with that and provide.
          Hide
          Uma Maheswara Rao G added a comment -

          Since we are giving the message dinamically by appending the path, it is difficult to assert. I am not familiar with thi TestHDFSCLI framework.

          Expected output: [get: hdfs:*///*user/file: No such file or directory]
          Actual output: [get: hdfs:/user/file: No such file or directory

          ................
          One question here: can i use * (regex) for matching the path in testHDFSConf.xml file?

          for example:
          copyToLocal: *: No such file or directory....will it work with this framework?

          Show
          Uma Maheswara Rao G added a comment - Since we are giving the message dinamically by appending the path, it is difficult to assert. I am not familiar with thi TestHDFSCLI framework. Expected output: [get: hdfs:*///*user/file: No such file or directory] Actual output: [get: hdfs:/user/file: No such file or directory ................ One question here: can i use * (regex) for matching the path in testHDFSConf.xml file? for example: copyToLocal: *: No such file or directory....will it work with this framework?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12474663/HDFS-1786.patch
          against trunk revision 1085509.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.server.datanode.TestBlockReport
          org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
          org.apache.hadoop.hdfs.TestDFSShell

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/286//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/286//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/286//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12474663/HDFS-1786.patch against trunk revision 1085509. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.datanode.TestTransferRbw org.apache.hadoop.hdfs.TestDFSShell -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/286//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/286//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/286//console This message is automatically generated.
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Uma Maheswara Rao G added a comment -

          I have checked the Test failures which are caused because of null message and has been corrected. All null message expectation failures are from testHDFSConf.xml file. changed the messages in that file.

          Show
          Uma Maheswara Rao G added a comment - I have checked the Test failures which are caused because of null message and has been corrected. All null message expectation failures are from testHDFSConf.xml file. changed the messages in that file.
          Uma Maheswara Rao G made changes -
          Attachment HDFS-1786.patch [ 12474663 ]
          Tsz Wo Nicholas Sze made changes -
          Priority Major [ 3 ] Minor [ 4 ]
          Component/s test [ 12312916 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Uma for signing this up.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Uma for signing this up.
          Uma Maheswara Rao G made changes -
          Assignee Uma Maheswara Rao G [ umamaheswararao ]
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Link This issue relates to HADOOP-7174 [ HADOOP-7174 ]
          Tsz Wo Nicholas Sze created issue -

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development