Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2505

Add a test to verify getFileChecksum works with ViewFS

    Details

    • Type: Test Test
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.0-alpha, 3.0.0
    • Component/s: test
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Please refer to HADOOP-7770. getFileChecksum was failing on files such as /tmp/someFile, but working fine for /someDir/someFile.
      The fix is in HADOOP-7770 but we should have a test in HDFS which checks this functionality (this test will fail until HADOOP-7770 is checked in)

      1. HDFS-2505.patch
        4 kB
        Ravi Prakash
      2. HDFS-2505.patch
        2 kB
        Ravi Prakash
      3. HDFS-2505.patch
        2 kB
        Ravi Prakash
      4. HDFS-2505.patch
        1 kB
        Ravi Prakash

        Issue Links

          Activity

          Hide
          Ravi Prakash added a comment -

          This patch contains the test. It should fail until HADOOP-7770 goes in

          Show
          Ravi Prakash added a comment - This patch contains the test. It should fail until HADOOP-7770 goes in
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12500896/HDFS-2505.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestBalancerBandwidth
          org.apache.hadoop.fs.viewfs.TestViewFsFileStatusHdfs

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1461//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1461//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500896/HDFS-2505.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.fs.viewfs.TestViewFsFileStatusHdfs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1461//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1461//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          This patch couldn't possibly fail any of those tests because it doesn't have any src/main code. Only src/test code.

          Show
          Ravi Prakash added a comment - This patch couldn't possibly fail any of those tests because it doesn't have any src/main code. Only src/test code.
          Hide
          Ravi Prakash added a comment -

          Related to HADOOP-7770. This patch adds a test in HDFS related to something fixed in common

          Show
          Ravi Prakash added a comment - Related to HADOOP-7770 . This patch adds a test in HDFS related to something fixed in common
          Hide
          Jitendra Nath Pandey added a comment -

          Is it possible to have a similar test for viewFS and Hdfs?

          Show
          Jitendra Nath Pandey added a comment - Is it possible to have a similar test for viewFS and Hdfs?
          Hide
          Ravi Prakash added a comment -

          Thanks Jitendra! Unit tests were added as part of HADOOP-7770 for the changes made in ViewFS.java and ViewFileSystem.java. There weren't any changes in HDFS code. Could you please specify what method / interface you'd like tested?
          Would this test not make sense, because it might break without any changes in HDFS code (if there are changes in the Hadoop getFileChecksum code)? If so, we can close this JIRA as invalid. Or do we WANT that check? So that if Hadoop Common changes things, HDFS tests break and we are notified to fix the issue in HDFS also?

          Show
          Ravi Prakash added a comment - Thanks Jitendra! Unit tests were added as part of HADOOP-7770 for the changes made in ViewFS.java and ViewFileSystem.java. There weren't any changes in HDFS code. Could you please specify what method / interface you'd like tested? Would this test not make sense, because it might break without any changes in HDFS code (if there are changes in the Hadoop getFileChecksum code)? If so, we can close this JIRA as invalid. Or do we WANT that check? So that if Hadoop Common changes things, HDFS tests break and we are notified to fix the issue in HDFS also?
          Hide
          Jitendra Nath Pandey added a comment -

          ViewFS also has getFileChecksum method similar to ViewFileSystem. Since this patch adds a test for ViewFileSystem, probably a similar test in ViewFS also makes sense. By Hdfs, I actually meant Hdfs.java which is an AbstractFileSystem implementation unlike DistributedFileSystem (which is FileSystem). If you use ViewFS for hdfs, it will use Hdfs underneath, therefore you shouldn't have to do anything explicit for Hdfs.

          Show
          Jitendra Nath Pandey added a comment - ViewFS also has getFileChecksum method similar to ViewFileSystem. Since this patch adds a test for ViewFileSystem, probably a similar test in ViewFS also makes sense. By Hdfs, I actually meant Hdfs.java which is an AbstractFileSystem implementation unlike DistributedFileSystem (which is FileSystem). If you use ViewFS for hdfs, it will use Hdfs underneath, therefore you shouldn't have to do anything explicit for Hdfs.
          Hide
          Jitendra Nath Pandey added a comment -

          The jira is just adding a test. It should not be marked critical.

          Show
          Jitendra Nath Pandey added a comment - The jira is just adding a test. It should not be marked critical.
          Hide
          Tsz Wo Nicholas Sze added a comment -
          • The toString() below should be removed.
            +    ConfigUtil.addLink(conf, "/tmp", new URI(fHdfs.getUri().toString() + "/tmp"));
            

            Also, how about use different names for "/tmp", say "/hdfstmp" and "/viewfstmp"?

          • Instead of checking null, how about get a file checksum from DistributedFileSystem for the same file and then check equality?
          Show
          Tsz Wo Nicholas Sze added a comment - The toString() below should be removed. + ConfigUtil.addLink(conf, "/tmp" , new URI(fHdfs.getUri().toString() + "/tmp" )); Also, how about use different names for "/tmp", say "/hdfstmp" and "/viewfstmp"? Instead of checking null, how about get a file checksum from DistributedFileSystem for the same file and then check equality?
          Hide
          Ravi Prakash added a comment -

          Thanks for the review Nicholas! I've updated the patch with your suggestions. Could you please review and commit it?

          Show
          Ravi Prakash added a comment - Thanks for the review Nicholas! I've updated the patch with your suggestions. Could you please review and commit it?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Ravi,

          It looks like the that the new test depends on the file created by testFileStatusSerialziation. I think it is better to either create another file in the new test or combine two tests.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Ravi, It looks like the that the new test depends on the file created by testFileStatusSerialziation. I think it is better to either create another file in the new test or combine two tests.
          Hide
          Ravi Prakash added a comment -

          Thanks for that suggestion Nicholas! I am now creating a new file. Could you please review and commit this updated patch?

          Show
          Ravi Prakash added a comment - Thanks for that suggestion Nicholas! I am now creating a new file. Could you please review and commit this updated patch?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hey Ravi, I actually don't understand why the test in the new patch works since it create a file in /tmp but the link target is /hdfstmp. Is it still using some side-effect from testFileStatusSerialziation? If you want to have two test, please make sure the new test can run without the existing test.

          Show
          Tsz Wo Nicholas Sze added a comment - Hey Ravi, I actually don't understand why the test in the new patch works since it create a file in /tmp but the link target is /hdfstmp. Is it still using some side-effect from testFileStatusSerialziation? If you want to have two test, please make sure the new test can run without the existing test.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521594/HDFS-2505.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2204//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2204//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521594/HDFS-2505.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2204//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2204//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521604/HDFS-2505.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2206//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2206//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521604/HDFS-2505.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2206//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2206//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          Hey Ravi, I actually don't understand why the test in the new patch works since it create a file in /tmp but the link target is /hdfstmp. Is it still using some side-effect from testFileStatusSerialziation? If you want to have two test, please make sure the new test can run without the existing test.

          Thanks a lot Nicholas! That was a great catch. ViewFS is still confusing to me. I've added another test to check that checksums of different files are different. The earlier patch was working because /tmp had been setup in the ViewFS mount tables in the first test, and this was being resolved as /tmp in HDFS in the second test.
          Apparently FileSystem.get caches the conf and so setting it a second time is useless. By that time viewfs:// has already been configured and can not be changed (from what I can tell).
          This new patch sets up ViewFS in the @BeforeTest method. Now the tests work independently as well as together

          Show
          Ravi Prakash added a comment - Hey Ravi, I actually don't understand why the test in the new patch works since it create a file in /tmp but the link target is /hdfstmp. Is it still using some side-effect from testFileStatusSerialziation? If you want to have two test, please make sure the new test can run without the existing test. Thanks a lot Nicholas! That was a great catch. ViewFS is still confusing to me. I've added another test to check that checksums of different files are different. The earlier patch was working because /tmp had been setup in the ViewFS mount tables in the first test, and this was being resolved as /tmp in HDFS in the second test. Apparently FileSystem.get caches the conf and so setting it a second time is useless. By that time viewfs:// has already been configured and can not be changed (from what I can tell). This new patch sets up ViewFS in the @BeforeTest method. Now the tests work independently as well as together
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521684/HDFS-2505.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2213//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2213//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521684/HDFS-2505.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2213//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2213//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the latest patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the latest patch looks good.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2020 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2020/)
          HDFS-2505. Add a test to verify getFileChecksum(..) with ViewFS. Contributed by Ravi Prakash (Revision 1310522)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310522
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsFileStatusHdfs.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2020 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2020/ ) HDFS-2505 . Add a test to verify getFileChecksum(..) with ViewFS. Contributed by Ravi Prakash (Revision 1310522) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310522 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsFileStatusHdfs.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2095 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2095/)
          HDFS-2505. Add a test to verify getFileChecksum(..) with ViewFS. Contributed by Ravi Prakash (Revision 1310522)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310522
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsFileStatusHdfs.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2095 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2095/ ) HDFS-2505 . Add a test to verify getFileChecksum(..) with ViewFS. Contributed by Ravi Prakash (Revision 1310522) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310522 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsFileStatusHdfs.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2033 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2033/)
          HDFS-2505. Add a test to verify getFileChecksum(..) with ViewFS. Contributed by Ravi Prakash (Revision 1310522)

          Result = ABORTED
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310522
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsFileStatusHdfs.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2033 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2033/ ) HDFS-2505 . Add a test to verify getFileChecksum(..) with ViewFS. Contributed by Ravi Prakash (Revision 1310522) Result = ABORTED szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310522 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsFileStatusHdfs.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Ravi!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Ravi!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1007 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1007/)
          HDFS-2505. Add a test to verify getFileChecksum(..) with ViewFS. Contributed by Ravi Prakash (Revision 1310522)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310522
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsFileStatusHdfs.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1007 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1007/ ) HDFS-2505 . Add a test to verify getFileChecksum(..) with ViewFS. Contributed by Ravi Prakash (Revision 1310522) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310522 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsFileStatusHdfs.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1042 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1042/)
          HDFS-2505. Add a test to verify getFileChecksum(..) with ViewFS. Contributed by Ravi Prakash (Revision 1310522)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310522
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsFileStatusHdfs.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1042 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1042/ ) HDFS-2505 . Add a test to verify getFileChecksum(..) with ViewFS. Contributed by Ravi Prakash (Revision 1310522) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310522 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsFileStatusHdfs.java

            People

            • Assignee:
              Ravi Prakash
              Reporter:
              Ravi Prakash
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development