Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1304

There is no unit test for HftpFileSystem.open(..)

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HftpFileSystem.open(..) first opens an URL connection to namenode's FileDataServlet and then is redirected to datanode's StreamFile servlet. Such redirection does not work in the unit test environment because the redirect URL uses real hostname instead of localhost.

      One way to get around it is to use fault-injection in order to replace the real hostname with localhost.

      1. h1304_20100922.patch
        9 kB
        Tsz Wo Nicholas Sze
      2. h1304_20100921.patch
        9 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          h1304_20100921.patch: fault injection unit test.

          Show
          Tsz Wo Nicholas Sze added a comment - h1304_20100921.patch: fault injection unit test.
          Hide
          Konstantin Boudnik added a comment -

          I have compiled and ran the test without any issues.
          A couple of nits:

          • please have messages for the asserts
          • remove commented code
          • these imports aren't used anywhere and can be removed as well
            import org.apache.hadoop.fs.Path;
            import org.apache.hadoop.hdfs.DFSClient;
            import org.apache.hadoop.hdfs.DFSTestUtil;
            import org.apache.hadoop.hdfs.DistributedFileSystem;
            import org.apache.hadoop.hdfs.HftpFileSystem;
            import org.apache.hadoop.hdfs.MiniDFSCluster;
            

            Looks good otherwise.

          Also, I have discussed with Nicholas why he didn't use Mockito to simulate the server side and his point was that with MiniDFSCluster he's getting a real webserver without any extra effort to emulate it, which makes sense.

          Show
          Konstantin Boudnik added a comment - I have compiled and ran the test without any issues. A couple of nits: please have messages for the asserts remove commented code these imports aren't used anywhere and can be removed as well import org.apache.hadoop.fs.Path; import org.apache.hadoop.hdfs.DFSClient; import org.apache.hadoop.hdfs.DFSTestUtil; import org.apache.hadoop.hdfs.DistributedFileSystem; import org.apache.hadoop.hdfs.HftpFileSystem; import org.apache.hadoop.hdfs.MiniDFSCluster; Looks good otherwise. Also, I have discussed with Nicholas why he didn't use Mockito to simulate the server side and his point was that with MiniDFSCluster he's getting a real webserver without any extra effort to emulate it, which makes sense.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Cos for reviewing it.

          h1304_20100922.patch:

          > please have messages for the asserts
          The assert messages will be printed only when the test failed. In such case, we can check the stack trace.

          > remove commented code
          Moved the commented codes to a method.

          > these imports aren't used anywhere and can be removed as well ...
          Removed.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Cos for reviewing it. h1304_20100922.patch: > please have messages for the asserts The assert messages will be printed only when the test failed. In such case, we can check the stack trace. > remove commented code Moved the commented codes to a method. > these imports aren't used anywhere and can be removed as well ... Removed.
          Hide
          Tsz Wo Nicholas Sze added a comment -
               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
               [exec]     +1 system tests framework.  The patch passed system tests framework compile.
               [exec] 
          

          I also have run the new unit test. It works fine.

          Show
          Tsz Wo Nicholas Sze added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 9 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system tests framework. The patch passed system tests framework compile. [exec] I also have run the new unit test. It works fine.
          Hide
          Konstantin Boudnik added a comment -

          +1 patch looks good.

          Show
          Konstantin Boudnik added a comment - +1 patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this.

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #396 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/396/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #396 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/396/ )

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development