Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8412

Fix the test failures in HTTPFS: In some tests setReplication called after fs close.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently 2 HTTFS test cases failing due to filesystem open check in fs operations

      This is the JIRA fix these failures.

      Failure seems like
      test case is closing fs first and then doing operation. Ideally such test could pas earlier as dfsClient was just contacting with NN directly. But that particular closed client will not be useful for any other ops like read/write. So, usage should be corrected here IMO.

       fs.close();
          fs.setReplication(path, (short) 2);
      
      1. HDFS-8412-0.patch
        0.8 kB
        Uma Maheswara Rao G

        Issue Links

          Activity

          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -
          Running org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
          Tests run: 46, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 42.801 sec <<<
          FAILURE! - in org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
          
          testOperation[14](org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSy
          stem)  Time elapsed: 0.43 sec  <<< ERROR!
          java.io.IOException: Filesystem closed
                  at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:458)
                  at org.apache.hadoop.hdfs.DFSClient.setReplication(DFSClient.java:1549)
                  at org.apache.hadoop.hdfs.DistributedFileSystem$9.doCall(DistributedFile
          System.java:519)
                  at org.apache.hadoop.hdfs.DistributedFileSystem$9.doCall(DistributedFile
          System.java:515)
                  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkRes
          olver.java:81)
                  at org.apache.hadoop.hdfs.DistributedFileSystem.setReplication(Distribut
          edFileSystem.java:515)
                  at org.apache.hadoop.fs.http.client.BaseTestHttpFSWith.testSetReplicatio
          n(BaseTestHttpFSWith.java:469)
                  at org.apache.hadoop.fs.http.client.BaseTestHttpFSWith.operation(BaseTes
          tHttpFSWith.java:865)
                  at org.apache.hadoop.fs.http.client.BaseTestHttpFSWith.testOperation(Bas
          eTestHttpFSWith.java:917)
          
          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Running org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem Tests run: 46, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 42.801 sec <<< FAILURE! - in org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem testOperation[14](org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSy stem) Time elapsed: 0.43 sec <<< ERROR! java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:458) at org.apache.hadoop.hdfs.DFSClient.setReplication(DFSClient.java:1549) at org.apache.hadoop.hdfs.DistributedFileSystem$9.doCall(DistributedFile System.java:519) at org.apache.hadoop.hdfs.DistributedFileSystem$9.doCall(DistributedFile System.java:515) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkRes olver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.setReplication(Distribut edFileSystem.java:515) at org.apache.hadoop.fs.http.client.BaseTestHttpFSWith.testSetReplicatio n(BaseTestHttpFSWith.java:469) at org.apache.hadoop.fs.http.client.BaseTestHttpFSWith.operation(BaseTes tHttpFSWith.java:865) at org.apache.hadoop.fs.http.client.BaseTestHttpFSWith.testOperation(Bas eTestHttpFSWith.java:917)
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Attached a simple patch for fixing the failures.

          Test results:

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.553 sec - in
           org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem
          Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
          Running org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.453 sec - in
           org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
          Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
          Running org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.924 sec - in
           org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem
          Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
          Running org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.806 sec - in
           org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem
          Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
          
          Results :
          
          Tests run: 184, Failures: 0, Errors: 0, Skipped: 0
          
          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Attached a simple patch for fixing the failures. Test results: ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.553 sec - in org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true Running org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.453 sec - in org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true Running org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.924 sec - in org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true Running org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.806 sec - in org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true Results : Tests run: 184, Failures: 0, Errors: 0, Skipped: 0
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 8s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 23s There were no new javac warning messages.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 27s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 0m 39s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 1m 9s Pre-build of native portion
          +1 hdfs tests 3m 34s Tests passed in hadoop-hdfs-httpfs.
              20m 45s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733116/HDFS-8412-0.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 1ea9014
          hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11003/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11003/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11003/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 23s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 27s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 0m 39s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 1m 9s Pre-build of native portion +1 hdfs tests 3m 34s Tests passed in hadoop-hdfs-httpfs.     20m 45s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733116/HDFS-8412-0.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 1ea9014 hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11003/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11003/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11003/console This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          +1 (non-binding)

          Show
          busbey Sean Busbey added a comment - +1 (non-binding)
          Hide
          hitliuyi Yi Liu added a comment -

          +1, thanks Uma.

          Show
          hitliuyi Yi Liu added a comment - +1, thanks Uma.
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Thanks Sean Busbey, Yi Liu for the reviews. I have just committed this to trunk.

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Thanks Sean Busbey , Yi Liu for the reviews. I have just committed this to trunk.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7851 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7851/)
          HDFS-8412. Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7851 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7851/ ) HDFS-8412 . Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #932 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/932/)
          HDFS-8412. Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #932 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/932/ ) HDFS-8412 . Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/201/)
          HDFS-8412. Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/201/ ) HDFS-8412 . Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2130 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2130/)
          HDFS-8412. Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2130 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2130/ ) HDFS-8412 . Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/190/)
          HDFS-8412. Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/190/ ) HDFS-8412 . Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #200 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/200/)
          HDFS-8412. Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #200 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/200/ ) HDFS-8412 . Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2148 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2148/)
          HDFS-8412. Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2148 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2148/ ) HDFS-8412 . Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. (umamahesh: rev a6af0248e9ec75e8e46ac96593070e0c9841a660) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Uma Maheswara Rao G,

          Thanks for your earlier work here. It seems you only committed to trunk but not branch-2, may I know if there is any concern so it's not done? I can cherry-pick to branch-2 if there is no concern. Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Uma Maheswara Rao G , Thanks for your earlier work here. It seems you only committed to trunk but not branch-2, may I know if there is any concern so it's not done? I can cherry-pick to branch-2 if there is no concern. Thanks.

            People

            • Assignee:
              umamaheswararao Uma Maheswara Rao G
              Reporter:
              umamaheswararao Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development