Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8332

DFS client API calls should check filesystem closed

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Users may need special attention for this change while upgrading to this version. Previously user could call some APIs(example: setReplication) wrongly even after closing the fs object. With this change DFS client will not allow any operations to call on closed fs objects. As calling fs operations on closed fs is not right thing to do, users need to correct the usage if any.
      Show
      Users may need special attention for this change while upgrading to this version. Previously user could call some APIs(example: setReplication) wrongly even after closing the fs object. With this change DFS client will not allow any operations to call on closed fs objects. As calling fs operations on closed fs is not right thing to do, users need to correct the usage if any.

      Description

      I could see listCacheDirectives() and listCachePools() APIs can be called even after the filesystem close. Instead these calls should do checkOpen and throws:

      java.io.IOException: Filesystem closed
      	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:464)
      
      1. HDFS-8332-000.patch
        4 kB
        Rakesh R
      2. HDFS-8332-001.patch
        20 kB
        Rakesh R
      3. HDFS-8332-002.patch
        21 kB
        Rakesh R
      4. HDFS-8332.001.branch-2.patch
        21 kB
        Akira Ajisaka
      5. HDFS-8332-002-Branch-2.patch
        22 kB
        Uma Maheswara Rao G

        Issue Links

          Activity

          Hide
          rakeshr Rakesh R added a comment -

          Thanks Uma Maheswara Rao G, Sean Busbey, Vinayakumar B, Chris Nauroth for the helpful discussions and resolving this.

          Show
          rakeshr Rakesh R added a comment - Thanks Uma Maheswara Rao G , Sean Busbey , Vinayakumar B , Chris Nauroth for the helpful discussions and resolving this.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks, Uma!

          Show
          cnauroth Chris Nauroth added a comment - Thanks, Uma!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2147 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2147/)
          Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2147 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2147/ ) Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #189 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/189/)
          Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #189 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/189/ ) Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2129 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2129/)
          Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2129 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2129/ ) Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #199 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/199/)
          Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #199 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/199/ ) Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #931 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/931/)
          Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #931 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/931/ ) Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #200 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/200/)
          Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #200 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/200/ ) Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7850 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7850/)
          Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7850 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7850/ ) Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk (umamahesh: rev 363c35541d4f9da4974f3e346cb397796173824c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          I have just reverted this from branch-2 and changed CHANGES.txt entry to trunk.

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - I have just reverted this from branch-2 and changed CHANGES.txt entry to trunk.
          Hide
          rakeshr Rakesh R added a comment -

          OK, I got it. +1 to keep this change only in trunk/branc-3.

          Show
          rakeshr Rakesh R added a comment - OK, I got it. +1 to keep this change only in trunk/branc-3.
          Hide
          busbey Sean Busbey added a comment -

          +1 for trunk/branc-3 only. A "Known Issue" note on the next set of branch-2 release notes would be a nice-to-have as well.

          Also, I'd like to suggest that we change pre-commit to trigger hadoop-hdfs-httpfs tests automatically for all hadoop-hdfs patches. We've seen problems like this in the past. hadoop-hdfs-httpfs gets patched so infrequently that it's easy to miss it when a hadoop-hdfs change introduces a test failure. As a practical matter, we might not be able to add those tests until the current HDFS test runs get optimized.

          Leave a note on HADOOP-11929, Allen Wittenauer is already specifying that hadoop-hdfs needs to have hadoop-common built with native bits. Not sure if expanding to "under tests always do this other module if this module changes" will be in scope or a new ticket.

          Show
          busbey Sean Busbey added a comment - +1 for trunk/branc-3 only. A "Known Issue" note on the next set of branch-2 release notes would be a nice-to-have as well. Also, I'd like to suggest that we change pre-commit to trigger hadoop-hdfs-httpfs tests automatically for all hadoop-hdfs patches. We've seen problems like this in the past. hadoop-hdfs-httpfs gets patched so infrequently that it's easy to miss it when a hadoop-hdfs change introduces a test failure. As a practical matter, we might not be able to add those tests until the current HDFS test runs get optimized. Leave a note on HADOOP-11929 , Allen Wittenauer is already specifying that hadoop-hdfs needs to have hadoop-common built with native bits. Not sure if expanding to "under tests always do this other module if this module changes" will be in scope or a new ticket.
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment - - edited

          In this kind of situation, I like to consider if the risks outweigh the benefits. This change isn't an absolute requirement to fix a critical bug or ship a new feature. Considering that, I think a conservative approach would be to re-target this patch to trunk/3.0.0 and revert from branch-2. We can set the incompatible flag and enter a release note for 3.0.0 stating that callers who were dependent on the buggy behavior must fix their code when upgrading. What do others think of this?

          Yes, The strange part here is api calls working even after clientRunning flag set to false I think. So users simply using even closed fs for some operation like setReplication. This we noticed from test cases already.
          Since this is not so critical issue, we can revert from branch-2. I am fine with that. Even though issue comes with wrong usage, some users might have that wrong code already in their app. So, upgrading system would expect code change from users. In this perspective we can mark this as incompatible change. Let's revert from branch-2 and leave it from trunk. What do you think Vinay/Rakesh?

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - - edited In this kind of situation, I like to consider if the risks outweigh the benefits. This change isn't an absolute requirement to fix a critical bug or ship a new feature. Considering that, I think a conservative approach would be to re-target this patch to trunk/3.0.0 and revert from branch-2. We can set the incompatible flag and enter a release note for 3.0.0 stating that callers who were dependent on the buggy behavior must fix their code when upgrading. What do others think of this? Yes, The strange part here is api calls working even after clientRunning flag set to false I think. So users simply using even closed fs for some operation like setReplication. This we noticed from test cases already. Since this is not so critical issue, we can revert from branch-2. I am fine with that. Even though issue comes with wrong usage, some users might have that wrong code already in their app. So, upgrading system would expect code change from users. In this perspective we can mark this as incompatible change. Let's revert from branch-2 and leave it from trunk. What do you think Vinay/Rakesh?
          Hide
          cnauroth Chris Nauroth added a comment -

          This is very strange. It appears that this only "worked" because the RPC proxy is still operable even after calling RPC#stopProxy inside DFSClient#closeConnectionToNamenode. This is not what I would have expected. I thought that this patch by calling checkOpen consistently just changed a failure to give a more descriptive error.

          This is going to be a gray area for compatibility. Code that uses a FileSystem after closing it is incorrect code. Many operations already fail fast. We might be within the letter of the law for the compatibility policy by making this change, but there is an argument that callers could be dependent on the existing bug.

          In this kind of situation, I like to consider if the risks outweigh the benefits. This change isn't an absolute requirement to fix a critical bug or ship a new feature. Considering that, I think a conservative approach would be to re-target this patch to trunk/3.0.0 and revert from branch-2. We can set the incompatible flag and enter a release note for 3.0.0 stating that callers who were dependent on the buggy behavior must fix their code when upgrading. What do others think of this?

          Also, I'd like to suggest that we change pre-commit to trigger hadoop-hdfs-httpfs tests automatically for all hadoop-hdfs patches. We've seen problems like this in the past. hadoop-hdfs-httpfs gets patched so infrequently that it's easy to miss it when a hadoop-hdfs change introduces a test failure. As a practical matter, we might not be able to add those tests until the current HDFS test runs get optimized.

          Show
          cnauroth Chris Nauroth added a comment - This is very strange. It appears that this only "worked" because the RPC proxy is still operable even after calling RPC#stopProxy inside DFSClient#closeConnectionToNamenode . This is not what I would have expected. I thought that this patch by calling checkOpen consistently just changed a failure to give a more descriptive error. This is going to be a gray area for compatibility. Code that uses a FileSystem after closing it is incorrect code. Many operations already fail fast. We might be within the letter of the law for the compatibility policy by making this change, but there is an argument that callers could be dependent on the existing bug. In this kind of situation, I like to consider if the risks outweigh the benefits. This change isn't an absolute requirement to fix a critical bug or ship a new feature. Considering that, I think a conservative approach would be to re-target this patch to trunk/3.0.0 and revert from branch-2. We can set the incompatible flag and enter a release note for 3.0.0 stating that callers who were dependent on the buggy behavior must fix their code when upgrading. What do others think of this? Also, I'd like to suggest that we change pre-commit to trigger hadoop-hdfs-httpfs tests automatically for all hadoop-hdfs patches. We've seen problems like this in the past. hadoop-hdfs-httpfs gets patched so infrequently that it's easy to miss it when a hadoop-hdfs change introduces a test failure. As a practical matter, we might not be able to add those tests until the current HDFS test runs get optimized.
          Hide
          busbey Sean Busbey added a comment -

          Being incompatible and breaking some tests are two different problems. It's true that just because tests fail it does not mean a change is incompatible. However, this change is still incompatible.

          I agree that this change is good and one we should do. However, it will break some downstream user code that worked before. A good sign of this is that it broke some code maintained by the Hadoop community, ostensibly those most familiar with how HDFS works.

          It's important that we properly document when we change things in a way that might break downstream users (wether or not they were doing the correct thing before) so that they can make appropriate adjustments before upgrading, especially when those changes are in a minor version.

          Show
          busbey Sean Busbey added a comment - Being incompatible and breaking some tests are two different problems. It's true that just because tests fail it does not mean a change is incompatible. However, this change is still incompatible. The FileSystem specification doesn't say that all operations must fail after a close Neither does the javadoc on FileSystem.close (though it does imply it) The specification specifically says that HDFS' behavior is correct I agree that this change is good and one we should do. However, it will break some downstream user code that worked before. A good sign of this is that it broke some code maintained by the Hadoop community, ostensibly those most familiar with how HDFS works. It's important that we properly document when we change things in a way that might break downstream users (wether or not they were doing the correct thing before) so that they can make appropriate adjustments before upgrading, especially when those changes are in a minor version.
          Hide
          rakeshr Rakesh R added a comment -

          Thanks Vinayakumar B and Uma Mahesh for finding out the root cause.

          IMO, incompatible change is only when the user's valid code fails. Not when error code fails after change.

          True. +1(non-binding) from me

          Show
          rakeshr Rakesh R added a comment - Thanks Vinayakumar B and Uma Mahesh for finding out the root cause. IMO, incompatible change is only when the user's valid code fails. Not when error code fails after change. True. +1(non-binding) from me
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          After correcting the usage, please find the test results:

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.553 sec - in
           org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem
          Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
          Running org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.453 sec - in
           org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
          Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
          Running org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.924 sec - in
           org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem
          Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
          Running org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.806 sec - in
           org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem
          Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
          
          Results :
          
          Tests run: 184, Failures: 0, Errors: 0, Skipped: 0
          
          Show
          umamaheswararao Uma Maheswara Rao G added a comment - After correcting the usage, please find the test results: ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.553 sec - in org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true Running org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.453 sec - in org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true Running org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.924 sec - in org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true Running org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.806 sec - in org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true Results : Tests run: 184, Failures: 0, Errors: 0, Skipped: 0
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Yeah, you are right Vinay. I had filed a jira for it. HDFS-8412
          Just correcting the usage of test should be fine.

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Yeah, you are right Vinay. I had filed a jira for it. HDFS-8412 Just correcting the usage of test should be fine.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Hi Sean Busbey,
          I understand that Tests started failing after the commit, that doesnt mean that this Jira change is incompatible.
          Failure was due to the error in test, which was calling setReplication(..) even after fs.close(). of-course it was passing due to this bug.
          Below code is from BaseTestHttpFSWith.java

            private void testSetReplication() throws Exception {
              FileSystem fs = FileSystem.get(getProxiedFSConf());
              Path path = new Path(getProxiedFSTestDir(), "foo.txt");
              OutputStream os = fs.create(path);
              os.write(1);
              os.close();
              fs.close();
              fs.setReplication(path, (short) 2);
          
              fs = getHttpFSFileSystem();
              fs.setReplication(path, (short) 1);
              fs.close();
          
              fs = FileSystem.get(getProxiedFSConf());
              FileStatus status1 = fs.getFileStatus(path);
              fs.close();
              Assert.assertEquals(status1.getReplication(), (short) 1);
            }

          IMO, incompatible change is only when the user's valid code fails. Not when error code fails after change.

          Agree?

          Show
          vinayrpet Vinayakumar B added a comment - Hi Sean Busbey , I understand that Tests started failing after the commit, that doesnt mean that this Jira change is incompatible. Failure was due to the error in test, which was calling setReplication(..) even after fs.close() . of-course it was passing due to this bug. Below code is from BaseTestHttpFSWith.java private void testSetReplication() throws Exception { FileSystem fs = FileSystem.get(getProxiedFSConf()); Path path = new Path(getProxiedFSTestDir(), "foo.txt" ); OutputStream os = fs.create(path); os.write(1); os.close(); fs.close(); fs.setReplication(path, ( short ) 2); fs = getHttpFSFileSystem(); fs.setReplication(path, ( short ) 1); fs.close(); fs = FileSystem.get(getProxiedFSConf()); FileStatus status1 = fs.getFileStatus(path); fs.close(); Assert.assertEquals(status1.getReplication(), ( short ) 1); } IMO, incompatible change is only when the user's valid code fails. Not when error code fails after change. Agree?
          Hide
          rakeshr Rakesh R added a comment -

          Sean Busbey Do you have the test run logs, it would be very helpful in understanding the background and do more analysis. Thanks!

          Show
          rakeshr Rakesh R added a comment - Sean Busbey Do you have the test run logs, it would be very helpful in understanding the background and do more analysis. Thanks!
          Hide
          busbey Sean Busbey added a comment -

          also, please release note this as an incompatible change in behavior.

          Show
          busbey Sean Busbey added a comment - also, please release note this as an incompatible change in behavior.
          Hide
          busbey Sean Busbey added a comment -

          According to git-bisect, this is the commit that started failing the following tests in hadoop-hdfs-httpfs on trunk:

          • TestHttpFSFWithSWebhdfsFileSystem
          • TestHttpFSWithHttpFSFileSystem
          • TestHttpFSFWithWebhdfsFileSystem
          $ git bisect start trunk b46c2bb51ae524e6640756620f70e5925cda7592
          Bisecting: 272 revisions left to test after this (roughly 8 steps)
          [baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6] HDFS-4448. Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh)
          $ git bisect run mvn -Dtest=TestHttpFSF*,TestHttpFSWithHttpFSFileSystem clean package
          ...SNIP...
          e16f4b7f70b8675760cf5aaa471dfe29d48041e6 is the first bad commit
          commit e16f4b7f70b8675760cf5aaa471dfe29d48041e6
          Author: Uma Maheswara Rao G <umamahesh@apache.org>
          Date:   Fri May 8 12:26:47 2015 +0530
          
              HDFS-8332. DFS client API calls should check filesystem closed. Contributed by Rakesh R.
          
          :040000 040000 db7a6b4555c1bd18e8fe0a97a6689f7cf9ce15ec f9e0818f6198fbc0ac94b2d82bef7f065a90cc03 M      hadoop-hdfs-project
          bisect run success
          
          Show
          busbey Sean Busbey added a comment - According to git-bisect, this is the commit that started failing the following tests in hadoop-hdfs-httpfs on trunk: TestHttpFSFWithSWebhdfsFileSystem TestHttpFSWithHttpFSFileSystem TestHttpFSFWithWebhdfsFileSystem $ git bisect start trunk b46c2bb51ae524e6640756620f70e5925cda7592 Bisecting: 272 revisions left to test after this (roughly 8 steps) [baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6] HDFS-4448. Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) $ git bisect run mvn -Dtest=TestHttpFSF*,TestHttpFSWithHttpFSFileSystem clean package ...SNIP... e16f4b7f70b8675760cf5aaa471dfe29d48041e6 is the first bad commit commit e16f4b7f70b8675760cf5aaa471dfe29d48041e6 Author: Uma Maheswara Rao G <umamahesh@apache.org> Date: Fri May 8 12:26:47 2015 +0530 HDFS-8332. DFS client API calls should check filesystem closed. Contributed by Rakesh R. :040000 040000 db7a6b4555c1bd18e8fe0a97a6689f7cf9ce15ec f9e0818f6198fbc0ac94b2d82bef7f065a90cc03 M hadoop-hdfs-project bisect run success
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #189 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/189/)
          HDFS-8332. DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #189 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/189/ ) HDFS-8332 . DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2137 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2137/)
          HDFS-8332. DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2137 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2137/ ) HDFS-8332 . DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java
          Hide
          cnauroth Chris Nauroth added a comment -

          Rakesh, thank you for the patch. Thanks also to Uma and Akira for finishing off the review and commit.

          Show
          cnauroth Chris Nauroth added a comment - Rakesh, thank you for the patch. Thanks also to Uma and Akira for finishing off the review and commit.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #179 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/179/)
          HDFS-8332. DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #179 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/179/ ) HDFS-8332 . DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2119 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2119/)
          HDFS-8332. DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2119 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2119/ ) HDFS-8332 . DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/190/)
          HDFS-8332. DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/190/ ) HDFS-8332 . DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #921 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/921/)
          HDFS-8332. DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #921 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/921/ ) HDFS-8332 . DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          I missed your comment Akira Ajisaka. I merge it to branch-2 as well. Thankyou

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - I missed your comment Akira Ajisaka . I merge it to branch-2 as well. Thankyou
          Hide
          rakeshr Rakesh R added a comment -

          Thank you Uma Maheswara Rao G for reviewing and committing the patch. Also, thank you Akira Ajisaka for the help.

          Show
          rakeshr Rakesh R added a comment - Thank you Uma Maheswara Rao G for reviewing and committing the patch. Also, thank you Akira Ajisaka for the help.
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          minor conflicts to branch-2. What I committed to branch-2.

          I have just committed the patch to trunk and branch-2.
          Thanks a lot, Rakesh!

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - minor conflicts to branch-2. What I committed to branch-2. I have just committed the patch to trunk and branch-2. Thanks a lot, Rakesh!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Uma Maheswara Rao G, how is the issue going? Looks like it stops at cherry-picking the commit to branch-2. I've prepared a patch for branch-2. Hope it helps.

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Uma Maheswara Rao G , how is the issue going? Looks like it stops at cherry-picking the commit to branch-2. I've prepared a patch for branch-2. Hope it helps.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7771 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7771/)
          HDFS-8332. DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7771 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7771/ ) HDFS-8332 . DFS client API calls should check filesystem closed. Contributed by Rakesh R. (umamahesh: rev e16f4b7f70b8675760cf5aaa471dfe29d48041e6) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgradeRollback.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          +1 committing it.

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - +1 committing it.
          Hide
          rakeshr Rakesh R added a comment -

          Jenkins complains about few checkstyle issues but those are unrelated to my patch. Kindly review. Thanks!

          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field.
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field.
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,218 lines (max allowed is 2,000).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field.
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,218 lines (max allowed is 2,000).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,241 lines (max allowed is 2,000).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field.
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,218 lines (max allowed is 2,000).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,241 lines (max allowed is 2,000).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,241 lines (max allowed is 2,000).
          
          Show
          rakeshr Rakesh R added a comment - Jenkins complains about few checkstyle issues but those are unrelated to my patch. Kindly review. Thanks! ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,218 lines (max allowed is 2,000). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,218 lines (max allowed is 2,000). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,241 lines (max allowed is 2,000). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,218 lines (max allowed is 2,000). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,241 lines (max allowed is 2,000). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,241 lines (max allowed is 2,000).
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 52s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 11s The applied patch generated 14 new checkstyle issues (total was 734, now 735).
          +1 whitespace 0m 5s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 3s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 17s Pre-build of native portion
          -1 hdfs tests 167m 33s Tests failed in hadoop-hdfs.
              210m 49s  



          Reason Tests
          Failed unit tests hadoop.tracing.TestTraceAdmin



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731054/HDFS-8332-002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4c7b9b6
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10840/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10840/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10840/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10840/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 11s The applied patch generated 14 new checkstyle issues (total was 734, now 735). +1 whitespace 0m 5s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 3s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 17s Pre-build of native portion -1 hdfs tests 167m 33s Tests failed in hadoop-hdfs.     210m 49s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731054/HDFS-8332-002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4c7b9b6 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10840/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10840/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10840/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10840/console This message was automatically generated.
          Hide
          rakeshr Rakesh R added a comment -

          Attached another patch fixing the test case failure TestRollingUpgradeRollback.testRollbackWithHAQJM

          java.io.IOException: Filesystem closed
          	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:458)
          	at org.apache.hadoop.hdfs.DFSClient.rollingUpgrade(DFSClient.java:2612)
          

          Again there are few checkstyle warnings, but those are unrelated to my patch.

          Show
          rakeshr Rakesh R added a comment - Attached another patch fixing the test case failure TestRollingUpgradeRollback.testRollbackWithHAQJM java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:458) at org.apache.hadoop.hdfs.DFSClient.rollingUpgrade(DFSClient.java:2612) Again there are few checkstyle warnings, but those are unrelated to my patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 59s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 46s There were no new javac warning messages.
          +1 javadoc 9m 46s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 14s The applied patch generated 9 new checkstyle issues (total was 616, now 617).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 19s Pre-build of native portion
          -1 hdfs tests 168m 4s Tests failed in hadoop-hdfs.
              211m 59s  



          Reason Tests
          Failed unit tests hadoop.tracing.TestTraceAdmin
            hadoop.hdfs.TestRollingUpgradeRollback



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730900/HDFS-8332-001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 185e63a
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10834/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10834/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10834/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10834/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 59s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 46s There were no new javac warning messages. +1 javadoc 9m 46s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 14s The applied patch generated 9 new checkstyle issues (total was 616, now 617). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 19s Pre-build of native portion -1 hdfs tests 168m 4s Tests failed in hadoop-hdfs.     211m 59s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin   hadoop.hdfs.TestRollingUpgradeRollback Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730900/HDFS-8332-001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 185e63a checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10834/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10834/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10834/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10834/console This message was automatically generated.
          Hide
          rakeshr Rakesh R added a comment -

          Attached another patch covers the following APIs other than listCacheDirectives and listCachePools. Please have a look at it. Thanks!

          getBlockSize
          getServerDefaults
          reportBadBlocks
          getBlockLocations
          getBlockStorageLocations
          createSymlink
          setReplication
          setStoragePolicy
          getStoragePolicies
          setSafeMode
          refreshNodes
          metaSave
          setBalancerBandwidth
          finalizeUpgrade
          rollingUpgrade
          getInotifyEventStream
          getInotifyEventStream(x)
          saveNamespace
          rollEdits
          restoreFailedStorage
          getContentSummary
          setQuota
          setQuotaByStorageType
          
          Show
          rakeshr Rakesh R added a comment - Attached another patch covers the following APIs other than listCacheDirectives and listCachePools . Please have a look at it. Thanks! getBlockSize getServerDefaults reportBadBlocks getBlockLocations getBlockStorageLocations createSymlink setReplication setStoragePolicy getStoragePolicies setSafeMode refreshNodes metaSave setBalancerBandwidth finalizeUpgrade rollingUpgrade getInotifyEventStream getInotifyEventStream(x) saveNamespace rollEdits restoreFailedStorage getContentSummary setQuota setQuotaByStorageType
          Hide
          rakeshr Rakesh R added a comment -

          Thanks a lot Chris Nauroth for the comments and pointing out the list of APIs to be covered. I'll try to do the modifications as part of this jira.

          Show
          rakeshr Rakesh R added a comment - Thanks a lot Chris Nauroth for the comments and pointing out the list of APIs to be covered. I'll try to do the modifications as part of this jira.
          Hide
          cnauroth Chris Nauroth added a comment -

          Hello Rakesh R. Nice find! Thank you for providing a patch. It looks good. I verified locally that the tests pass, and I agree that the checkstyle warnings are unrelated.

          We're very inconsistent about this logic in DFSClient, even looking beyond these 2 methods. See below for a list of the methods I spotted that don't call checkOpen.

          Are you interested in providing a patch that covers all of these, or are you specifically interested in patching just listCacheDirectives and listCachePools? I'm fine with either approach. If you just want get the current patch committed, then I can file a separate jira for a comprehensive fix across all of these methods. Please let me know how you'd like to proceed.

          getBlockSize
          getServerDefaults
          reportBadBlocks
          getLocatedBlocks
          getBlockLocations
          getBlockStorageLocations
          createSymlink
          getLinkTarget
          setReplication
          setStoragePolicy
          getStoragePolicies
          setSafeMode
          listCacheDirectives
          refreshNodes
          metaSave
          setBalancerBandwidth
          finalizeUpgrade
          getInotifyEventStream
          
          Show
          cnauroth Chris Nauroth added a comment - Hello Rakesh R . Nice find! Thank you for providing a patch. It looks good. I verified locally that the tests pass, and I agree that the checkstyle warnings are unrelated. We're very inconsistent about this logic in DFSClient , even looking beyond these 2 methods. See below for a list of the methods I spotted that don't call checkOpen . Are you interested in providing a patch that covers all of these, or are you specifically interested in patching just listCacheDirectives and listCachePools ? I'm fine with either approach. If you just want get the current patch committed, then I can file a separate jira for a comprehensive fix across all of these methods. Please let me know how you'd like to proceed. getBlockSize getServerDefaults reportBadBlocks getLocatedBlocks getBlockLocations getBlockStorageLocations createSymlink getLinkTarget setReplication setStoragePolicy getStoragePolicies setSafeMode listCacheDirectives refreshNodes metaSave setBalancerBandwidth finalizeUpgrade getInotifyEventStream
          Hide
          rakeshr Rakesh R added a comment -

          Following checkstyle warnings are unrelated to my patch.

          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field.
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field.
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85).
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,218 lines (max allowed is 2,000).
          

          Also, jenkins complains about few test case failures. It looks like these are unrelated to the patch.

          Show
          rakeshr Rakesh R added a comment - Following checkstyle warnings are unrelated to my patch. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:711:41: 'blocks' hides a field. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java:717: Line is longer than 80 characters (found 85). ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:1: File length is 3,218 lines (max allowed is 2,000). Also, jenkins complains about few test case failures. It looks like these are unrelated to the patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 49s There were no new javac warning messages.
          +1 javadoc 10m 4s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 23s The applied patch generated 5 new checkstyle issues (total was 498, now 499).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 39s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 3m 8s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 23s Pre-build of native portion
          -1 hdfs tests 99m 19s Tests failed in hadoop-hdfs.
              143m 51s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            hadoop.hdfs.TestDFSFinalize
          Timed out tests org.apache.hadoop.hdfs.TestFSOutputSummer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730775/HDFS-8332-000.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a583a40
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10831/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10831/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10831/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10831/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 49s There were no new javac warning messages. +1 javadoc 10m 4s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 23s The applied patch generated 5 new checkstyle issues (total was 498, now 499). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 3m 8s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 23s Pre-build of native portion -1 hdfs tests 99m 19s Tests failed in hadoop-hdfs.     143m 51s   Reason Tests Failed unit tests hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   hadoop.hdfs.TestDFSFinalize Timed out tests org.apache.hadoop.hdfs.TestFSOutputSummer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730775/HDFS-8332-000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a583a40 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10831/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10831/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10831/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10831/console This message was automatically generated.
          Hide
          rakeshr Rakesh R added a comment -

          Attached patch where it checks the file system is open to perform the cache operations.

          Show
          rakeshr Rakesh R added a comment - Attached patch where it checks the file system is open to perform the cache operations.

            People

            • Assignee:
              rakeshr Rakesh R
              Reporter:
              rakeshr Rakesh R
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development