Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4774

Backport HDFS-4525 'Provide an API for knowing whether file is closed or not' to branch-1

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: hdfs-client, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-4525 compliments lease recovery API which allows user to know whether the recovery has completed.

      This JIRA backports the API to branch-1.

      1. 4774-v3b.txt
        6 kB
        Tsz Wo Nicholas Sze
      2. 4774-v3.txt
        6 kB
        Ted Yu
      3. 4774-v2.txt
        6 kB
        Ted Yu
      4. 4774.txt
        5 kB
        Ted Yu

        Issue Links

          Activity

          Enis Soztutar made changes -
          Link This issue is related to HBASE-8670 [ HBASE-8670 ]
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 1.2.0 [ 12321657 ]
          Resolution Fixed [ 1 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Ted!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Ted!
          Tsz Wo Nicholas Sze made changes -
          Attachment 4774-v3b.txt [ 12581599 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          It is better to update ClientProtocol javadoc for the versionID.

          -   * Added concat() - since this an addition of method, it does not break
          +   * Added concat() and isFileClosed()
          +   * - since this is addition of methods, it does not break
          

          4774-v3b.txt

          Show
          Tsz Wo Nicholas Sze added a comment - It is better to update ClientProtocol javadoc for the versionID. - * Added concat() - since this an addition of method, it does not break + * Added concat() and isFileClosed() + * - since this is addition of methods, it does not break 4774-v3b.txt
          Hide
          Tsz Wo Nicholas Sze added a comment -
               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     -1 findbugs.  The patch appears to introduce 19 new Findbugs (version 1.3.9) warnings.
          

          The findbugs warnings are not related to the patch.

          Show
          Tsz Wo Nicholas Sze added a comment - [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 19 new Findbugs (version 1.3.9) warnings. The findbugs warnings are not related to the patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          TestFsShellReturnCode also fails in machine with "chown: admin: illegal user name". It has a hardcoded user "admin" and it may fail if "admin" is not a valid user in the local OS.

          TestNNThroughputBenchmark and TestBalancerWithNodeGroup do not fail in my machine.

          TestRumenJobTraces and TestFairSchedulerPoolNames are not related to HDFS at all.

          Show
          Tsz Wo Nicholas Sze added a comment - TestFsShellReturnCode also fails in machine with "chown: admin: illegal user name". It has a hardcoded user "admin" and it may fail if "admin" is not a valid user in the local OS. TestNNThroughputBenchmark and TestBalancerWithNodeGroup do not fail in my machine. TestRumenJobTraces and TestFairSchedulerPoolNames are not related to HDFS at all.
          Hide
          Ted Yu added a comment -

          Without my patch, the same tests failed:

              [junit] Test org.apache.hadoop.fs.TestFsShellReturnCode FAILED
              [junit] Test org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark FAILED
              [junit] Test org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup FAILED
              [junit] Test org.apache.hadoop.tools.rumen.TestRumenJobTraces FAILED
              [junit] Test org.apache.hadoop.mapred.TestFairSchedulerPoolNames FAILED
          
          Show
          Ted Yu added a comment - Without my patch, the same tests failed: [junit] Test org.apache.hadoop.fs.TestFsShellReturnCode FAILED [junit] Test org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark FAILED [junit] Test org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup FAILED [junit] Test org.apache.hadoop.tools.rumen.TestRumenJobTraces FAILED [junit] Test org.apache.hadoop.mapred.TestFairSchedulerPoolNames FAILED
          Hide
          Ted Yu added a comment -

          I got the following test failures running test suite:

              [junit] Test org.apache.hadoop.fs.TestFsShellReturnCode FAILED
              [junit] Test org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark FAILED
              [junit] Test org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup FAILED
              [junit] Test org.apache.hadoop.tools.rumen.TestRumenJobTraces FAILED
              [junit] Test org.apache.hadoop.mapred.TestFairSchedulerPoolNames FAILED
          

          In TEST-org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark.txt, I found:

          2013-05-02 03:14:31,718 ERROR namenode.NNThroughputBenchmark (NNThroughputBenchmark.java:runBenchmark(1206)) - java.io.IOException: NNThroughputBenchmark: cannot mkdir build/test/dfs/hosts/exclude
                  at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.<init>(NNThroughputBenchmark.java:106)
                  at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1163)
                  at org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark.testNNThroughput(TestNNThroughputBenchmark.java:38)
          

          I am not sure if the test failures were related to my patch.

          Here is the OS:

          Linux 2.6.32-220.23.1.el6.YAHOO.20120713.x86_64 #1 SMP Fri Jul 13 11:40:51 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux

          Show
          Ted Yu added a comment - I got the following test failures running test suite: [junit] Test org.apache.hadoop.fs.TestFsShellReturnCode FAILED [junit] Test org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark FAILED [junit] Test org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup FAILED [junit] Test org.apache.hadoop.tools.rumen.TestRumenJobTraces FAILED [junit] Test org.apache.hadoop.mapred.TestFairSchedulerPoolNames FAILED In TEST-org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark.txt, I found: 2013-05-02 03:14:31,718 ERROR namenode.NNThroughputBenchmark (NNThroughputBenchmark.java:runBenchmark(1206)) - java.io.IOException: NNThroughputBenchmark: cannot mkdir build/test/dfs/hosts/exclude at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.<init>(NNThroughputBenchmark.java:106) at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1163) at org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark.testNNThroughput(TestNNThroughputBenchmark.java:38) I am not sure if the test failures were related to my patch. Here is the OS: Linux 2.6.32-220.23.1.el6.YAHOO.20120713.x86_64 #1 SMP Fri Jul 13 11:40:51 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good. Please post the results once you have finished running the tests. Thanks.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good. Please post the results once you have finished running the tests. Thanks.
          Ted Yu made changes -
          Attachment 4774-v3.txt [ 12581407 ]
          Hide
          Ted Yu added a comment -

          Thanks for the review, Nicolas.

          Patch v3 addresses your comments.

          Show
          Ted Yu added a comment - Thanks for the review, Nicolas. Patch v3 addresses your comments.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          For the new patch:

          • In FSNamesystem.isFileClosed(..),
            • getPermissionChecker() should not be synchronized (see getFileInfo(String src) as an example.)
            • checkPathAccess(pc, src, FsAction.READ) is not needed.
            • The following can be simplified as "return !inode.isUnderConstruction()".
              +  if (!inode.isUnderConstruction()) {
              +      return true;
              +    }
              +    return false;
              +  }
              
          Show
          Tsz Wo Nicholas Sze added a comment - For the new patch: In FSNamesystem.isFileClosed(..), getPermissionChecker() should not be synchronized (see getFileInfo(String src) as an example.) checkPathAccess(pc, src, FsAction.READ) is not needed. The following can be simplified as "return !inode.isUnderConstruction()". + if (!inode.isUnderConstruction()) { + return true ; + } + return false ; + }
          Hide
          Ted Yu added a comment -

          Test suite passed under branch-1

          Show
          Ted Yu added a comment - Test suite passed under branch-1
          Ted Yu made changes -
          Attachment 4774-v2.txt [ 12581333 ]
          Hide
          Ted Yu added a comment -

          Patch v2 compiles under branch 1

          Running test suite.

          Show
          Ted Yu added a comment - Patch v2 compiles under branch 1 Running test suite.
          Tsz Wo Nicholas Sze made changes -
          Summary Backport HDFS-4525 'Provide an API for knowing whether file is closed or not' to branch 1.1 Backport HDFS-4525 'Provide an API for knowing whether file is closed or not' to branch-1
          Description HDFS-4525 compliments lease recovery API which allows user to know whether the recovery has completed.

          This JIRA backports the API to branch 1.1
          HDFS-4525 compliments lease recovery API which allows user to know whether the recovery has completed.

          This JIRA backports the API to branch-1.
          Tsz Wo Nicholas Sze made changes -
          Issue Type Improvement [ 4 ] New Feature [ 2 ]
          Component/s hdfs-client [ 12312928 ]
          Component/s namenode [ 12312926 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Ted,

          The patch does not compile with branch-1. Could you update it?

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Ted, The patch does not compile with branch-1. Could you update it?
          Ted Yu made changes -
          Summary Backport HDFS-4525 'Provide an API for knowing that whether file is closed or not' to branch 1.1 Backport HDFS-4525 'Provide an API for knowing whether file is closed or not' to branch 1.1
          Hide
          Ted Yu added a comment -

          I got the following when running test suite:

              [junit] Test org.apache.hadoop.hdfs.TestFileAppend4 FAILED
              [junit] Test org.apache.hadoop.hdfs.TestLargeBlock FAILED (timeout)
              [junit] Test org.apache.hadoop.metrics2.impl.TestSinkQueue FAILED
              [junit] Test org.apache.hadoop.streaming.TestUlimit FAILED
              [junit] Test org.apache.hadoop.mapred.TestFairSchedulerPoolNames FAILED
          

          They all passed when run standalone.

          Show
          Ted Yu added a comment - I got the following when running test suite: [junit] Test org.apache.hadoop.hdfs.TestFileAppend4 FAILED [junit] Test org.apache.hadoop.hdfs.TestLargeBlock FAILED (timeout) [junit] Test org.apache.hadoop.metrics2.impl.TestSinkQueue FAILED [junit] Test org.apache.hadoop.streaming.TestUlimit FAILED [junit] Test org.apache.hadoop.mapred.TestFairSchedulerPoolNames FAILED They all passed when run standalone.
          Ted Yu made changes -
          Attachment 4774.txt [ 12580996 ]
          Ted Yu made changes -
          Field Original Value New Value
          Link This issue relates to HDFS-4525 [ HDFS-4525 ]
          Ted Yu created issue -

            People

            • Assignee:
              Ted Yu
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development