Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8217

During block recovery for truncate Log new Block Id in case of copy-on-truncate is true.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      During the block recovery for the truncate, include the newblock Id in case copy-on-truncate set to true.
      Current log message is just as below.

      NameNode at /127.0.0.1:55036 calls recoverBlock(BP-409377067-xx.xx.xx.253-1429679063755:blk_1073741825_1001, targets=[DatanodeInfoWithStorage[127.0.0.1:55045,null,null]], newGenerationStamp=1002)

        Activity

        Hide
        vinayrpet Vinayakumar B added a comment -

        Attached the patch.
        Please review.

        Show
        vinayrpet Vinayakumar B added a comment - Attached the patch. Please review.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 29s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 26s There were no new javac warning messages.
        +1 javadoc 9m 31s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 7m 34s The applied patch generated 2 additional checkstyle issues.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 13s Pre-build of native portion
        +1 hdfs tests 166m 3s Tests passed in hadoop-hdfs.
            213m 54s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12727131/HDFS-8217-01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 6f8003d
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10341/artifact/patchprocess/checkstyle-result-diff.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10341/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10341/testReport/
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10341//console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 29s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 26s There were no new javac warning messages. +1 javadoc 9m 31s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 7m 34s The applied patch generated 2 additional checkstyle issues. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 13s Pre-build of native portion +1 hdfs tests 166m 3s Tests passed in hadoop-hdfs.     213m 54s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727131/HDFS-8217-01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6f8003d checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10341/artifact/patchprocess/checkstyle-result-diff.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10341/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10341/testReport/ Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10341//console This message was automatically generated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Handled checkstyle comments

        Show
        vinayrpet Vinayakumar B added a comment - Handled checkstyle comments
        Hide
        zero45 Plamen Jeliazkov added a comment -

        Hi Vinayakumar B,

        Just looked over the work.

        My only nit, I see you are changing the style of the LOG.info from String concat to StringBuilder.
        It does change the overall style of other LOG.info statements and the two are equivalent after compilation so long as the String concat is not within a loop.

        Other than that, looks good! Thanks!

        Show
        zero45 Plamen Jeliazkov added a comment - Hi Vinayakumar B , Just looked over the work. My only nit, I see you are changing the style of the LOG.info from String concat to StringBuilder. It does change the overall style of other LOG.info statements and the two are equivalent after compilation so long as the String concat is not within a loop. Other than that, looks good! Thanks!
        Hide
        vinayrpet Vinayakumar B added a comment -

        Thanks Plamen Jeliazkov for taking a look at the patch.

        My only nit, I see you are changing the style of the LOG.info from String concat to StringBuilder.

        Yes I agree that style is changed. But It depends on how you use while building big messages and conditional messages.
        In the first patch (.01.patch) if you see, I followed the earlier approach, but since I used inline conditional statements I got checkstyle comment from Hadoop QA.
        And I thought StringBuilder would be better.

        Show
        vinayrpet Vinayakumar B added a comment - Thanks Plamen Jeliazkov for taking a look at the patch. My only nit, I see you are changing the style of the LOG.info from String concat to StringBuilder. Yes I agree that style is changed. But It depends on how you use while building big messages and conditional messages. In the first patch (.01.patch) if you see, I followed the earlier approach, but since I used inline conditional statements I got checkstyle comment from Hadoop QA . And I thought StringBuilder would be better.
        Hide
        shv Konstantin Shvachko added a comment -

        I think it is fine to ignore AvoidInlineConditionalsCheck.
        Automation tools should help improve code, but this one just forces you to write bunch of code for a simple LOG message.
        +1 on your first version.

        Show
        shv Konstantin Shvachko added a comment - I think it is fine to ignore AvoidInlineConditionalsCheck. Automation tools should help improve code, but this one just forces you to write bunch of code for a simple LOG message. +1 on your first version.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Thanks Konstantin Shvachko and Plamen Jeliazkov
        I will commit the first version.

        Show
        vinayrpet Vinayakumar B added a comment - Thanks Konstantin Shvachko and Plamen Jeliazkov I will commit the first version.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Removed the .02 patch to not to confuse further

        Show
        vinayrpet Vinayakumar B added a comment - Removed the .02 patch to not to confuse further
        Hide
        vinayrpet Vinayakumar B added a comment -

        Committed to trunk and branch-2

        Show
        vinayrpet Vinayakumar B added a comment - Committed to trunk and branch-2
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7660 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7660/)
        HDFS-8217. During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7660 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7660/ ) HDFS-8217 . During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2105 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2105/)
        HDFS-8217. During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2105 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2105/ ) HDFS-8217 . During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #164 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/164/)
        HDFS-8217. During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #164 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/164/ ) HDFS-8217 . During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/173/)
        HDFS-8217. During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/173/ ) HDFS-8217 . During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #907 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/907/)
        HDFS-8217. During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #907 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/907/ ) HDFS-8217 . During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #174 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/174/)
        HDFS-8217. During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #174 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/174/ ) HDFS-8217 . During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2123 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2123/)
        HDFS-8217. During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2123 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2123/ ) HDFS-8217 . During block recovery for truncate Log new Block Id in case of copy-on-truncate is true. (Contributed by Vinayakumar B) (vinayakumarb: rev 262c1bc3398ce2ede03f9d86fc97c35ca7a8e9db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            vinayrpet Vinayakumar B
            Reporter:
            vinayrpet Vinayakumar B
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development