Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10769

BlockIdManager.clear doesn't reset the counter for blockGroupIdGenerator

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1, 3.0.0-beta1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: hdfs
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Tags:
      erasurecode

      Description

      BlockIdManager.clear resets the block id generator for contiguous blocks but not for striped blocks:

        void clear() {
          legacyGenerationStamp.setCurrentValue(GenerationStamp.LAST_RESERVED_STAMP);
          generationStamp.setCurrentValue(GenerationStamp.LAST_RESERVED_STAMP);
          getBlockIdGenerator().setCurrentValue(SequentialBlockIdGenerator
            .LAST_RESERVED_BLOCK_ID);
          legacyGenerationStampLimit = HdfsConstants.GRANDFATHER_GENERATION_STAMP;
        }
      

      This should have:

      getBlockGroupIdGenerator().setCurrentValue(Long.MIN_VALUE);
      
      1. HDFS-10769-002.patch
        2 kB
        Rakesh R
      2. HDFS-10769.001.patch
        0.8 kB
        Yiqun Lin

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10700 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10700/)
        HDFS-10769. BlockIdManager.clear doesn't reset the counter for (vinayakumarb: rev 77f2684dd3d480bdc16e95ffcd1d189a38d0acff)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSequentialBlockGroupId.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10700 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10700/ ) HDFS-10769 . BlockIdManager.clear doesn't reset the counter for (vinayakumarb: rev 77f2684dd3d480bdc16e95ffcd1d189a38d0acff) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSequentialBlockGroupId.java
        Hide
        vinayrpet Vinayakumar B added a comment -

        Committed to trunk,
        Thanks Yiqun Lin and Rakesh R

        Show
        vinayrpet Vinayakumar B added a comment - Committed to trunk, Thanks Yiqun Lin and Rakesh R
        Hide
        linyiqun Yiqun Lin added a comment -

        +1 for the latest patch, the failed test are not related. Thanks Rakesh R for updating the patch!

        Show
        linyiqun Yiqun Lin added a comment - +1 for the latest patch, the failed test are not related. Thanks Rakesh R for updating the patch!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 55s trunk passed
        +1 compile 0m 52s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 0m 59s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 57s trunk passed
        +1 javadoc 0m 43s trunk passed
        +1 mvninstall 0m 54s the patch passed
        +1 compile 0m 51s the patch passed
        +1 javac 0m 51s the patch passed
        +1 checkstyle 0m 26s the patch passed
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 6s the patch passed
        +1 javadoc 0m 42s the patch passed
        -1 unit 47m 29s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        68m 37s



        Reason Tests
        Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          org.apache.hadoop.hdfs.server.datanode.TestLargeBlockReport
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
          org.apache.hadoop.cli.TestDeleteCLI
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
          org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
          org.apache.hadoop.hdfs.server.datanode.TestBlockScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10769
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835501/HDFS-10769-002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d1962d2a7c0a 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a895c66
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17317/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17317/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17317/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 55s trunk passed +1 compile 0m 52s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 57s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 51s the patch passed +1 javac 0m 51s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 0m 42s the patch passed -1 unit 47m 29s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 68m 37s Reason Tests Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   org.apache.hadoop.hdfs.server.datanode.TestLargeBlockReport   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   org.apache.hadoop.cli.TestDeleteCLI   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache   org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   org.apache.hadoop.hdfs.server.datanode.TestBlockScanner Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10769 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835501/HDFS-10769-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d1962d2a7c0a 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a895c66 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17317/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17317/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17317/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        +1 for the latest patch.
        Thanks Rakesh R for the test.
        Pending Jenkins

        Show
        vinayrpet Vinayakumar B added a comment - +1 for the latest patch. Thanks Rakesh R for the test. Pending Jenkins
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Yiqun Lin, Ewan Higgs. +1 for the fix.

        I've rebased the patch and added a unit test case to verify the same. Please review it. Thanks!

        Show
        rakeshr Rakesh R added a comment - Thanks Yiqun Lin , Ewan Higgs . +1 for the fix. I've rebased the patch and added a unit test case to verify the same. Please review it. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 11s trunk passed
        +1 compile 0m 57s trunk passed
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 0m 57s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 51s trunk passed
        +1 javadoc 0m 58s trunk passed
        +1 mvninstall 0m 55s the patch passed
        +1 compile 0m 48s the patch passed
        +1 javac 0m 48s the patch passed
        +1 checkstyle 0m 23s the patch passed
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 3s the patch passed
        +1 javadoc 1m 0s the patch passed
        -1 unit 67m 53s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        89m 22s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824269/HDFS-10769.001.patch
        JIRA Issue HDFS-10769
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux adc36d47d5fc 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 20f0eb8
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16466/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16466/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16466/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 11s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 0m 58s trunk passed +1 mvninstall 0m 55s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 1m 0s the patch passed -1 unit 67m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 89m 22s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824269/HDFS-10769.001.patch JIRA Issue HDFS-10769 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux adc36d47d5fc 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 20f0eb8 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16466/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16466/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16466/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Ewan Higgs for reporting this. I think this is a bug for here. Attach a simple patch for this.

        Show
        linyiqun Yiqun Lin added a comment - Thanks Ewan Higgs for reporting this. I think this is a bug for here. Attach a simple patch for this.

          People

          • Assignee:
            linyiqun Yiqun Lin
            Reporter:
            ehiggs Ewan Higgs
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development