Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12084

Scheduled Count will not decrement when file is deleted before all IBR's received

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      When small files creation && deletion happens so frequently and DN's did not report blocks to NN before deletion, then scheduled count will keep on increment and which will not deleted as blocks are deleted.

      Note: Every 20 mins,this can be rolled, but with in 20 mins, count can be more as so many operations.
      when batchIBR enabled with committed allowed=1 this will be observed more.

      1. HDFS-12084-001.patch
        6 kB
        Brahma Reddy Battula
      2. HDFS-12084-002.patch
        6 kB
        Brahma Reddy Battula
      3. HDFS-12084-003.patch
        9 kB
        Brahma Reddy Battula
      4. HDFS-12084-branch-2.patch
        6 kB
        Brahma Reddy Battula

        Activity

        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploaded the patch,kindly review

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch,kindly review
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 14m 21s trunk passed
        +1 compile 0m 49s trunk passed
        +1 checkstyle 0m 38s trunk passed
        +1 mvnsite 0m 55s trunk passed
        -1 findbugs 1m 45s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 41s trunk passed
        +1 mvninstall 0m 50s the patch passed
        +1 compile 0m 52s the patch passed
        +1 javac 0m 52s the patch passed
        -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 118 unchanged - 0 fixed = 122 total (was 118)
        +1 mvnsite 0m 50s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 53s the patch passed
        +1 javadoc 0m 41s the patch passed
        -1 unit 65m 29s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        92m 16s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12084
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875631/HDFS-12084-001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0d63622129d4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b17e655
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20151/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20151/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20151/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20151/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20151/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 21s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 55s trunk passed -1 findbugs 1m 45s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 118 unchanged - 0 fixed = 122 total (was 118) +1 mvnsite 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 41s the patch passed -1 unit 65m 29s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 92m 16s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12084 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875631/HDFS-12084-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0d63622129d4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b17e655 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20151/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20151/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20151/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20151/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20151/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Updated the patch to address the checkystyle issues.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Updated the patch to address the checkystyle issues.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 13m 17s trunk passed
        +1 compile 0m 47s trunk passed
        +1 checkstyle 0m 41s trunk passed
        +1 mvnsite 0m 57s trunk passed
        -1 findbugs 1m 47s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 40s trunk passed
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 34s the patch passed
        +1 mvnsite 0m 50s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 44s the patch passed
        +1 javadoc 0m 37s the patch passed
        -1 unit 94m 8s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        119m 21s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12084
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875908/HDFS-12084-002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2e28ec4c2384 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 946dd25
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20174/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20174/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20174/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20174/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 17s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 0m 57s trunk passed -1 findbugs 1m 47s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 34s the patch passed +1 mvnsite 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 94m 8s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 119m 21s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12084 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875908/HDFS-12084-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2e28ec4c2384 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 946dd25 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20174/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20174/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20174/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20174/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Testfailures are unrelated.Kindly review.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Testfailures are unrelated.Kindly review.
        Hide
        shahrs87 Rushabh S Shah added a comment - - edited

        Brahma Reddy Battula: thanks for the patch.
        taking a look now.

        Show
        shahrs87 Rushabh S Shah added a comment - - edited Brahma Reddy Battula : thanks for the patch. taking a look now.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        I don't have enough context for Erasure coding. I will let some other member review. sorry.

        Show
        shahrs87 Rushabh S Shah added a comment - I don't have enough context for Erasure coding. I will let some other member review. sorry.
        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        Hi Tsz Wo Nicholas Sze/Vinayakumar B/Andrew Wang/Jing Zhao

        if you get time, kindly review this patch..

        thanks

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited Hi Tsz Wo Nicholas Sze / Vinayakumar B / Andrew Wang / Jing Zhao if you get time, kindly review this patch.. thanks
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploading the branch-2 patch.IMO, this should go to 2.8.2 release.Kindly review.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploading the branch-2 patch.IMO, this should go to 2.8.2 release.Kindly review.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploading the trunk patch. Updated the testcase to reproduce all the time without fix. Earlier there was race in test.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploading the trunk patch. Updated the testcase to reproduce all the time without fix. Earlier there was race in test.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 17m 2s trunk passed
        +1 compile 0m 55s trunk passed
        +1 checkstyle 0m 42s trunk passed
        +1 mvnsite 1m 4s trunk passed
        +1 findbugs 2m 0s trunk passed
        +1 javadoc 0m 44s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 55s the patch passed
        +1 compile 0m 57s the patch passed
        +1 javac 0m 57s the patch passed
        +1 checkstyle 0m 42s the patch passed
        +1 mvnsite 1m 3s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 3s the patch passed
        +1 javadoc 0m 41s the patch passed
              Other Tests
        -1 unit 78m 6s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        109m 3s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
        Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12084
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883518/HDFS-12084-003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 346fc1ddc46b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3123895
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20843/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20843/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20843/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 17m 2s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 4s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 0m 44s trunk passed       Patch Compile Tests +1 mvninstall 0m 55s the patch passed +1 compile 0m 57s the patch passed +1 javac 0m 57s the patch passed +1 checkstyle 0m 42s the patch passed +1 mvnsite 1m 3s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 0m 41s the patch passed       Other Tests -1 unit 78m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 109m 3s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12084 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883518/HDFS-12084-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 346fc1ddc46b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3123895 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20843/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20843/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20843/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment -

        I don't know if it is related, but I see reserve RBW space gets stuck and never going downon certain datanode.

        Show
        kihwal Kihwal Lee added a comment - I don't know if it is related, but I see reserve RBW space gets stuck and never going downon certain datanode.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Kihwal Lee thanks for taking a look.

        but I see reserve RBW space gets stuck and never going downon certain datanode.

        are you telling about DN side reserveSpaceForReplica..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - Kihwal Lee thanks for taking a look. but I see reserve RBW space gets stuck and never going downon certain datanode. are you telling about DN side reserveSpaceForReplica..?
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Hi Kihwal Lee
        If you get chance,can you please review the patch??
        Thanks

        Show
        brahmareddy Brahma Reddy Battula added a comment - Hi Kihwal Lee If you get chance,can you please review the patch?? Thanks
        Hide
        kihwal Kihwal Lee added a comment -

        Sorry, I will get to it today.

        Show
        kihwal Kihwal Lee added a comment - Sorry, I will get to it today.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        No Problem.I feel,this should go in 2.8.2 too.

        Show
        brahmareddy Brahma Reddy Battula added a comment - No Problem.I feel,this should go in 2.8.2 too.
        Hide
        kihwal Kihwal Lee added a comment -

        I was looking at the 2.8 patch.

        • Explicit calls to abandonBlock() : unprotectedRemoveBlock() will decrement it.
        • Delete before commit : not covered?
        • Delete after commit : this jira
        • Abandoned. Lease recovery and commitBlockSynchronization() will take care of it.
        • pipeline updates (i.e. abandoned replicas): not covered?

        It feels that we need to improve deletion path to take care of both before and after commit cases. We need to think a bit more about pipeline updates.

        Show
        kihwal Kihwal Lee added a comment - I was looking at the 2.8 patch. Explicit calls to abandonBlock() : unprotectedRemoveBlock() will decrement it. Delete before commit : not covered? Delete after commit : this jira Abandoned. Lease recovery and commitBlockSynchronization() will take care of it. pipeline updates (i.e. abandoned replicas): not covered? It feels that we need to improve deletion path to take care of both before and after commit cases. We need to think a bit more about pipeline updates.

          People

          • Assignee:
            brahmareddy Brahma Reddy Battula
            Reporter:
            brahmareddy Brahma Reddy Battula
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development