Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8850

VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks

        Details

        • Type: Bug
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 2.7.0
        • Fix Version/s: 2.8.0, 2.7.2, 3.0.0-alpha1
        • Component/s: datanode
        • Labels:
          None
        • Target Version/s:
        • Hadoop Flags:
          Reviewed

          Description

          The VolumeScanner threads inside the BlockScanner exit with an exception if there is no block pool to be scanned but there are suspicious blocks.

          1. Text File
            HDFS-8850.001.patch
            1 kB
            Colin P. McCabe

            Issue Links

            duplicates

            Bug - A problem which impairs or prevents the functions of the product. HDFS-9190 VolumeScanner throwing NPE while scanning suspect block.

            • Critical - Crashes, loss of data, severe memory leak.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              hitliuyi Yi Liu added a comment -

              Yes, you are right. +1 pending Jenkins.

              Show
              hitliuyi Yi Liu added a comment - Yes, you are right. +1 pending Jenkins.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -



              -1 overall



              Vote Subsystem Runtime Comment
              0 pre-patch 17m 11s Pre-patch trunk compilation is healthy.
              +1 @author 0m 0s The patch does not contain any @author tags.
              -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              +1 javac 7m 40s There were no new javac warning messages.
              +1 javadoc 9m 40s There were no new javadoc warning messages.
              +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
              +1 checkstyle 1m 20s There were no new checkstyle issues.
              +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              +1 install 1m 23s mvn install still works.
              +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
              +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
              +1 native 3m 2s Pre-build of native portion
              -1 hdfs tests 158m 28s Tests failed in hadoop-hdfs.
                  202m 14s  



              Reason Tests
              Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyIsHot



              Subsystem Report/Notes
              Patch URL http://issues.apache.org/jira/secure/attachment/12748576/HDFS-8850.001.patch
              Optional Tests javadoc javac unit findbugs checkstyle
              git revision trunk / c3364ca
              hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11892/artifact/patchprocess/testrun_hadoop-hdfs.txt
              Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11892/testReport/
              Java 1.7.0_55
              uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
              Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11892/console

              This message was automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 20s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 2s Pre-build of native portion -1 hdfs tests 158m 28s Tests failed in hadoop-hdfs.     202m 14s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyIsHot Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748576/HDFS-8850.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c3364ca hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11892/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11892/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11892/console This message was automatically generated.
              Hide
              Permalink
              hitliuyi Yi Liu added a comment -

              Test failure is unrelated. Committed to trunk and branch-2.
              Thanks Colin P. McCabe for the contribution.

              Show
              hitliuyi Yi Liu added a comment - Test failure is unrelated. Committed to trunk and branch-2. Thanks Colin P. McCabe for the contribution.
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-trunk-Commit #8259 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8259/)
              HDFS-8850. VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a)

              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8259 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8259/ ) HDFS-8850 . VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/277/)
              HDFS-8850. VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a)

              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/277/ ) HDFS-8850 . VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Yarn-trunk #1007 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1007/)
              HDFS-8850. VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a)

              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1007 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1007/ ) HDFS-8850 . VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Hdfs-trunk #2204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2204/)
              HDFS-8850. VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a)

              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2204/ ) HDFS-8850 . VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Mapreduce-trunk #2223 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2223/)
              HDFS-8850. VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a)

              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2223 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2223/ ) HDFS-8850 . VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #266 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/266/)
              HDFS-8850. VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a)

              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #266 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/266/ ) HDFS-8850 . VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/274/)
              HDFS-8850. VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a)

              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/274/ ) HDFS-8850 . VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) (yliu: rev 9a08999906824e8982ec3b47315f9b960474186a) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Hide
              Permalink
              sjlee0 Sangjin Lee added a comment -

              Removing the 2.6.1-candidate label, as this is an issue for a class that exists only in 2.7 and as such I don't think it applies to 2.6. Let me know if you think this issue exists in 2.6.

              Show
              sjlee0 Sangjin Lee added a comment - Removing the 2.6.1-candidate label, as this is an issue for a class that exists only in 2.7 and as such I don't think it applies to 2.6. Let me know if you think this issue exists in 2.6.
              Hide
              Permalink
              xuchenCN Xu Chen added a comment -

              +1 LGTM

              Show
              xuchenCN Xu Chen added a comment - +1 LGTM
              Hide
              Permalink
              vinodkv Vinod Kumar Vavilapalli added a comment -

              Adding back 2.6.1-candidate label for tracking. Will remove it once 2.6.1 is done.

              Show
              vinodkv Vinod Kumar Vavilapalli added a comment - Adding back 2.6.1-candidate label for tracking. Will remove it once 2.6.1 is done.
              Hide
              Permalink
              vinodkv Vinod Kumar Vavilapalli added a comment -

              About to make an RC for 2.6.1. Dropping the 2.6.1 label as we have decided to drop this. Adding it to 2.7.2 though.

              Show
              vinodkv Vinod Kumar Vavilapalli added a comment - About to make an RC for 2.6.1. Dropping the 2.6.1 label as we have decided to drop this. Adding it to 2.7.2 though.
              Hide
              Permalink
              shahrs87 Rushabh S Shah added a comment -

              Yi Liu, Colin P. McCabe: Does it make sense to commit to 2.7.2 ?

              Show
              shahrs87 Rushabh S Shah added a comment - Yi Liu , Colin P. McCabe : Does it make sense to commit to 2.7.2 ?
              Hide
              Permalink
              kihwal Kihwal Lee added a comment -

              Cherry-picked to branch-2.7.

              Show
              kihwal Kihwal Lee added a comment - Cherry-picked to branch-2.7.
              Hide
              Permalink
              cmccabe Colin P. McCabe added a comment -

              I think this makes sense for 2.7.2. Thanks, all.

              Show
              cmccabe Colin P. McCabe added a comment - I think this makes sense for 2.7.2. Thanks, all.

                People

                • Assignee:
                  cmccabe Colin P. McCabe
                  Reporter:
                  cmccabe Colin P. McCabe
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  10 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development