Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8778

TestBlockReportRateLimiting#testLeaseExpiration can deadlock

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      requestBlockReportLease blocks on DataNode registration while holding the NameSystem read lock.

      DataNode registration can block on the NameSystem read lock if a writer gets in the queue.

      1. HDFS-8778.02.patch
        5 kB
        Arpit Agarwal
      2. HDFS-8778.01.patch
        5 kB
        Arpit Agarwal

        Activity

        Hide
        arpitagarwal Arpit Agarwal added a comment -

        The fix is to make requestBlockReportLease non-blocking.

        Also ensure that the cluster is shutdown on test failure.

        Show
        arpitagarwal Arpit Agarwal added a comment - The fix is to make requestBlockReportLease non-blocking. Also ensure that the cluster is shutdown on test failure.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 7m 40s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 37s There were no new javac warning messages.
        +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 24s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 20s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 1m 5s Pre-build of native portion
        -1 hdfs tests 161m 27s Tests failed in hadoop-hdfs.
            184m 0s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestDistributedFileSystem
          hadoop.hdfs.server.namenode.ha.TestStandbyIsHot



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12745354/HDFS-8778.01.patch
        Optional Tests javac unit findbugs checkstyle
        git revision trunk / 0a16ee6
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11707/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11707/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11707/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 7m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 37s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 24s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 1m 5s Pre-build of native portion -1 hdfs tests 161m 27s Tests failed in hadoop-hdfs.     184m 0s   Reason Tests Failed unit tests hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745354/HDFS-8778.01.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 0a16ee6 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11707/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11707/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11707/console This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Add one more assert to make sure DN shutdown call succeeded.

        Show
        arpitagarwal Arpit Agarwal added a comment - Add one more assert to make sure DN shutdown call succeeded.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 8m 11s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 43s There were no new javac warning messages.
        +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 21s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 17s mvn install still works.
        +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
        +1 findbugs 2m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 1m 5s Pre-build of native portion
        -1 hdfs tests 159m 25s Tests failed in hadoop-hdfs.
            182m 22s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestDistributedFileSystem



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12745471/HDFS-8778.02.patch
        Optional Tests javac unit findbugs checkstyle
        git revision trunk / edcaae4
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11716/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11716/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11716/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 8m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 21s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 17s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 2m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 1m 5s Pre-build of native portion -1 hdfs tests 159m 25s Tests failed in hadoop-hdfs.     182m 22s   Reason Tests Failed unit tests hadoop.hdfs.TestDistributedFileSystem Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745471/HDFS-8778.02.patch Optional Tests javac unit findbugs checkstyle git revision trunk / edcaae4 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11716/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11716/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11716/console This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM +1, thanks Arpit for finding and fixing. Test failure looks unrelated.

        Show
        andrew.wang Andrew Wang added a comment - LGTM +1, thanks Arpit for finding and fixing. Test failure looks unrelated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks for the review Andrew. Committed for 2.8.0.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks for the review Andrew. Committed for 2.8.0.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8169 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8169/)
        HDFS-8778. TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8169 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8169/ ) HDFS-8778 . TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        walter.k.su Walter Su added a comment -

        TestStandbyIsHot failed again. I have filed HDFS-8772 to fix that. Please take a look if you have time. Thanks.

        Show
        walter.k.su Walter Su added a comment - TestStandbyIsHot failed again. I have filed HDFS-8772 to fix that. Please take a look if you have time. Thanks.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #258 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/258/)
        HDFS-8778. TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #258 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/258/ ) HDFS-8778 . TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #988 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/988/)
        HDFS-8778. TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #988 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/988/ ) HDFS-8778 . TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Hdfs-trunk #2185 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2185/)
        HDFS-8778. TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk #2185 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2185/ ) HDFS-8778 . TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Mapreduce-trunk #2204 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2204/)
        HDFS-8778. TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Mapreduce-trunk #2204 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2204/ ) HDFS-8778 . TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #246 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/246/)
        HDFS-8778. TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #246 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/246/ ) HDFS-8778 . TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #256 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/256/)
        HDFS-8778. TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #256 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/256/ ) HDFS-8778 . TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) (arp: rev 3ec0a0444f75c8743289ec7c8645d4bdf51fc45a) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportRateLimiting.java

          People

          • Assignee:
            arpitagarwal Arpit Agarwal
            Reporter:
            arpitagarwal Arpit Agarwal
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development