Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4593

Deadlock in AbstractService.getConfig()

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: yarn
    • Labels:
      None
    • Environment:

      AM restarting on kerberized cluster

    • Target Version/s:

      Description

      SLIDER-1052 has found a deadlock which can arise in it during AM restart. Looking at the thread trace, one of the blockages is actually AbstractService.getConfig() —this is synchronized and so blocked.

      1. YARN-4593-001.patch
        0.6 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          I don't see why we need this to be synchronized (it's that way in branch-205); the actual type is volatile so access is thread safe anyway.

          Show
          stevel@apache.org Steve Loughran added a comment - I don't see why we need this to be synchronized (it's that way in branch-205); the actual type is volatile so access is thread safe anyway.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          SLIDER-1052 shows this. I'm not going to blame this JIRA for that, but it's certainly where the problem arises

          Show
          stevel@apache.org Steve Loughran added a comment - SLIDER-1052 shows this. I'm not going to blame this JIRA for that, but it's certainly where the problem arises
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 001, removes the -synchronized attributed. The field is still volatile (and only written in Service.init()

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 001, removes the -synchronized attributed. The field is still volatile (and only written in Service.init()
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 42s trunk passed
          -1 compile 1m 42s root in trunk failed with JDK v1.8.0_66.
          +1 compile 6m 49s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 0m 51s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 3s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 40s the patch passed
          -1 compile 0m 37s root in the patch failed with JDK v1.8.0_66.
          -1 javac 0m 37s root in the patch failed with JDK v1.8.0_66.
          +1 compile 6m 49s the patch passed with JDK v1.7.0_91
          +1 javac 6m 49s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 1m 2s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 58s the patch passed
          +1 javadoc 0m 53s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 4s the patch passed with JDK v1.7.0_91
          -1 unit 0m 27s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 7m 34s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          45m 44s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782366/YARN-4593-001.patch
          JIRA Issue YARN-4593
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2c4964c87234 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 817cc1f
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          compile https://builds.apache.org/job/PreCommit-YARN-Build/10290/artifact/patchprocess/branch-compile-root-jdk1.8.0_66.txt
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-YARN-Build/10290/artifact/patchprocess/patch-compile-root-jdk1.8.0_66.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/10290/artifact/patchprocess/patch-compile-root-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10290/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10290/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10290/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 42s trunk passed -1 compile 1m 42s root in trunk failed with JDK v1.8.0_66. +1 compile 6m 49s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 51s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 40s the patch passed -1 compile 0m 37s root in the patch failed with JDK v1.8.0_66. -1 javac 0m 37s root in the patch failed with JDK v1.8.0_66. +1 compile 6m 49s the patch passed with JDK v1.7.0_91 +1 javac 6m 49s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 58s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 4s the patch passed with JDK v1.7.0_91 -1 unit 0m 27s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 7m 34s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 45m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782366/YARN-4593-001.patch JIRA Issue YARN-4593 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2c4964c87234 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 817cc1f Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 compile https://builds.apache.org/job/PreCommit-YARN-Build/10290/artifact/patchprocess/branch-compile-root-jdk1.8.0_66.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-YARN-Build/10290/artifact/patchprocess/patch-compile-root-jdk1.8.0_66.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/10290/artifact/patchprocess/patch-compile-root-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10290/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10290/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10290/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          build failed due to make. Resubmitting as this is a .java only patch

          Show
          stevel@apache.org Steve Loughran added a comment - build failed due to make. Resubmitting as this is a .java only patch
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 48s trunk passed
          +1 compile 6m 1s trunk passed with JDK v1.8.0_72
          +1 compile 6m 55s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 36s trunk passed
          +1 javadoc 0m 57s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 6s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 43s the patch passed
          +1 compile 5m 56s the patch passed with JDK v1.8.0_72
          +1 javac 5m 56s the patch passed
          +1 compile 6m 59s the patch passed with JDK v1.7.0_95
          +1 javac 6m 59s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 1m 6s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 54s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 10s the patch passed with JDK v1.7.0_95
          +1 unit 7m 10s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 7m 35s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          60m 53s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782366/YARN-4593-001.patch
          JIRA Issue YARN-4593
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux be0b5e7d5f34 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1cb2f93
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10664/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10664/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 48s trunk passed +1 compile 6m 1s trunk passed with JDK v1.8.0_72 +1 compile 6m 55s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 36s trunk passed +1 javadoc 0m 57s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 6s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 43s the patch passed +1 compile 5m 56s the patch passed with JDK v1.8.0_72 +1 javac 5m 56s the patch passed +1 compile 6m 59s the patch passed with JDK v1.7.0_95 +1 javac 6m 59s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 6s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 54s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 10s the patch passed with JDK v1.7.0_95 +1 unit 7m 10s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 7m 35s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 60m 53s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782366/YARN-4593-001.patch JIRA Issue YARN-4593 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux be0b5e7d5f34 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1cb2f93 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10664/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/10664/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          No tests. I'd have to think of a way to recreate the deadlock then make sure it was gone. someone will need to look at the code instead

          Show
          stevel@apache.org Steve Loughran added a comment - No tests. I'd have to think of a way to recreate the deadlock then make sure it was gone. someone will need to look at the code instead
          Hide
          kasha Karthik Kambatla added a comment -

          Seems like a safe change. +1

          Show
          kasha Karthik Kambatla added a comment - Seems like a safe change. +1
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks -patch applied

          Show
          stevel@apache.org Steve Loughran added a comment - thanks -patch applied
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9469 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9469/)
          YARN-4593 Deadlock in AbstractService.getConfig() (stevel) (stevel: rev 605fdcbb81687c73ba91a3bd0d607cabd3dc5a67)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/AbstractService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9469 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9469/ ) YARN-4593 Deadlock in AbstractService.getConfig() (stevel) (stevel: rev 605fdcbb81687c73ba91a3bd0d607cabd3dc5a67) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/AbstractService.java

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development