Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6714

IllegalStateException while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0, 3.0.0-alpha3
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently in async-scheduling mode of CapacityScheduler, after AM failover and unreserve all reserved containers, it still have chance to get and commit the outdated reserve proposal of the failed app attempt. This problem happened on an app in our cluster, when this app stopped, it unreserved all reserved containers and compared these appAttemptId with current appAttemptId, if not match it will throw IllegalStateException and make RM crashed.

      Error log:

      2017-06-08 11:02:24,339 FATAL [ResourceManager Event Processor] org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type APP_ATTEMPT_REMOVED to the scheduler
      java.lang.IllegalStateException: Trying to unreserve  for application appattempt_1495188831758_0121_000002 when currently reserved  for application application_1495188831758_0121 on node host: node1:45454 #containers=2 available=... used=...
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode.unreserveResource(FiCaSchedulerNode.java:123)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.unreserve(FiCaSchedulerApp.java:845)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1787)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainerInternal(CapacityScheduler.java:1957)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.completedContainer(AbstractYarnScheduler.java:586)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.doneApplicationAttempt(CapacityScheduler.java:966)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1740)
              at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:152)
              at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:822)
              at java.lang.Thread.run(Thread.java:834)
      

      When async-scheduling enabled, CapacityScheduler#doneApplicationAttempt and CapacityScheduler#tryCommit both need to get write_lock before executing, so we can check the app attempt state in commit process to avoid committing outdated proposals.

      1. YARN-6714.001.patch
        12 kB
        Tao Yang
      2. YARN-6714.002.patch
        10 kB
        Tao Yang
      3. YARN-6714.003.patch
        10 kB
        Tao Yang
      4. YARN-6714.branch-2.003.patch
        10 kB
        Tao Yang
      5. YARN-6714.branch-2.004.patch
        10 kB
        Tao Yang
      6. YARN-6714.branch-2.005.patch
        10 kB
        Tao Yang
      7. YARN-6714.branch-2.006.patch
        10 kB
        Tao Yang

        Activity

        Hide
        sunilg Sunil G added a comment -

        Committed to branch-2 as well. Thanks Tao Yang for the contributions. Thanks Wangda Tan and Naganarasimha Garla for additional reviews.

        Show
        sunilg Sunil G added a comment - Committed to branch-2 as well. Thanks Tao Yang for the contributions. Thanks Wangda Tan and Naganarasimha Garla for additional reviews.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2 Compile Tests
        +1 mvninstall 7m 1s branch-2 passed
        +1 compile 0m 28s branch-2 passed with JDK v1.8.0_131
        +1 compile 0m 32s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 21s branch-2 passed
        +1 mvnsite 0m 38s branch-2 passed
        +1 findbugs 1m 19s branch-2 passed
        +1 javadoc 0m 20s branch-2 passed with JDK v1.8.0_131
        +1 javadoc 0m 23s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.8.0_131
        +1 javac 0m 26s the patch passed
        +1 compile 0m 29s the patch passed with JDK v1.7.0_131
        +1 javac 0m 29s the patch passed
        +1 checkstyle 0m 19s the patch passed
        +1 mvnsite 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 20s the patch passed
        +1 javadoc 0m 17s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 19s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 45m 56s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        106m 44s



        Reason Tests
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue YARN-6714
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877334/YARN-6714.branch-2.006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a8047908572c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / af80d10
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16447/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16447/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16447/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests +1 mvninstall 7m 1s branch-2 passed +1 compile 0m 28s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 32s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 21s branch-2 passed +1 mvnsite 0m 38s branch-2 passed +1 findbugs 1m 19s branch-2 passed +1 javadoc 0m 20s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 23s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 30s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_131 +1 javac 0m 26s the patch passed +1 compile 0m 29s the patch passed with JDK v1.7.0_131 +1 javac 0m 29s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 20s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 19s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 45m 56s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 106m 44s Reason Tests JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-6714 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877334/YARN-6714.branch-2.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a8047908572c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / af80d10 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/16447/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16447/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16447/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2 Compile Tests
        +1 mvninstall 6m 33s branch-2 passed
        +1 compile 0m 28s branch-2 passed with JDK v1.8.0_131
        +1 compile 0m 33s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 23s branch-2 passed
        +1 mvnsite 0m 39s branch-2 passed
        +1 findbugs 1m 11s branch-2 passed
        +1 javadoc 0m 20s branch-2 passed with JDK v1.8.0_131
        +1 javadoc 0m 24s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.8.0_131
        +1 javac 0m 26s the patch passed
        +1 compile 0m 30s the patch passed with JDK v1.7.0_131
        +1 javac 0m 30s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 0m 37s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 21s the patch passed
        +1 javadoc 0m 18s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 22s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 62m 5s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 24s The patch does not generate ASF License warnings.
        122m 50s



        Reason Tests
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.server.resourcemanager.TestRMRestart
        JDK v1.7.0_131 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue YARN-6714
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877308/YARN-6714.branch-2.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2a62a8b82705 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / af80d10
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16444/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16444/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests +1 mvninstall 6m 33s branch-2 passed +1 compile 0m 28s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 33s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 23s branch-2 passed +1 mvnsite 0m 39s branch-2 passed +1 findbugs 1m 11s branch-2 passed +1 javadoc 0m 20s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 24s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 32s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_131 +1 javac 0m 26s the patch passed +1 compile 0m 30s the patch passed with JDK v1.7.0_131 +1 javac 0m 30s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 21s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 62m 5s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 122m 50s Reason Tests JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.TestRMRestart JDK v1.7.0_131 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-6714 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877308/YARN-6714.branch-2.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2a62a8b82705 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / af80d10 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/16444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16444/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16444/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2 Compile Tests
        +1 mvninstall 6m 41s branch-2 passed
        +1 compile 0m 29s branch-2 passed with JDK v1.8.0_131
        +1 compile 0m 33s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 23s branch-2 passed
        +1 mvnsite 0m 39s branch-2 passed
        +1 findbugs 1m 13s branch-2 passed
        +1 javadoc 0m 21s branch-2 passed with JDK v1.8.0_131
        +1 javadoc 0m 23s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 27s the patch passed with JDK v1.8.0_131
        -1 javac 0m 27s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131 with JDK v1.8.0_131 generated 1 new + 14 unchanged - 1 fixed = 15 total (was 15)
        +1 compile 0m 30s the patch passed with JDK v1.7.0_131
        -1 javac 0m 30s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131 with JDK v1.7.0_131 generated 1 new + 13 unchanged - 1 fixed = 14 total (was 14)
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 0m 37s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 23s the patch passed
        +1 javadoc 0m 18s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 22s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 43m 30s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        103m 26s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue YARN-6714
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877273/YARN-6714.branch-2.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ab628882ef58 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / af80d10
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/16443/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/16443/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16443/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16443/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16443/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests +1 mvninstall 6m 41s branch-2 passed +1 compile 0m 29s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 33s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 23s branch-2 passed +1 mvnsite 0m 39s branch-2 passed +1 findbugs 1m 13s branch-2 passed +1 javadoc 0m 21s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 23s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 33s the patch passed +1 compile 0m 27s the patch passed with JDK v1.8.0_131 -1 javac 0m 27s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131 with JDK v1.8.0_131 generated 1 new + 14 unchanged - 1 fixed = 15 total (was 15) +1 compile 0m 30s the patch passed with JDK v1.7.0_131 -1 javac 0m 30s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131 with JDK v1.7.0_131 generated 1 new + 13 unchanged - 1 fixed = 14 total (was 14) +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 23s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 43m 30s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 103m 26s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-6714 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877273/YARN-6714.branch-2.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ab628882ef58 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / af80d10 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/16443/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/16443/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16443/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16443/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16443/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Tao Yang Tao Yang added a comment - - edited

        Sorry to have misplaced the actual types, and there are more custom generic types should be explicitly specified. Upload a new patch.

        Show
        Tao Yang Tao Yang added a comment - - edited Sorry to have misplaced the actual types, and there are more custom generic types should be explicitly specified. Upload a new patch.
        Hide
        sunilg Sunil G added a comment -

        Tao Yang Could you please help to check the warnings.

        Show
        sunilg Sunil G added a comment - Tao Yang Could you please help to check the warnings.
        Hide
        sunilg Sunil G added a comment -

        I think still error is there.

        Show
        sunilg Sunil G added a comment - I think still error is there.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2 Compile Tests
        -1 mvninstall 11m 51s root in branch-2 failed.
        +1 compile 0m 37s branch-2 passed with JDK v1.8.0_131
        +1 compile 0m 37s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 27s branch-2 passed
        +1 mvnsite 0m 44s branch-2 passed
        +1 findbugs 1m 28s branch-2 passed
        +1 javadoc 0m 26s branch-2 passed with JDK v1.8.0_131
        +1 javadoc 0m 27s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 38s the patch passed
        +1 compile 0m 36s the patch passed with JDK v1.8.0_131
        -1 javac 0m 36s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131 with JDK v1.8.0_131 generated 1 new + 14 unchanged - 1 fixed = 15 total (was 15)
        +1 compile 0m 38s the patch passed with JDK v1.7.0_131
        -1 javac 0m 38s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131 with JDK v1.7.0_131 generated 1 new + 13 unchanged - 1 fixed = 14 total (was 14)
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 0m 42s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 40s the patch passed
        +1 javadoc 0m 20s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 26s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 43m 54s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        114m 0s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue YARN-6714
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877030/YARN-6714.branch-2.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 897f4b8a06ab 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / af8a227
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/16411/artifact/patchprocess/branch-mvninstall-root.txt
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/16411/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/16411/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16411/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16411/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16411/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests -1 mvninstall 11m 51s root in branch-2 failed. +1 compile 0m 37s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 37s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 27s branch-2 passed +1 mvnsite 0m 44s branch-2 passed +1 findbugs 1m 28s branch-2 passed +1 javadoc 0m 26s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 27s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 38s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_131 -1 javac 0m 36s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131 with JDK v1.8.0_131 generated 1 new + 14 unchanged - 1 fixed = 15 total (was 15) +1 compile 0m 38s the patch passed with JDK v1.7.0_131 -1 javac 0m 38s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131 with JDK v1.7.0_131 generated 1 new + 13 unchanged - 1 fixed = 14 total (was 14) +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 40s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 43m 54s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 114m 0s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-6714 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877030/YARN-6714.branch-2.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 897f4b8a06ab 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / af8a227 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/16411/artifact/patchprocess/branch-mvninstall-root.txt findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/16411/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/16411/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16411/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16411/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16411/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Tao Yang Tao Yang added a comment -

        For the check javac warning, It seems that the custom generic types of SchedulerContainer<A extends SchedulerApplicationAttempt, N extends SchedulerNode> should be explicitly specified while creating a new instance in branch-2.
        Upload new patch to add the actual types:
        SchedulerContainer<FiCaSchedulerApp, FiCaSchedulerNode> reservedContainer = ...

        Show
        Tao Yang Tao Yang added a comment - For the check javac warning, It seems that the custom generic types of SchedulerContainer<A extends SchedulerApplicationAttempt, N extends SchedulerNode> should be explicitly specified while creating a new instance in branch-2. Upload new patch to add the actual types: SchedulerContainer<FiCaSchedulerApp, FiCaSchedulerNode> reservedContainer = ...
        Hide
        sunilg Sunil G added a comment -

        Tao Yang could you please help to check javac warning as per above jenkins run.

        Show
        sunilg Sunil G added a comment - Tao Yang could you please help to check javac warning as per above jenkins run.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2 Compile Tests
        -1 mvninstall 9m 34s root in branch-2 failed.
        +1 compile 0m 32s branch-2 passed with JDK v1.8.0_131
        +1 compile 0m 33s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 24s branch-2 passed
        +1 mvnsite 0m 40s branch-2 passed
        +1 findbugs 1m 15s branch-2 passed
        +1 javadoc 0m 22s branch-2 passed with JDK v1.8.0_131
        +1 javadoc 0m 25s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.8.0_131
        -1 javac 0m 26s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131 with JDK v1.8.0_131 generated 1 new + 14 unchanged - 1 fixed = 15 total (was 15)
        +1 compile 0m 31s the patch passed with JDK v1.7.0_131
        -1 javac 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131 with JDK v1.7.0_131 generated 1 new + 13 unchanged - 1 fixed = 14 total (was 14)
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 23s the patch passed
        +1 javadoc 0m 18s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 23s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 43m 48s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        110m 20s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue YARN-6714
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876819/YARN-6714.branch-2.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 54d452243b35 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 3419381
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/16380/artifact/patchprocess/branch-mvninstall-root.txt
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/16380/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/16380/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16380/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16380/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16380/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests -1 mvninstall 9m 34s root in branch-2 failed. +1 compile 0m 32s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 33s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 24s branch-2 passed +1 mvnsite 0m 40s branch-2 passed +1 findbugs 1m 15s branch-2 passed +1 javadoc 0m 22s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 25s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 32s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_131 -1 javac 0m 26s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131 with JDK v1.8.0_131 generated 1 new + 14 unchanged - 1 fixed = 15 total (was 15) +1 compile 0m 31s the patch passed with JDK v1.7.0_131 -1 javac 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131 with JDK v1.7.0_131 generated 1 new + 13 unchanged - 1 fixed = 14 total (was 14) +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 23s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 23s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 43m 48s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 110m 20s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-6714 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876819/YARN-6714.branch-2.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 54d452243b35 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 3419381 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/16380/artifact/patchprocess/branch-mvninstall-root.txt findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/16380/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/16380/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16380/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16380/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16380/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Tao Yang Tao Yang added a comment -

        Upload a patch for branch-2. Thanks Sunil G for review and committing.

        Show
        Tao Yang Tao Yang added a comment - Upload a patch for branch-2. Thanks Sunil G for review and committing.
        Hide
        sunilg Sunil G added a comment -

        Committed to trunk. Tao Yang please help to share branch-2 patch as its not cleanly applying.

        Show
        sunilg Sunil G added a comment - Committed to trunk. Tao Yang please help to share branch-2 patch as its not cleanly applying.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11986 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11986/)
        YARN-6714. IllegalStateException while handling APP_ATTEMPT_REMOVED (sunilg: rev 34f113df5cff2cc330fb671296932b8227b11975)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAsyncScheduling.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11986 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11986/ ) YARN-6714 . IllegalStateException while handling APP_ATTEMPT_REMOVED (sunilg: rev 34f113df5cff2cc330fb671296932b8227b11975) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAsyncScheduling.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        Hide
        sunilg Sunil G added a comment -

        Test case failures are unrelated. Also raised a ticket to handle FS test case failure.

        +1. I could commit this if there are no objection tomorrow.

        Show
        sunilg Sunil G added a comment - Test case failures are unrelated. Also raised a ticket to handle FS test case failure. +1. I could commit this if there are no objection tomorrow.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 39s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 14m 26s trunk passed
        +1 compile 0m 34s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 findbugs 1m 3s trunk passed
        +1 javadoc 0m 23s trunk passed
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 33s the patch passed
        +1 javac 0m 33s the patch passed
        +1 checkstyle 0m 24s the patch passed
        +1 mvnsite 0m 34s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 11s the patch passed
        +1 javadoc 0m 20s the patch passed
        -1 unit 43m 39s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        66m 56s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation
          hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6714
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874225/YARN-6714.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 08a97adf2341 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ee243e5
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16263/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16263/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16263/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 39s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 26s trunk passed +1 compile 0m 34s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 35s trunk passed +1 findbugs 1m 3s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 34s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 11s the patch passed +1 javadoc 0m 20s the patch passed -1 unit 43m 39s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 66m 56s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation   hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6714 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874225/YARN-6714.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 08a97adf2341 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ee243e5 Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/16263/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16263/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16263/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Changing the status to submit patch, as patch seems to be in decent state !

        Show
        Naganarasimha Naganarasimha G R added a comment - Changing the status to submit patch, as patch seems to be in decent state !
        Hide
        Tao Yang Tao Yang added a comment -

        Update the patch with adding comments for sanity check of attemptId. Thanks Sunil G for your suggestion.

        Show
        Tao Yang Tao Yang added a comment - Update the patch with adding comments for sanity check of attemptId. Thanks Sunil G for your suggestion.
        Hide
        sunilg Sunil G added a comment -

        Tao Yang
        I think your fix is correct. However could you add some more comments in code for future reference. It could help to know in which cases running attemptId may mismatch with app.getApplicationAttemptId().

        Show
        sunilg Sunil G added a comment - Tao Yang I think your fix is correct. However could you add some more comments in code for future reference. It could help to know in which cases running attemptId may mismatch with app.getApplicationAttemptId().
        Hide
        Tao Yang Tao Yang added a comment -

        Updated the patch with moving test case to TestCapacitySchedulerAsyncScheduling.

        Show
        Tao Yang Tao Yang added a comment - Updated the patch with moving test case to TestCapacitySchedulerAsyncScheduling.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Can I move these test cases to TestCapacitySchedulerAsyncScheduling but not enable async-scheduling ?

        This is OK to me.

        Show
        leftnoteasy Wangda Tan added a comment - Can I move these test cases to TestCapacitySchedulerAsyncScheduling but not enable async-scheduling ? This is OK to me.
        Hide
        Tao Yang Tao Yang added a comment - - edited

        Thanks Wangda Tan for reviewing the patch.

        Could you move test case from TestCapacityScheduler to TestCapacitySchedulerAsyncScheduling (same comment to YARN-6678 as well).

        I remembered why these test cases are not in TestCapacitySchedulerAsyncScheduling before, these cases are complex and hard to reproduce when async-scheduling enabled, for example, it's hard to allocate multiple containers as we need. Can I move these test cases to TestCapacitySchedulerAsyncScheduling but not enable async-scheduling ?

        could you file a separate JIRA for that? (And welcome if you can work on that ).

        I'm glad to work on that

        Show
        Tao Yang Tao Yang added a comment - - edited Thanks Wangda Tan for reviewing the patch. Could you move test case from TestCapacityScheduler to TestCapacitySchedulerAsyncScheduling (same comment to YARN-6678 as well). I remembered why these test cases are not in TestCapacitySchedulerAsyncScheduling before, these cases are complex and hard to reproduce when async-scheduling enabled, for example, it's hard to allocate multiple containers as we need. Can I move these test cases to TestCapacitySchedulerAsyncScheduling but not enable async-scheduling ? could you file a separate JIRA for that? (And welcome if you can work on that ). I'm glad to work on that
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Tao Yang again for investigations and working on the patch. Could you move test case from TestCapacityScheduler to TestCapacitySchedulerAsyncScheduling (same comment to YARN-6678 as well).

        The root cause of the issue is behavior of AbstractYarnScheduler#getApplicationAttempt is inconsistent to its name, it discarded application_attempt_id and always return the latest attempt. We should: 1) Rename it to getCurrentAttempt, 2) Change parameter from attemptId to applicationId. 3) Took a scan of all usages to see if any similar issue could happen.

        Tao Yang, could you file a separate JIRA for that? (And welcome if you can work on that ).

        + Sunil G.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Tao Yang again for investigations and working on the patch. Could you move test case from TestCapacityScheduler to TestCapacitySchedulerAsyncScheduling (same comment to YARN-6678 as well). The root cause of the issue is behavior of AbstractYarnScheduler#getApplicationAttempt is inconsistent to its name, it discarded application_attempt_id and always return the latest attempt. We should: 1) Rename it to getCurrentAttempt, 2) Change parameter from attemptId to applicationId. 3) Took a scan of all usages to see if any similar issue could happen. Tao Yang , could you file a separate JIRA for that? (And welcome if you can work on that ). + Sunil G .
        Hide
        Tao Yang Tao Yang added a comment -

        Attach a patch for review

        Show
        Tao Yang Tao Yang added a comment - Attach a patch for review

          People

          • Assignee:
            Tao Yang Tao Yang
            Reporter:
            Tao Yang Tao Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development