Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5068

Expose scheduler queue to application master

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The AM needs to know the queue name in which it was launched.

      1. MAPREDUCE-6692.patch
        5 kB
        Harish Jaiprakash
      2. YARN-5068.1.patch
        5 kB
        Harish Jaiprakash
      3. YARN-5068.2.patch
        5 kB
        Harish Jaiprakash
      4. YARN-5068-branch-2.1.patch
        5 kB
        Harish Jaiprakash

        Issue Links

          Activity

          Hide
          harishjp Harish Jaiprakash added a comment -

          Varun Vasudev I've made the changes please do review it. Tested it out by running a local instance and firing up an example tez job.

          Show
          harishjp Harish Jaiprakash added a comment - Varun Vasudev I've made the changes please do review it. Tested it out by running a local instance and firing up an example tez job.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Does ApplicationMaster really required to know queue name? Already ApplicationReport via ApplicationClientProtocol gives this details. Any specific use case can you explain?

          Show
          rohithsharma Rohith Sharma K S added a comment - Does ApplicationMaster really required to know queue name? Already ApplicationReport via ApplicationClientProtocol gives this details. Any specific use case can you explain?
          Hide
          vvasudev Varun Vasudev added a comment -

          Rohith Sharma K S I believe the reason is for post application analysis. The Tez UI uses the ATS to fetch information about the apps that were run. Without the AM storing the queue in the ATS, there is no way for the Tez UI to display the queues in which the apps ran. Harish Jaiprakash can correct me if I'm wrong.

          Show
          vvasudev Varun Vasudev added a comment - Rohith Sharma K S I believe the reason is for post application analysis. The Tez UI uses the ATS to fetch information about the apps that were run. Without the AM storing the queue in the ATS, there is no way for the Tez UI to display the queues in which the apps ran. Harish Jaiprakash can correct me if I'm wrong.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 8m 58s trunk passed
          +1 compile 3m 13s trunk passed with JDK v1.8.0_91
          +1 compile 2m 59s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 50s trunk passed
          +1 mvnsite 1m 15s trunk passed
          +1 mvneclipse 0m 34s trunk passed
          +1 findbugs 2m 49s trunk passed
          +1 javadoc 1m 23s trunk passed with JDK v1.8.0_91
          +1 javadoc 3m 59s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 7s the patch passed
          +1 compile 2m 45s the patch passed with JDK v1.8.0_91
          +1 javac 2m 45s the patch passed
          +1 compile 2m 39s the patch passed with JDK v1.7.0_95
          +1 javac 2m 39s the patch passed
          -1 checkstyle 0m 44s hadoop-yarn-project/hadoop-yarn: patch generated 3 new + 43 unchanged - 0 fixed = 46 total (was 43)
          +1 mvnsite 1m 12s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 16s the patch passed
          +1 javadoc 1m 18s the patch passed with JDK v1.8.0_91
          +1 javadoc 3m 50s the patch passed with JDK v1.7.0_95
          +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.8.0_91.
          -1 unit 35m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.7.0_95.
          -1 unit 36m 30s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          118m 59s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803168/MAPREDUCE-6692.patch
          JIRA Issue YARN-5068
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 991ca62a4568 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 87f5e35
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11397/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11397/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11397/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11397/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11397/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11397/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11397/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 8m 58s trunk passed +1 compile 3m 13s trunk passed with JDK v1.8.0_91 +1 compile 2m 59s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 50s trunk passed +1 mvnsite 1m 15s trunk passed +1 mvneclipse 0m 34s trunk passed +1 findbugs 2m 49s trunk passed +1 javadoc 1m 23s trunk passed with JDK v1.8.0_91 +1 javadoc 3m 59s trunk passed with JDK v1.7.0_95 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 7s the patch passed +1 compile 2m 45s the patch passed with JDK v1.8.0_91 +1 javac 2m 45s the patch passed +1 compile 2m 39s the patch passed with JDK v1.7.0_95 +1 javac 2m 39s the patch passed -1 checkstyle 0m 44s hadoop-yarn-project/hadoop-yarn: patch generated 3 new + 43 unchanged - 0 fixed = 46 total (was 43) +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 16s the patch passed +1 javadoc 1m 18s the patch passed with JDK v1.8.0_91 +1 javadoc 3m 50s the patch passed with JDK v1.7.0_95 +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.8.0_91. -1 unit 35m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.7.0_95. -1 unit 36m 30s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 118m 59s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803168/MAPREDUCE-6692.patch JIRA Issue YARN-5068 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 991ca62a4568 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 87f5e35 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11397/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11397/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11397/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11397/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11397/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11397/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/11397/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Right, ATS client to publish queue name is required which is not there currently.

          On the other hand, ATS stores 2 types of data

          1. generic-application-history includes RMApp specific details such as start time, queue name, priority and others.
          2. Application specific data where clients are published to ATS.
            Application specific data can be retrieved using REST service, so thinking why can't this service used?
          Show
          rohithsharma Rohith Sharma K S added a comment - Right, ATS client to publish queue name is required which is not there currently. On the other hand, ATS stores 2 types of data generic-application-history includes RMApp specific details such as start time, queue name, priority and others. Application specific data where clients are published to ATS. Application specific data can be retrieved using REST service, so thinking why can't this service used?
          Hide
          hitesh Hitesh Shah added a comment -

          The main use-case is this:

          • tez runs multiple dags in a single yarn application
          • for each dag, we publish data to yarn timeline and support searching for these dags based on the data published.
          • Timeline does not support doing searches which require a join between app-specific data and data published by the yarn framework.
          • Even though AHS has queue data, a single webservice call to ATS cannot be used to retrieve dags that were submitted to a particular queue.
          • To do this, we need to access the queue information in the AM and publish it along with the dag data as app-specific data and then use it for filtering.
          Show
          hitesh Hitesh Shah added a comment - The main use-case is this: tez runs multiple dags in a single yarn application for each dag, we publish data to yarn timeline and support searching for these dags based on the data published. Timeline does not support doing searches which require a join between app-specific data and data published by the yarn framework. Even though AHS has queue data, a single webservice call to ATS cannot be used to retrieve dags that were submitted to a particular queue. To do this, we need to access the queue information in the AM and publish it along with the dag data as app-specific data and then use it for filtering.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Thanks Hitesh Shah for providing detailed use-case.
          +1 LGTM, I will commit it if there is no more objections

          Show
          rohithsharma Rohith Sharma K S added a comment - Thanks Hitesh Shah for providing detailed use-case. +1 LGTM, I will commit it if there is no more objections
          Hide
          vvasudev Varun Vasudev added a comment -

          One minor fix from my end. Harish Jaiprakash - can you please change APP_QUEUE to YARN_RESOURCEMANAGER_APPLICATION_QUEUE?

          Show
          vvasudev Varun Vasudev added a comment - One minor fix from my end. Harish Jaiprakash - can you please change APP_QUEUE to YARN_RESOURCEMANAGER_APPLICATION_QUEUE?
          Hide
          harishjp Harish Jaiprakash added a comment -

          Changed the environment variable to YARN_RESOURCEMANAGER_APPLICATION_QUEUE.

          Show
          harishjp Harish Jaiprakash added a comment - Changed the environment variable to YARN_RESOURCEMANAGER_APPLICATION_QUEUE.
          Hide
          harishjp Harish Jaiprakash added a comment -

          Updated to fix checkstyle issues.

          Show
          harishjp Harish Jaiprakash added a comment - Updated to fix checkstyle issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 9m 24s trunk passed
          +1 compile 3m 28s trunk passed with JDK v1.8.0_91
          +1 compile 3m 1s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 50s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 findbugs 3m 4s trunk passed
          +1 javadoc 1m 26s trunk passed with JDK v1.8.0_91
          +1 javadoc 3m 55s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 5s the patch passed
          +1 compile 2m 46s the patch passed with JDK v1.8.0_91
          +1 javac 2m 46s the patch passed
          +1 compile 2m 37s the patch passed with JDK v1.7.0_95
          +1 javac 2m 37s the patch passed
          -1 checkstyle 0m 44s hadoop-yarn-project/hadoop-yarn: patch generated 5 new + 43 unchanged - 0 fixed = 48 total (was 43)
          +1 mvnsite 1m 13s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 16s the patch passed
          +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91
          +1 javadoc 3m 47s the patch passed with JDK v1.7.0_95
          +1 unit 0m 30s hadoop-yarn-api in the patch passed with JDK v1.8.0_91.
          -1 unit 35m 22s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.7.0_95.
          -1 unit 35m 53s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          119m 12s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803389/YARN-5068.1.patch
          JIRA Issue YARN-5068
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 96ffe2fad7f3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d971bf2
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11412/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11412/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11412/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11412/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11412/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11412/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11412/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 9m 24s trunk passed +1 compile 3m 28s trunk passed with JDK v1.8.0_91 +1 compile 3m 1s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 50s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 33s trunk passed +1 findbugs 3m 4s trunk passed +1 javadoc 1m 26s trunk passed with JDK v1.8.0_91 +1 javadoc 3m 55s trunk passed with JDK v1.7.0_95 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 5s the patch passed +1 compile 2m 46s the patch passed with JDK v1.8.0_91 +1 javac 2m 46s the patch passed +1 compile 2m 37s the patch passed with JDK v1.7.0_95 +1 javac 2m 37s the patch passed -1 checkstyle 0m 44s hadoop-yarn-project/hadoop-yarn: patch generated 5 new + 43 unchanged - 0 fixed = 48 total (was 43) +1 mvnsite 1m 13s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 16s the patch passed +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91 +1 javadoc 3m 47s the patch passed with JDK v1.7.0_95 +1 unit 0m 30s hadoop-yarn-api in the patch passed with JDK v1.8.0_91. -1 unit 35m 22s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.7.0_95. -1 unit 35m 53s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 119m 12s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803389/YARN-5068.1.patch JIRA Issue YARN-5068 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 96ffe2fad7f3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d971bf2 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11412/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11412/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11412/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11412/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11412/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11412/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/11412/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 6m 34s trunk passed
          +1 compile 1m 46s trunk passed with JDK v1.8.0_91
          +1 compile 2m 7s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 2m 13s trunk passed
          +1 javadoc 1m 0s trunk passed with JDK v1.8.0_91
          +1 javadoc 3m 21s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 53s the patch passed
          +1 compile 1m 49s the patch passed with JDK v1.8.0_91
          +1 javac 1m 49s the patch passed
          +1 compile 2m 5s the patch passed with JDK v1.7.0_95
          +1 javac 2m 5s the patch passed
          +1 checkstyle 0m 37s the patch passed
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 37s the patch passed
          +1 javadoc 0m 57s the patch passed with JDK v1.8.0_91
          +1 javadoc 3m 22s the patch passed with JDK v1.7.0_95
          +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_91.
          -1 unit 29m 22s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.7.0_95.
          -1 unit 30m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          95m 35s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803403/YARN-5068.2.patch
          JIRA Issue YARN-5068
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 013544a1977a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d971bf2
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11413/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11413/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11413/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11413/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11413/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11413/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 34s trunk passed +1 compile 1m 46s trunk passed with JDK v1.8.0_91 +1 compile 2m 7s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 13s trunk passed +1 javadoc 1m 0s trunk passed with JDK v1.8.0_91 +1 javadoc 3m 21s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 53s the patch passed +1 compile 1m 49s the patch passed with JDK v1.8.0_91 +1 javac 1m 49s the patch passed +1 compile 2m 5s the patch passed with JDK v1.7.0_95 +1 javac 2m 5s the patch passed +1 checkstyle 0m 37s the patch passed +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 37s the patch passed +1 javadoc 0m 57s the patch passed with JDK v1.8.0_91 +1 javadoc 3m 22s the patch passed with JDK v1.7.0_95 +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_91. -1 unit 29m 22s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.7.0_95. -1 unit 30m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 95m 35s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803403/YARN-5068.2.patch JIRA Issue YARN-5068 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 013544a1977a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d971bf2 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11413/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11413/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11413/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11413/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11413/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/11413/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Latest patch looks good to me. Rohith Sharma K S - is it fine by you?

          Show
          vvasudev Varun Vasudev added a comment - Latest patch looks good to me. Rohith Sharma K S - is it fine by you?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          +1lgtm

          Show
          rohithsharma Rohith Sharma K S added a comment - +1lgtm
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          committed to trunk!! Patch does not apply for branch-2, Harish Jaiprakash would you mind providing patch for branch-2 please?

          Show
          rohithsharma Rohith Sharma K S added a comment - committed to trunk!! Patch does not apply for branch-2, Harish Jaiprakash would you mind providing patch for branch-2 please?
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9750 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9750/)
          YARN-5068. Expose scheduler queue to application master. (Harish (rohithsharmaks: rev b7ac85259c7d20c33bef9c9cb40b8aabcab70755)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9750 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9750/ ) YARN-5068 . Expose scheduler queue to application master. (Harish (rohithsharmaks: rev b7ac85259c7d20c33bef9c9cb40b8aabcab70755) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
          Hide
          harishjp Harish Jaiprakash added a comment -

          Thanks Rohith. Created patch file for branch-2.

          Show
          harishjp Harish Jaiprakash added a comment - Thanks Rohith. Created patch file for branch-2.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          +1 for branch patch.. committing shortly

          Show
          rohithsharma Rohith Sharma K S added a comment - +1 for branch patch.. committing shortly
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          committed to trunk/branch-2.. thanks Harish Jaiprakash for the patch!! thanks Varun Vasudev for the review!

          Show
          rohithsharma Rohith Sharma K S added a comment - committed to trunk/branch-2.. thanks Harish Jaiprakash for the patch!! thanks Varun Vasudev for the review!
          Hide
          harishjp Harish Jaiprakash added a comment -

          Thanks Rohith. Is it possible to get this patch on 2.8.0 release?

          Show
          harishjp Harish Jaiprakash added a comment - Thanks Rohith. Is it possible to get this patch on 2.8.0 release?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          backported to 2.8.. Thanks!!

          Show
          rohithsharma Rohith Sharma K S added a comment - backported to 2.8.. Thanks!!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          It's a bit of an old JIRA but I just realized that YARN-1623 already added this information as part of RegisterApplicationMasterResponse. Shall we revert this change? /cc Rohith Sharma K S Varun Vasudev

          Also, this JIRA misses the case when an application is moved between queues. To address that, mirroring YARN-1623, we can also send the updated queue-name as part of AllocateResponse . The AM can see if the queue name changed between any subsequent calls and correspondingly record the same. /cc Junping Du

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - It's a bit of an old JIRA but I just realized that YARN-1623 already added this information as part of RegisterApplicationMasterResponse . Shall we revert this change? /cc Rohith Sharma K S Varun Vasudev Also, this JIRA misses the case when an application is moved between queues. To address that, mirroring YARN-1623 , we can also send the updated queue-name as part of AllocateResponse . The AM can see if the queue name changed between any subsequent calls and correspondingly record the same. /cc Junping Du
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Oh missed that RegisterApplicationMasterRespoonse has queue name! Since it is long time ago committed, if there any conflicts I will raise a new JIRA for reverting this and mirroring YARN-1623 solution in AllocateResponse for move queue.
          Since this JIRA is committed to 2.8 as well, reverting this change required to go to 2.8 as well otherwise compatibility issue would occur.

          Show
          rohithsharma Rohith Sharma K S added a comment - Oh missed that RegisterApplicationMasterRespoonse has queue name! Since it is long time ago committed, if there any conflicts I will raise a new JIRA for reverting this and mirroring YARN-1623 solution in AllocateResponse for move queue. Since this JIRA is committed to 2.8 as well, reverting this change required to go to 2.8 as well otherwise compatibility issue would occur.
          Hide
          djp Junping Du added a comment -

          Since this JIRA is committed to 2.8 as well, reverting this change required to go to 2.8 as well otherwise compatibility issue would occur.

          Agree. +1 to revert this patch from 2.8.0 as well. Thanks Vinod Kumar Vavilapalli for identify this issue and Rohith Sharma K S for working on it.

          Show
          djp Junping Du added a comment - Since this JIRA is committed to 2.8 as well, reverting this change required to go to 2.8 as well otherwise compatibility issue would occur. Agree. +1 to revert this patch from 2.8.0 as well. Thanks Vinod Kumar Vavilapalli for identify this issue and Rohith Sharma K S for working on it.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I created separate JIRA YARN-6200 for tracking and reverted in all the respective versions.

          Show
          rohithsharma Rohith Sharma K S added a comment - I created separate JIRA YARN-6200 for tracking and reverted in all the respective versions.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing this accurately as a dup of YARN-1623.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing this accurately as a dup of YARN-1623 .

            People

            • Assignee:
              harishjp Harish Jaiprakash
              Reporter:
              harishjp Harish Jaiprakash
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development