Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6082

Invalid REST api response for getApps since queueUsagePercentage is coming as INF

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.3
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      2017-01-11 07:17:11,475 WARN  ipc.Server (Server.java:run(2202)) - Large response size 4476919 for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 172.27.0.101:39950 Call#951474 Retry#0
      

      Above server log is coming and JSON o/p was coming invalid

      For eg:

      Unexpected token I in JSON at position 851
      
      1. YARN-6082.0001.patch
        3 kB
        Sunil G
      2. YARN-6082.0002.patch
        3 kB
        Sunil G

        Activity

        Hide
        sunilg Sunil G added a comment -

        Thanks Rohith Sharma K S for review and commit.

        Show
        sunilg Sunil G added a comment - Thanks Rohith Sharma K S for review and commit.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11165 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11165/)
        YARN-6082. Invalid REST api response for getApps since (rohithsharmaks: rev 0101267d9d801eab4cb3b4df289c402ecb591685)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11165 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11165/ ) YARN-6082 . Invalid REST api response for getApps since (rohithsharmaks: rev 0101267d9d801eab4cb3b4df289c402ecb591685) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        committed to trunk/branch-2.. thanks Sunil G for contributions.

        Show
        rohithsharma Rohith Sharma K S added a comment - committed to trunk/branch-2.. thanks Sunil G for contributions.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        +1 LGTM

        Show
        rohithsharma Rohith Sharma K S added a comment - +1 LGTM
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 17m 2s trunk passed
        +1 compile 0m 42s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 45s trunk passed
        +1 mvneclipse 0m 19s trunk passed
        +1 findbugs 1m 18s trunk passed
        +1 javadoc 0m 27s trunk passed
        +1 mvninstall 0m 42s the patch passed
        +1 compile 0m 35s the patch passed
        +1 javac 0m 35s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 0m 39s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 20s the patch passed
        +1 javadoc 0m 24s the patch passed
        -1 unit 39m 32s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        66m 52s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6082
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848858/YARN-6082.0002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 251d049b2403 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3fa0d54
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14731/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14731/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14731/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 17m 2s trunk passed +1 compile 0m 42s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 18s trunk passed +1 javadoc 0m 27s trunk passed +1 mvninstall 0m 42s the patch passed +1 compile 0m 35s the patch passed +1 javac 0m 35s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 39s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 20s the patch passed +1 javadoc 0m 24s the patch passed -1 unit 39m 32s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 66m 52s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6082 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848858/YARN-6082.0002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 251d049b2403 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fa0d54 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/14731/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14731/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14731/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Thanks Rohith Sharma K S

        Updating new patch addressing the comments.

        Show
        sunilg Sunil G added a comment - Thanks Rohith Sharma K S Updating new patch addressing the comments.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        overall logically patch looks good to me ..
        nit : could queueCapacityPerc calculation bring inside if check?

        if (!calc.isInvalidDivisor(cluster)) {
        float queueCapacityPerc = queue.getQueueInfo(false, false).getCapacity();
        }
        
        Show
        rohithsharma Rohith Sharma K S added a comment - overall logically patch looks good to me .. nit : could queueCapacityPerc calculation bring inside if check? if (!calc.isInvalidDivisor(cluster)) { float queueCapacityPerc = queue.getQueueInfo( false , false ).getCapacity(); }
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 13m 9s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 48s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 1m 9s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 34s the patch passed
        +1 javac 0m 34s the patch passed
        -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 48 unchanged - 0 fixed = 49 total (was 48)
        +1 mvnsite 0m 34s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 11s the patch passed
        +1 javadoc 0m 21s the patch passed
        -1 unit 40m 24s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        62m 48s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6082
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848591/YARN-6082.0001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 54f5682a4737 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3fa0d54
        Default Java 1.8.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14730/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14730/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14730/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14730/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 9s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 48s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 9s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 48 unchanged - 0 fixed = 49 total (was 48) +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 11s the patch passed +1 javadoc 0m 21s the patch passed -1 unit 40m 24s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 62m 48s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6082 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848591/YARN-6082.0001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 54f5682a4737 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fa0d54 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14730/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14730/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14730/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14730/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Attaching an initial patch. Also added a test case.

        Since base capacity is 0, we can mark queueUsedPercentage as 0 I think. Thoughts? cc/ Wangda Tan Rohith Sharma K S

        Show
        sunilg Sunil G added a comment - Attaching an initial patch. Also added a test case. Since base capacity is 0, we can mark queueUsedPercentage as 0 I think. Thoughts? cc/ Wangda Tan Rohith Sharma K S
        Hide
        sunilg Sunil G added a comment -

        Yes Vinod Kumar Vavilapalli. You are correct. Both are un related.

        Analysis

        In SchedulerApplicationAttempt#getResourceUsageReport, QueueUsagePercent is calculated as follows

            float queueUsagePerc = calc.divide(cluster, usedResourceClone, Resources
                .multiply(cluster, queue.getQueueInfo(false, false).getCapacity()))
                * 100;
        

        Hence if queue.getQueueInfo(false, false).getCapacity()) is 0, we will get divide by 0 error (INF). hence queueUsagePercentage will come as INF and will corrupt the JSON o/p.

        Show
        sunilg Sunil G added a comment - Yes Vinod Kumar Vavilapalli . You are correct. Both are un related. Analysis In SchedulerApplicationAttempt#getResourceUsageReport , QueueUsagePercent is calculated as follows float queueUsagePerc = calc.divide(cluster, usedResourceClone, Resources .multiply(cluster, queue.getQueueInfo( false , false ).getCapacity())) * 100; Hence if queue.getQueueInfo(false, false).getCapacity()) is 0, we will get divide by 0 error (INF). hence queueUsagePercentage will come as INF and will corrupt the JSON o/p.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        The large-response error is happening on the RPC server. Don't think they are related.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - The large-response error is happening on the RPC server. Don't think they are related.

          People

          • Assignee:
            sunilg Sunil G
            Reporter:
            sunilg Sunil G
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development