Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7076

yarn application -list -appTypes <appType> is not working

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: None
    • Labels:
      None

      Description

      yarn application -list -appTypes <appType> is not working

      Looks like it's because the ApplicationCLI pass in the appType as uppercase, but ClientRMService#getApplications is case sensitive, so if user submits an app with lowercase appType, it wont work

      1. YARN-7076.01.patch
        7 kB
        Jian He
      2. YARN-7076.02.patch
        8 kB
        Jian He

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12257 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12257/)
        YARN-7076. yarn application -list -appTypes is not working. Contributed (junping_du: rev 312b1fd9da2781da97f8c76fe1262c4d99b9c37f)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12257 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12257/ ) YARN-7076 . yarn application -list -appTypes is not working. Contributed (junping_du: rev 312b1fd9da2781da97f8c76fe1262c4d99b9c37f) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        Hide
        djp Junping Du added a comment -

        btw, personally, I don't think it will be an incompatible issue as case-sensitive never work for CLI or REST API and ClientRMService is a private API so it seems nothing get break here with the patch. So I didn't put incompatible tag on this jira.

        Show
        djp Junping Du added a comment - btw, personally, I don't think it will be an incompatible issue as case-sensitive never work for CLI or REST API and ClientRMService is a private API so it seems nothing get break here with the patch. So I didn't put incompatible tag on this jira.
        Hide
        djp Junping Du added a comment -

        I have commit the patch to trunk, branch-2, branch-2.8 and branch-2.8.2. Thanks Jian He for reporting the issue and delivering a fix. Also thanks Jason Lowe for comments and suggestions.

        Show
        djp Junping Du added a comment - I have commit the patch to trunk, branch-2, branch-2.8 and branch-2.8.2. Thanks Jian He for reporting the issue and delivering a fix. Also thanks Jason Lowe for comments and suggestions.
        Hide
        djp Junping Du added a comment -

        Patch LGTM. +1. Committing it now.

        Show
        djp Junping Du added a comment - Patch LGTM. +1. Committing it now.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 16m 51s trunk passed
        +1 compile 0m 49s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 findbugs 1m 24s trunk passed
        +1 javadoc 0m 29s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 31s the patch passed
        +1 mvnsite 0m 47s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 33s the patch passed
        +1 javadoc 0m 27s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 343 unchanged - 4 fixed = 343 total (was 347)
              Other Tests
        -1 unit 44m 29s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 13s The patch does not generate ASF License warnings.
        72m 23s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-7076
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884134/YARN-7076.02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8d8ae2ece1df 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 51881a8
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-YARN-Build/17165/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17165/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17165/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 51s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 52s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 29s trunk passed       Patch Compile Tests +1 mvninstall 0m 48s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 31s the patch passed +1 mvnsite 0m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 27s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 343 unchanged - 4 fixed = 343 total (was 347)       Other Tests -1 unit 44m 29s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 13s The patch does not generate ASF License warnings. 72m 23s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-7076 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884134/YARN-7076.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8d8ae2ece1df 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 51881a8 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/17165/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17165/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/17165/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        Thanks Jian He for updating the patch. 02 patch LGTM. +1 based on Jenkins report.

        Show
        djp Junping Du added a comment - Thanks Jian He for updating the patch. 02 patch LGTM. +1 based on Jenkins report.
        Hide
        djp Junping Du added a comment -

        Thanks Jian He for delivering a quick fix. The patch looks general OK to me.
        A minor issue here is we better to add a UT to test ClientRMService is case insensitive now - a simple way is to add something similar to below code in TestClientRMService.java but using all upper case or/and lower case of "matchType".

            Set<String> appTypes = new HashSet<String>();
            appTypes.add("matchType");
        
            getAllAppsRequest = GetApplicationsRequest.newInstance(appTypes);
            getAllApplicationsResponse =
                rmService.getApplications(getAllAppsRequest);
            Assert.assertEquals(1,
                getAllApplicationsResponse.getApplicationList().size());
        
        Show
        djp Junping Du added a comment - Thanks Jian He for delivering a quick fix. The patch looks general OK to me. A minor issue here is we better to add a UT to test ClientRMService is case insensitive now - a simple way is to add something similar to below code in TestClientRMService.java but using all upper case or/and lower case of "matchType". Set<String> appTypes = new HashSet<String>(); appTypes.add("matchType"); getAllAppsRequest = GetApplicationsRequest.newInstance(appTypes); getAllApplicationsResponse = rmService.getApplications(getAllAppsRequest); Assert.assertEquals(1, getAllApplicationsResponse.getApplicationList().size());
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 30s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 19m 27s trunk passed
        +1 compile 0m 46s trunk passed
        +1 checkstyle 0m 33s trunk passed
        +1 mvnsite 0m 49s trunk passed
        +1 findbugs 1m 21s trunk passed
        +1 javadoc 0m 34s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 42s the patch passed
        +1 compile 0m 40s the patch passed
        +1 javac 0m 40s the patch passed
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 0m 43s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 24s the patch passed
        +1 javadoc 0m 26s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 343 unchanged - 4 fixed = 343 total (was 347)
              Other Tests
        -1 unit 53m 20s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 25s The patch does not generate ASF License warnings.
        83m 45s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
        Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands
          org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
          org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-7076
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884093/YARN-7076.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 15a3506d879e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 51881a8
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-YARN-Build/17155/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17155/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17155/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 19m 27s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 49s trunk passed +1 findbugs 1m 21s trunk passed +1 javadoc 0m 34s trunk passed       Patch Compile Tests +1 mvninstall 0m 42s the patch passed +1 compile 0m 40s the patch passed +1 javac 0m 40s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 24s the patch passed +1 javadoc 0m 26s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 343 unchanged - 4 fixed = 343 total (was 347)       Other Tests -1 unit 53m 20s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 83m 45s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands   org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-7076 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884093/YARN-7076.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 15a3506d879e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 51881a8 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/17155/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17155/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/17155/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        attached a patch which makes ClientRMService#getApplications to be case insensitive for applicationTypes

        Show
        jianhe Jian He added a comment - attached a patch which makes ClientRMService#getApplications to be case insensitive for applicationTypes
        Hide
        djp Junping Du added a comment -

        Forget about my comments above, applicaitonState and other elements should be case insensitive. So +1 to Jason's proposal that make it case insensitive.

        Show
        djp Junping Du added a comment - Forget about my comments above, applicaitonState and other elements should be case insensitive. So +1 to Jason's proposal that make it case insensitive.
        Hide
        djp Junping Du added a comment -

        It sounds like submit an application against appType with lowercase (no need for all) never work for list in command line or REST API, so this is more like a bug rather than incompatible issue.
        By get rid of incompatible, I think making case-sensitive in all places could be better given other elements like applicationState, etc. is case sensitive. May be we should keep consistent here?

        Show
        djp Junping Du added a comment - It sounds like submit an application against appType with lowercase (no need for all) never work for list in command line or REST API, so this is more like a bug rather than incompatible issue. By get rid of incompatible, I think making case-sensitive in all places could be better given other elements like applicationState, etc. is case sensitive. May be we should keep consistent here?
        Hide
        jlowe Jason Lowe added a comment -

        Seems like we're going to be technically incompatible no matter how we fix this. Either we make ClientRMService#getApplications be case-insensitive which theoretically breaks someone who was leveraging case-sensitivity via that API call, or we change the CLI to be case-sensitive in its inputs. The latter is probably going to be more surprising than the latter, so I propose we change ClientRMService#getApplications to be case-insensitive. Also it appears the equivalent API call in the web services already expects appType matching to be case-insensitive, so changing this to be case-insensitive would make it consistent.

        Show
        jlowe Jason Lowe added a comment - Seems like we're going to be technically incompatible no matter how we fix this. Either we make ClientRMService#getApplications be case-insensitive which theoretically breaks someone who was leveraging case-sensitivity via that API call, or we change the CLI to be case-sensitive in its inputs. The latter is probably going to be more surprising than the latter, so I propose we change ClientRMService#getApplications to be case-insensitive. Also it appears the equivalent API call in the web services already expects appType matching to be case-insensitive, so changing this to be case-insensitive would make it consistent.

          People

          • Assignee:
            jianhe Jian He
            Reporter:
            jianhe Jian He
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development