Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5575

Many classes use bare yarn. properties instead of the defined constants

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MAPREDUCE-5870 introduced the following line:

            conf.setInt("yarn.cluster.max-application-priority", 10);
      

      It should instead be:

            conf.setInt(YarnConfiguration.MAX_CLUSTER_LEVEL_APPLICATION_PRIORITY, 10);
      
      1. YARN-5575.003.patch
        48 kB
        Daniel Templeton
      2. YARN-5575.002.patch
        46 kB
        Daniel Templeton
      3. YARN-5575.001.patch
        47 kB
        Daniel Templeton

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10682 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10682/)
        YARN-5575. Many classes use bare yarn. properties instead of the defined (aajisaka: rev d3bb69a66776e9f410150c4030ddb15981f58fb9)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ahs/TestRMApplicationHistoryWriter.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMProxyUsersConf.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerQueueACLs.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerQueueACLs.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ReservationACLsTestBase.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesReservation.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRM.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesHttpStaticUserPermissions.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilterInitializer.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShellWithNodeLabels.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesAppsModification.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokenAuthentication.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
        • (edit) hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/GridmixTestUtils.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/ipc/TestMRCJCSocketFactory.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10682 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10682/ ) YARN-5575 . Many classes use bare yarn. properties instead of the defined (aajisaka: rev d3bb69a66776e9f410150c4030ddb15981f58fb9) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ahs/TestRMApplicationHistoryWriter.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMProxyUsersConf.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerQueueACLs.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerQueueACLs.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ReservationACLsTestBase.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesReservation.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRM.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesHttpStaticUserPermissions.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilterInitializer.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShellWithNodeLabels.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesAppsModification.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokenAuthentication.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java (edit) hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/GridmixTestUtils.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/ipc/TestMRCJCSocketFactory.java
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Committed this to trunk and branch-2. Thanks Daniel Templeton for the contribution and thanks Miklos Szegedi for the review!

        Show
        ajisakaa Akira Ajisaka added a comment - Committed this to trunk and branch-2. Thanks Daniel Templeton for the contribution and thanks Miklos Szegedi for the review!
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1 make sense to me. Committing this.

        Show
        ajisakaa Akira Ajisaka added a comment - +1 make sense to me. Committing this.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks, Akira Ajisaka. I'd rather not address those two issues as I'd be making awkward line splits to break up 81-character lines. I think in those two cases the cure is worse than the disease.

        Show
        templedf Daniel Templeton added a comment - Thanks, Akira Ajisaka . I'd rather not address those two issues as I'd be making awkward line splits to break up 81-character lines. I think in those two cases the cure is worse than the disease.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Hi Daniel Templeton, would you fix checkstyle warnings? I'm +1 if that is addressed. Thanks.

        Show
        ajisakaa Akira Ajisaka added a comment - Hi Daniel Templeton , would you fix checkstyle warnings? I'm +1 if that is addressed. Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 25 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 8m 19s trunk passed
        +1 compile 8m 28s trunk passed
        +1 checkstyle 1m 53s trunk passed
        +1 mvnsite 3m 6s trunk passed
        +1 mvneclipse 1m 41s trunk passed
        +1 findbugs 3m 45s trunk passed
        +1 javadoc 1m 44s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 2m 10s the patch passed
        +1 compile 7m 14s the patch passed
        +1 javac 7m 14s the patch passed
        -1 checkstyle 1m 50s root: The patch generated 2 new + 1765 unchanged - 57 fixed = 1767 total (was 1822)
        +1 mvnsite 2m 38s the patch passed
        +1 mvneclipse 1m 32s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 53s the patch passed
        +1 javadoc 1m 28s the patch passed
        -1 unit 3m 11s hadoop-yarn-server-applicationhistoryservice in the patch failed.
        +1 unit 38m 41s hadoop-yarn-server-resourcemanager in the patch passed.
        -1 unit 16m 15s hadoop-yarn-client in the patch failed.
        +1 unit 9m 25s hadoop-yarn-applications-distributedshell in the patch passed.
        -1 unit 119m 30s hadoop-mapreduce-client-jobclient in the patch failed.
        +1 unit 14m 11s hadoop-gridmix in the patch passed.
        +1 asflicense 0m 32s The patch does not generate ASF License warnings.
        253m 39s



        Reason Tests
        Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices
          hadoop.yarn.client.cli.TestLogsCLI
          hadoop.hdfs.TestNNBench



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834469/YARN-5575.003.patch
        JIRA Issue YARN-5575
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 13c0ad5f7647 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f872c6b
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13451/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-gridmix U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13451/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 25 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 8m 19s trunk passed +1 compile 8m 28s trunk passed +1 checkstyle 1m 53s trunk passed +1 mvnsite 3m 6s trunk passed +1 mvneclipse 1m 41s trunk passed +1 findbugs 3m 45s trunk passed +1 javadoc 1m 44s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 10s the patch passed +1 compile 7m 14s the patch passed +1 javac 7m 14s the patch passed -1 checkstyle 1m 50s root: The patch generated 2 new + 1765 unchanged - 57 fixed = 1767 total (was 1822) +1 mvnsite 2m 38s the patch passed +1 mvneclipse 1m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 53s the patch passed +1 javadoc 1m 28s the patch passed -1 unit 3m 11s hadoop-yarn-server-applicationhistoryservice in the patch failed. +1 unit 38m 41s hadoop-yarn-server-resourcemanager in the patch passed. -1 unit 16m 15s hadoop-yarn-client in the patch failed. +1 unit 9m 25s hadoop-yarn-applications-distributedshell in the patch passed. -1 unit 119m 30s hadoop-mapreduce-client-jobclient in the patch failed. +1 unit 14m 11s hadoop-gridmix in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 253m 39s Reason Tests Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices   hadoop.yarn.client.cli.TestLogsCLI   hadoop.hdfs.TestNNBench Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834469/YARN-5575.003.patch JIRA Issue YARN-5575 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 13c0ad5f7647 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f872c6b Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt https://builds.apache.org/job/PreCommit-YARN-Build/13451/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13451/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-gridmix U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13451/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        +1 (non-binding) The change looks good to me.
        Thank you, Daniel Templeton!

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - +1 (non-binding) The change looks good to me. Thank you, Daniel Templeton !
        Hide
        templedf Daniel Templeton added a comment -

        Cleaned up some checkstyle issues. The unit test failures appear unrelated.

        Show
        templedf Daniel Templeton added a comment - Cleaned up some checkstyle issues. The unit test failures appear unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 25 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 7m 6s trunk passed
        +1 compile 7m 16s trunk passed
        +1 checkstyle 1m 52s trunk passed
        +1 mvnsite 2m 45s trunk passed
        +1 mvneclipse 1m 36s trunk passed
        +1 findbugs 3m 14s trunk passed
        +1 javadoc 1m 27s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 2m 6s the patch passed
        +1 compile 7m 15s the patch passed
        +1 javac 7m 15s the patch passed
        -1 checkstyle 1m 52s root: The patch generated 14 new + 1793 unchanged - 29 fixed = 1807 total (was 1822)
        +1 mvnsite 2m 41s the patch passed
        +1 mvneclipse 1m 32s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 12s the patch passed
        +1 javadoc 1m 29s the patch passed
        -1 unit 3m 19s hadoop-yarn-server-applicationhistoryservice in the patch failed.
        +1 unit 39m 23s hadoop-yarn-server-resourcemanager in the patch passed.
        -1 unit 16m 12s hadoop-yarn-client in the patch failed.
        +1 unit 9m 41s hadoop-yarn-applications-distributedshell in the patch passed.
        +1 unit 106m 44s hadoop-mapreduce-client-jobclient in the patch passed.
        +1 unit 15m 28s hadoop-gridmix in the patch passed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        239m 55s



        Reason Tests
        Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices
          hadoop.yarn.client.cli.TestLogsCLI



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834245/YARN-5575.002.patch
        JIRA Issue YARN-5575
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 09f28523c556 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e9c4616
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13444/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt https://builds.apache.org/job/PreCommit-YARN-Build/13444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13444/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-gridmix U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13444/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 25 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 6s trunk passed +1 compile 7m 16s trunk passed +1 checkstyle 1m 52s trunk passed +1 mvnsite 2m 45s trunk passed +1 mvneclipse 1m 36s trunk passed +1 findbugs 3m 14s trunk passed +1 javadoc 1m 27s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 6s the patch passed +1 compile 7m 15s the patch passed +1 javac 7m 15s the patch passed -1 checkstyle 1m 52s root: The patch generated 14 new + 1793 unchanged - 29 fixed = 1807 total (was 1822) +1 mvnsite 2m 41s the patch passed +1 mvneclipse 1m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 12s the patch passed +1 javadoc 1m 29s the patch passed -1 unit 3m 19s hadoop-yarn-server-applicationhistoryservice in the patch failed. +1 unit 39m 23s hadoop-yarn-server-resourcemanager in the patch passed. -1 unit 16m 12s hadoop-yarn-client in the patch failed. +1 unit 9m 41s hadoop-yarn-applications-distributedshell in the patch passed. +1 unit 106m 44s hadoop-mapreduce-client-jobclient in the patch passed. +1 unit 15m 28s hadoop-gridmix in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 239m 55s Reason Tests Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices   hadoop.yarn.client.cli.TestLogsCLI Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834245/YARN-5575.002.patch JIRA Issue YARN-5575 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 09f28523c556 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e9c4616 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13444/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt https://builds.apache.org/job/PreCommit-YARN-Build/13444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13444/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-gridmix U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13444/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks for the review, Miklos Szegedi.

        The split lines in your first comment are split only because of line length. There's no particular convention to splitting a property access. We fit it on one line if we can, and split it if we can't.

        Good catch on the duplicated PREFIX. This patch resolves that issue.

        It also looks like I never submitted the first patch, so Jenkins never weighed in. I'll make sure to Jenkins runs this time.

        Show
        templedf Daniel Templeton added a comment - Thanks for the review, Miklos Szegedi . The split lines in your first comment are split only because of line length. There's no particular convention to splitting a property access. We fit it on one line if we can, and split it if we can't. Good catch on the duplicated PREFIX . This patch resolves that issue. It also looks like I never submitted the first patch, so Jenkins never weighed in. I'll make sure to Jenkins runs this time.
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        TestTimelineAuthenticationFilterInitializer.java has a new issue. PREFIX already contains http-authentication, so there will be an extra http-authentication in the configuration.
        Example: yarn.timeline-service.http-authentication.http-authentication.proxyuser.foo.hosts

          public static final String PREFIX = "yarn.timeline-service.http-authentication.";
        ...
                  // yarn.timeline-service.http-authentication.proxyuser prefix
                  conf.set(PREFIX + "http-authentication.proxyuser.foo.hosts", "*");
                  conf.set(PREFIX + "http-authentication.proxyuser.foo.users", "*");
                  conf.set(PREFIX + "http-authentication.proxyuser.foo.groups", "*");
                  break;
                case 2:
                  // hadoop.proxyuser prefix has been overwritten by
                  // yarn.timeline-service.http-authentication.proxyuser prefix
                  conf.set("hadoop.proxyuser.foo.hosts", "bar");
                  conf.set("hadoop.proxyuser.foo.users", "bar");
                  conf.set("hadoop.proxyuser.foo.groups", "bar");
                  conf.set(PREFIX + "http-authentication.proxyuser.foo.hosts", "*");
                  conf.set(PREFIX + "http-authentication.proxyuser.foo.users", "*");
                  conf.set(PREFIX + "http-authentication.proxyuser.foo.groups", "*");
                  break;
        
        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - TestTimelineAuthenticationFilterInitializer.java has a new issue. PREFIX already contains http-authentication, so there will be an extra http-authentication in the configuration. Example: yarn.timeline-service. http-authentication.http-authentication .proxyuser.foo.hosts public static final String PREFIX = "yarn.timeline-service.http-authentication." ; ... // yarn.timeline-service.http-authentication.proxyuser prefix conf.set(PREFIX + "http-authentication.proxyuser.foo.hosts" , "*" ); conf.set(PREFIX + "http-authentication.proxyuser.foo.users" , "*" ); conf.set(PREFIX + "http-authentication.proxyuser.foo.groups" , "*" ); break ; case 2: // hadoop.proxyuser prefix has been overwritten by // yarn.timeline-service.http-authentication.proxyuser prefix conf.set( "hadoop.proxyuser.foo.hosts" , "bar" ); conf.set( "hadoop.proxyuser.foo.users" , "bar" ); conf.set( "hadoop.proxyuser.foo.groups" , "bar" ); conf.set(PREFIX + "http-authentication.proxyuser.foo.hosts" , "*" ); conf.set(PREFIX + "http-authentication.proxyuser.foo.users" , "*" ); conf.set(PREFIX + "http-authentication.proxyuser.foo.groups" , "*" ); break ;
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        Hi Daniel Templeton,

        TestRMApplicationHistoryWriter.java line 392 needs to be split into two lines to follow the style of the other branch.

        The patch currently looks like:

            if (isFS) {
              conf.setBoolean(FairSchedulerConfiguration.ASSIGN_MULTIPLE, true);
              conf.set(YarnConfiguration.RM_SCHEDULER, FairScheduler.class.getName());
            } else {
              conf.set(YarnConfiguration.RM_SCHEDULER,
                  CapacityScheduler.class.getName());
            }
        
        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Hi Daniel Templeton , TestRMApplicationHistoryWriter.java line 392 needs to be split into two lines to follow the style of the other branch. The patch currently looks like: if (isFS) { conf.setBoolean(FairSchedulerConfiguration.ASSIGN_MULTIPLE, true ); conf.set(YarnConfiguration.RM_SCHEDULER, FairScheduler.class.getName()); } else { conf.set(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class.getName()); }
        Hide
        templedf Daniel Templeton added a comment -

        This patch addresses all bare yarn.* configuration properties. In cleaning these up, I found a number of suspicious property uses. (See YARN-5572, YARN-5573, and YARN-5574.) Those I left as-is until the validity of the properties can be verified.

        Show
        templedf Daniel Templeton added a comment - This patch addresses all bare yarn.* configuration properties. In cleaning these up, I found a number of suspicious property uses. (See YARN-5572 , YARN-5573 , and YARN-5574 .) Those I left as-is until the validity of the properties can be verified.
        Hide
        templedf Daniel Templeton added a comment -

        For posterity's sake, here's the current culprits:

        bash-3.2$ find . -name '*.java' -exec grep -lsE 'conf\.set[^()]+\s*\(\s*"' {} \; | grep -v src/test
        ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
        ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Factory.java
        ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibFactory.java
        ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Compression.java
        ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
        ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
        ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainMapper.java
        ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainReducer.java
        ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        ./hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java
        
        Show
        templedf Daniel Templeton added a comment - For posterity's sake, here's the current culprits: bash-3.2$ find . -name '*.java' -exec grep -lsE 'conf\.set[^()]+\s*\(\s*"' {} \; | grep -v src/test ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Factory.java ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibFactory.java ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Compression.java ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainMapper.java ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainReducer.java ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java ./hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java
        Hide
        templedf Daniel Templeton added a comment -

        I just did a regex search through the code, and these bare string properties appear all over the place. There are a ridiculous number in the test classes, but still a good number in the product code. When I sit down to this one, I'll see if I have the stomach to take on all the product code cases.

        Show
        templedf Daniel Templeton added a comment - I just did a regex search through the code, and these bare string properties appear all over the place. There are a ridiculous number in the test classes, but still a good number in the product code. When I sit down to this one, I'll see if I have the stomach to take on all the product code cases.
        Hide
        neelesh77 Neelesh Srinivas Salian added a comment -

        Hi Daniel Templeton

        Are there other fields in methods/files that could use this change?
        This makes sense only in cases (as you pointed out) as long as the values are static/default.
        Maybe good to broaden this to other file(s) if needed.

        Show
        neelesh77 Neelesh Srinivas Salian added a comment - Hi Daniel Templeton Are there other fields in methods/files that could use this change? This makes sense only in cases (as you pointed out) as long as the values are static/default. Maybe good to broaden this to other file(s) if needed.

          People

          • Assignee:
            templedf Daniel Templeton
            Reporter:
            templedf Daniel Templeton
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development