Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3069

Document missing properties in yarn-default.xml

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Documented missing properties and added the regression test to verify that there are no missing properties in yarn-default.xml.

      Description

      The following properties are currently not defined in yarn-default.xml. These properties should either be

      A) documented in yarn-default.xml OR
      B) listed as an exception (with comments, e.g. for internal use) in the TestYarnConfigurationFields unit test

      Any comments for any of the properties below are welcome.

      org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker
      org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore
      security.applicationhistory.protocol.acl
      yarn.app.container.log.backups
      yarn.app.container.log.dir
      yarn.app.container.log.filesize
      yarn.client.app-submission.poll-interval
      yarn.client.application-client-protocol.poll-timeout-ms
      yarn.is.minicluster
      yarn.log.server.url
      yarn.minicluster.control-resource-monitoring
      yarn.minicluster.fixed.ports
      yarn.minicluster.use-rpc
      yarn.node-labels.fs-store.retry-policy-spec
      yarn.node-labels.fs-store.root-dir
      yarn.node-labels.manager-class
      yarn.nodemanager.container-executor.os.sched.priority.adjustment
      yarn.nodemanager.container-monitor.process-tree.class
      yarn.nodemanager.disk-health-checker.enable
      yarn.nodemanager.docker-container-executor.image-name
      yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms
      yarn.nodemanager.linux-container-executor.group
      yarn.nodemanager.log.deletion-threads-count
      yarn.nodemanager.user-home-dir
      yarn.nodemanager.webapp.https.address
      yarn.nodemanager.webapp.spnego-keytab-file
      yarn.nodemanager.webapp.spnego-principal
      yarn.nodemanager.windows-secure-container-executor.group
      yarn.resourcemanager.configuration.file-system-based-store
      yarn.resourcemanager.delegation-token-renewer.thread-count
      yarn.resourcemanager.delegation.key.update-interval
      yarn.resourcemanager.delegation.token.max-lifetime
      yarn.resourcemanager.delegation.token.renew-interval
      yarn.resourcemanager.history-writer.multi-threaded-dispatcher.pool-size
      yarn.resourcemanager.metrics.runtime.buckets
      yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs
      yarn.resourcemanager.reservation-system.class
      yarn.resourcemanager.reservation-system.enable
      yarn.resourcemanager.reservation-system.plan.follower
      yarn.resourcemanager.reservation-system.planfollower.time-step
      yarn.resourcemanager.rm.container-allocation.expiry-interval-ms
      yarn.resourcemanager.webapp.spnego-keytab-file
      yarn.resourcemanager.webapp.spnego-principal
      yarn.scheduler.include-port-in-node-name
      yarn.timeline-service.delegation.key.update-interval
      yarn.timeline-service.delegation.token.max-lifetime
      yarn.timeline-service.delegation.token.renew-interval
      yarn.timeline-service.generic-application-history.enabled
      yarn.timeline-service.generic-application-history.fs-history-store.compression-type
      yarn.timeline-service.generic-application-history.fs-history-store.uri
      yarn.timeline-service.generic-application-history.store-class
      yarn.timeline-service.http-cross-origin.enabled
      yarn.tracking.url.generator

      1. YARN-3069.001.patch
        11 kB
        Ray Chiang
      2. YARN-3069.002.patch
        22 kB
        Ray Chiang
      3. YARN-3069.003.patch
        21 kB
        Ray Chiang
      4. YARN-3069.004.patch
        22 kB
        Ray Chiang
      5. YARN-3069.005.patch
        23 kB
        Ray Chiang
      6. YARN-3069.006.patch
        35 kB
        Ray Chiang
      7. YARN-3069.007.patch
        24 kB
        Ray Chiang
      8. YARN-3069.008.patch
        24 kB
        Ray Chiang
      9. YARN-3069.009.patch
        26 kB
        Ray Chiang
      10. YARN-3069.010.patch
        26 kB
        Ray Chiang
      11. YARN-3069.011.patch
        23 kB
        Ray Chiang
      12. YARN-3069.012.patch
        23 kB
        Ray Chiang
      13. YARN-3069.013.patch
        23 kB
        Ray Chiang

        Issue Links

          Activity

          Hide
          ctrezzo Chris Trezzo added a comment -

          Ray Chiang Which properties are you referring to for org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker and org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore? These are both default values for properties yarn.sharedcache.store.class and yarn.sharedcache.app-checker.class. I see both in yarn-default.xml and YarnConfiguration.java on trunk. Thanks!

          Show
          ctrezzo Chris Trezzo added a comment - Ray Chiang Which properties are you referring to for org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker and org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore? These are both default values for properties yarn.sharedcache.store.class and yarn.sharedcache.app-checker.class. I see both in yarn-default.xml and YarnConfiguration.java on trunk. Thanks!
          Hide
          rchiang Ray Chiang added a comment -

          Yeah, the checking code is a bit rough. It looks through YarnConfiguration looking for any "public final static string" and assumes that it's a valid property. I'll add

          DEFAULT_SCM_STORE_CLASS
          DEFAULT_SCM_APP_CHECKER_CLASS

          to the list of exceptions, since those are valid config values and not properties.

          Thanks for letting me know!

          Show
          rchiang Ray Chiang added a comment - Yeah, the checking code is a bit rough. It looks through YarnConfiguration looking for any "public final static string" and assumes that it's a valid property. I'll add DEFAULT_SCM_STORE_CLASS DEFAULT_SCM_APP_CHECKER_CLASS to the list of exceptions, since those are valid config values and not properties. Thanks for letting me know!
          Hide
          rchiang Ray Chiang added a comment -

          Initial version:

          • Make section separations consistent
          • Put in all known missing properties. Some may need to be moved to TestYarnConfigurationFields as an exception instead of a documented property.
          • Currently, all new properties are undocumented, but do exists within the yarn-default.xml file.
          Show
          rchiang Ray Chiang added a comment - Initial version: Make section separations consistent Put in all known missing properties. Some may need to be moved to TestYarnConfigurationFields as an exception instead of a documented property. Currently, all new properties are undocumented, but do exists within the yarn-default.xml file.
          Hide
          rchiang Ray Chiang added a comment -

          Merge in unit test code from YARN-3107 for testing.

          Show
          rchiang Ray Chiang added a comment - Merge in unit test code from YARN-3107 for testing.
          Hide
          rchiang Ray Chiang added a comment -
          • Added minimal comments to most of the missing properties.
          • Could use help adding descriptions to any remaining properties with empty description sections.
          • Turn on error checking in both directions (XML->Java, Java->XML)
          Show
          rchiang Ray Chiang added a comment - Added minimal comments to most of the missing properties. Could use help adding descriptions to any remaining properties with empty description sections. Turn on error checking in both directions (XML->Java, Java->XML)
          Hide
          rchiang Ray Chiang added a comment -

          Submit for testing

          Show
          rchiang Ray Chiang added a comment - Submit for testing
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12706130/YARN-3069.003.patch
          against trunk revision e1feb4e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7062//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7062//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706130/YARN-3069.003.patch against trunk revision e1feb4e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7062//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7062//console This message is automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          Add exceptions for the yarn.nodemanager.resource.* properties.

          Show
          rchiang Ray Chiang added a comment - Add exceptions for the yarn.nodemanager.resource.* properties.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 30s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 3m 57s There were no new checkstyle issues.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
              41m 32s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728322/YARN-3069.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 618ba70
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7509/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7509/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7509/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 30s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 3m 57s There were no new checkstyle issues. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.     41m 32s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728322/YARN-3069.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 618ba70 hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7509/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7509/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7509/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Ray Chiang for creating the patch! Reviewed properties from yarn.resourcemanager.webapp.spnego-keytab-file to yarn.resourcemanager.configuration.file-system-based-store.

          yarn.resourcemanager.webapp.ui-actions.enabled

          Default value is true.

          yarn.resourcemanager.configuration.file-system-based-store

          Default value is /yarn/conf. The value specifies the file system (e.g. HDFS) path where ResourceManager loads configuration if yarn.resourcemanager.configuration.provider-class is set to org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider. Please see YARN-1611 for details. I'm thinking it's better to add the section next to yarn.resourcemanager.configuration.provider-class.

          I'll review the rest of the patch.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Ray Chiang for creating the patch! Reviewed properties from yarn.resourcemanager.webapp.spnego-keytab-file to yarn.resourcemanager.configuration.file-system-based-store. yarn.resourcemanager.webapp.ui-actions.enabled Default value is true. yarn.resourcemanager.configuration.file-system-based-store Default value is /yarn/conf . The value specifies the file system (e.g. HDFS) path where ResourceManager loads configuration if yarn.resourcemanager.configuration.provider-class is set to org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider . Please see YARN-1611 for details. I'm thinking it's better to add the section next to yarn.resourcemanager.configuration.provider-class . I'll review the rest of the patch.
          Hide
          rchiang Ray Chiang added a comment -

          Thanks Akira! I've made those changes. I definitely left some empty descriptions in yarn-default.xml where I couldn't figure out what the property was for.

          I'll wait for more of your review before uploading a new patch.

          Show
          rchiang Ray Chiang added a comment - Thanks Akira! I've made those changes. I definitely left some empty descriptions in yarn-default.xml where I couldn't figure out what the property was for. I'll wait for more of your review before uploading a new patch.
          Hide
          rchiang Ray Chiang added a comment -

          Updates for 3 properties.

          Show
          rchiang Ray Chiang added a comment - Updates for 3 properties.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 46s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 43s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 52s There were no new checkstyle issues.
          -1 whitespace 0m 1s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common.
              38m 49s  



          Reason Tests
          Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730122/YARN-3069.005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a319771
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7682/artifact/patchprocess/whitespace.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7682/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7682/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7682/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 46s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 43s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 52s There were no new checkstyle issues. -1 whitespace 0m 1s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common.     38m 49s   Reason Tests Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730122/YARN-3069.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a319771 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7682/artifact/patchprocess/whitespace.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7682/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7682/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7682/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -
          • Fix whitespace issues
          • Fix new property introduced with YARN-2619
          Show
          rchiang Ray Chiang added a comment - Fix whitespace issues Fix new property introduced with YARN-2619
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 46s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 7m 38s There were no new javac warning messages.
          +1 javadoc 9m 42s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 4m 45s There were no new checkstyle issues.
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 39s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 7m 10s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 23m 32s Tests passed in hadoop-common.
          +1 mapreduce tests 9m 42s Tests passed in hadoop-mapreduce-client-app.
          +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common.
          -1 hdfs tests 164m 48s Tests failed in hadoop-hdfs.
              246m 47s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestFileCreation
            hadoop.hdfs.TestHDFSFileSystemContract



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730267/YARN-3069.006.patch
          Optional Tests javac unit findbugs checkstyle javadoc
          git revision trunk / bf70c5a
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7691/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-YARN-Build/7691/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7691/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-YARN-Build/7691/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7691/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7691/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 46s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 38s There were no new javac warning messages. +1 javadoc 9m 42s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 4m 45s There were no new checkstyle issues. +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 7m 10s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 32s Tests passed in hadoop-common. +1 mapreduce tests 9m 42s Tests passed in hadoop-mapreduce-client-app. +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common. -1 hdfs tests 164m 48s Tests failed in hadoop-hdfs.     246m 47s   Reason Tests Failed unit tests hadoop.hdfs.TestFileCreation   hadoop.hdfs.TestHDFSFileSystemContract Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730267/YARN-3069.006.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / bf70c5a hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7691/artifact/patchprocess/testrun_hadoop-common.txt hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-YARN-Build/7691/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7691/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-YARN-Build/7691/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7691/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7691/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          RE: Failing unit tests

          Both tests pass in my tree.

          Show
          rchiang Ray Chiang added a comment - RE: Failing unit tests Both tests pass in my tree.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Ray Chiang for updating the patch. Reviewed parameters from yarn.resourcemanager.delegation-token-renewer.thread-count to mapreduce.job.hdfs-servers.

          yarn.resourcemanager.delegation.token.max-lifetime

          Would you add the time unit (ms) in the description?

          yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs

          Default value is 86400 (24*60*60) sec.

          yarn.resourcemanager.reservation-system.class

          By default, org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityReservationSystem is used when using CapacityScheduler, and org.apache.hadoop.yarn.server.resourcemanager.reservation.FairReservationSystem is used when using FairScheduler.

          yarn.resourcemanager.reservation-system.plan.follower

          By default, org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacitySchedulerPlanFollower is used when using CapacityScheduler, and org.apache.hadoop.yarn.server.resourcemanager.reservation.FairSchedulerPlanFollower is used when using FairScheduler.

          +   <name>yarn.nodemanager.linux-container-executor.cgroups.delete-delay-ms</name>
          +   <value>20></value>
          

          Misformatted value tag.

          yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms

          Default value is 1000.

          yarn.nodemanager.linux-container-executor.group

          I don't think the parameter should be on yarn-default.xml since the parameter is set by container-executor.cfg.

          yarn.nodemanager.user-home-dir

          The home directory of the user for containers. Default value is "/home/".

          -   <name>mapreduce.job.hdfs-servers</name>		
          -   <value>${fs.defaultFS}</value>
          

          I'm thinking we cannot delete the parameter since it is used by JobSubmitter#populateTokenCache.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Ray Chiang for updating the patch. Reviewed parameters from yarn.resourcemanager.delegation-token-renewer.thread-count to mapreduce.job.hdfs-servers . yarn.resourcemanager.delegation.token.max-lifetime Would you add the time unit (ms) in the description? yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs Default value is 86400 (24*60*60) sec. yarn.resourcemanager.reservation-system.class By default, org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityReservationSystem is used when using CapacityScheduler, and org.apache.hadoop.yarn.server.resourcemanager.reservation.FairReservationSystem is used when using FairScheduler. yarn.resourcemanager.reservation-system.plan.follower By default, org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacitySchedulerPlanFollower is used when using CapacityScheduler, and org.apache.hadoop.yarn.server.resourcemanager.reservation.FairSchedulerPlanFollower is used when using FairScheduler. + <name>yarn.nodemanager.linux-container-executor.cgroups.delete-delay-ms</name> + <value>20></value> Misformatted value tag. yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms Default value is 1000. yarn.nodemanager.linux-container-executor.group I don't think the parameter should be on yarn-default.xml since the parameter is set by container-executor.cfg. yarn.nodemanager.user-home-dir The home directory of the user for containers. Default value is "/home/". - <name>mapreduce.job.hdfs-servers</name> - <value>${fs.defaultFS}</value> I'm thinking we cannot delete the parameter since it is used by JobSubmitter#populateTokenCache .
          Hide
          rchiang Ray Chiang added a comment -

          Updated with the latest fixes

          Show
          rchiang Ray Chiang added a comment - Updated with the latest fixes
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 41s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 59s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 42s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 24s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.
              38m 54s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731064/YARN-3069.007.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4c7b9b6
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7751/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7751/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7751/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 41s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 59s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 42s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 24s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.     38m 54s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731064/YARN-3069.007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4c7b9b6 hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7751/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7751/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7751/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          Update for property added in YARN-1912.

          Show
          rchiang Ray Chiang added a comment - Update for property added in YARN-1912 .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 41s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 51s There were no new checkstyle issues.
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 1m 55s Tests passed in hadoop-yarn-common.
              38m 35s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732298/YARN-3069.008.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fe0df59
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7894/artifact/patchprocess/whitespace.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7894/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7894/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7894/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 41s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 51s There were no new checkstyle issues. -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 1m 55s Tests passed in hadoop-yarn-common.     38m 35s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732298/YARN-3069.008.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fe0df59 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7894/artifact/patchprocess/whitespace.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7894/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7894/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7894/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Ray Chiang for the update. Looked the patch from yarn.nodemanager.aux-services.mapreduce_shuffle.class to yarn.client.app-submission.poll-interval.

          <!-- Minicluster Configuration -->
          

          I'm thinking it would be better for users to document that the configuration is only used for testing.

          yarn.minicluster.yarn.nodemanager.resource.memory-mb

          Default value is 4096.

          yarn.node-labels.fs-store.retry-policy-spec

          Retry policy used for FileSystem node label store. The policy is specified by N pairs of sleep-time in milliseconds and number-of-retries "s1,n1,s2,n2,...". Default value is 2000, 500. (I'm thinking the default number of retries is too high.)

          <description>
            URI for NodeLabelManager
          </description>
          

          Would you document that default is in local: /tmp/hadoop-yarn-${user}/node-labels/ in the description? It is described in FileSystemNodeLabelsStore#getDefaultFSNodeLabelsRootDir.

          yarn.node-labels.configuration-type

          Set configuration type for node labels. Administrators can specify "centralized" or "distributed".

          yarn.client.app-submission.poll-interval

          Can we move this parameter to DeprecatedProperties.md?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Ray Chiang for the update. Looked the patch from yarn.nodemanager.aux-services.mapreduce_shuffle.class to yarn.client.app-submission.poll-interval . <!-- Minicluster Configuration --> I'm thinking it would be better for users to document that the configuration is only used for testing. yarn.minicluster.yarn.nodemanager.resource.memory-mb Default value is 4096. yarn.node-labels.fs-store.retry-policy-spec Retry policy used for FileSystem node label store. The policy is specified by N pairs of sleep-time in milliseconds and number-of-retries "s1,n1,s2,n2,...". Default value is 2000, 500. (I'm thinking the default number of retries is too high.) <description> URI for NodeLabelManager </description> Would you document that default is in local: /tmp/hadoop-yarn-${user}/node-labels/ in the description? It is described in FileSystemNodeLabelsStore#getDefaultFSNodeLabelsRootDir . yarn.node-labels.configuration-type Set configuration type for node labels. Administrators can specify "centralized" or "distributed". yarn.client.app-submission.poll-interval Can we move this parameter to DeprecatedProperties.md?
          Hide
          rchiang Ray Chiang added a comment -
          • Move yarn.client.app-submission.poll-interval to DeprecatedProperties.md
          • Add new property yarn.application.classpath.prepend.distcache to yarn-default.xml
          • Update properties descriptions and values based on Akira's feedback
          Show
          rchiang Ray Chiang added a comment - Move yarn.client.app-submission.poll-interval to DeprecatedProperties.md Add new property yarn.application.classpath.prepend.distcache to yarn-default.xml Update properties descriptions and values based on Akira's feedback
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 28s The applied patch generated 1 additional warning messages.
          +1 javadoc 9m 32s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 59s Site still builds.
          +1 checkstyle 2m 1s There were no new checkstyle issues.
          -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 2s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 13s Tests passed in hadoop-common.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
              70m 19s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733585/YARN-3069.009.patch
          Optional Tests site javadoc javac unit findbugs checkstyle
          git revision trunk / 060c84e
          javac https://builds.apache.org/job/PreCommit-YARN-Build/7975/artifact/patchprocess/diffJavacWarnings.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7975/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7975/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7975/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7975/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7975/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 28s The applied patch generated 1 additional warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 site 2m 59s Site still builds. +1 checkstyle 2m 1s There were no new checkstyle issues. -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 2s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 13s Tests passed in hadoop-common. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.     70m 19s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733585/YARN-3069.009.patch Optional Tests site javadoc javac unit findbugs checkstyle git revision trunk / 060c84e javac https://builds.apache.org/job/PreCommit-YARN-Build/7975/artifact/patchprocess/diffJavacWarnings.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7975/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7975/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7975/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7975/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7975/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -
          • Fix whitespace and deprecation warnings
          Show
          rchiang Ray Chiang added a comment - Fix whitespace and deprecation warnings
          Hide
          rchiang Ray Chiang added a comment -
          • Leave out MR bits from previous patch.
          Show
          rchiang Ray Chiang added a comment - Leave out MR bits from previous patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 56s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 48s There were no new javac warning messages.
          +1 javadoc 9m 55s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 57s Site still builds.
          +1 checkstyle 1m 34s There were no new checkstyle issues.
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 1s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 8s Tests passed in hadoop-common.
          +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.
              70m 50s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733618/YARN-3069.010.patch
          Optional Tests site javadoc javac unit findbugs checkstyle
          git revision trunk / 0790275
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7979/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7979/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7979/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7979/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 56s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 48s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 site 2m 57s Site still builds. +1 checkstyle 1m 34s There were no new checkstyle issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 1s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 8s Tests passed in hadoop-common. +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.     70m 50s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733618/YARN-3069.010.patch Optional Tests site javadoc javac unit findbugs checkstyle git revision trunk / 0790275 hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7979/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7979/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7979/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7979/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Ray Chiang for updating the patch.

          1. Would you reflect the previous comment for yarn.node-labels.fs-store.retry-policy-spec?
          2. For YARN registry, the parameters are written in core-site.xml. Can we remove them from the patch?

          My review is almost done.
          @Watchers: I would appreciate if you could review this patch. It includes a lot of descriptions for parameters, so it should be reviewed by a lot of developers.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Ray Chiang for updating the patch. Would you reflect the previous comment for yarn.node-labels.fs-store.retry-policy-spec ? For YARN registry, the parameters are written in core-site.xml. Can we remove them from the patch? My review is almost done. @Watchers: I would appreciate if you could review this patch. It includes a lot of descriptions for parameters, so it should be reviewed by a lot of developers.
          Hide
          rchiang Ray Chiang added a comment -

          Thanks Akira! New patch with the following changes:

          • Fix description for yarn.node-labels.fs-store.retry-policy-spec
          • Remove YARN registry entries from yarn-default.xml
          • Remove one outdated entry yarn.application.classpath.prepend.distcache
          • Add entry for yarn.intermediate-data-encryption.enable

          I'll also go through the yarn-default.xml file once more to make sure no default values will change.

          Show
          rchiang Ray Chiang added a comment - Thanks Akira! New patch with the following changes: Fix description for yarn.node-labels.fs-store.retry-policy-spec Remove YARN registry entries from yarn-default.xml Remove one outdated entry yarn.application.classpath.prepend.distcache Add entry for yarn.intermediate-data-encryption.enable I'll also go through the yarn-default.xml file once more to make sure no default values will change.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 19m 46s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 58s Site still builds.
          +1 checkstyle 1m 36s There were no new checkstyle issues.
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 34s Tests passed in hadoop-common.
          +1 yarn tests 1m 55s Tests passed in hadoop-yarn-common.
              72m 56s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736976/YARN-3069.011.patch
          Optional Tests site javadoc javac unit findbugs checkstyle
          git revision trunk / a2bd621
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8168/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8168/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8168/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8168/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8168/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 46s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 site 2m 58s Site still builds. +1 checkstyle 1m 36s There were no new checkstyle issues. -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 34s Tests passed in hadoop-common. +1 yarn tests 1m 55s Tests passed in hadoop-yarn-common.     72m 56s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736976/YARN-3069.011.patch Optional Tests site javadoc javac unit findbugs checkstyle git revision trunk / a2bd621 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8168/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8168/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8168/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8168/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8168/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Ray Chiang for updating the patch.

          I'll also go through the yarn-default.xml file once more to make sure no default values will change.

          Thanks. I checked the file and found 3 issues.

          yarn.log-aggregation-status.time-out.ms

          The default value is 600000, not 60000.

          yarn.nodemanager.webapp.https.address

          The default value is 0.0.0.0:8044

          yarn.intermediate-data-encryption.enable

          The default value is false.

          I'm +1 if these are addressed.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Ray Chiang for updating the patch. I'll also go through the yarn-default.xml file once more to make sure no default values will change. Thanks. I checked the file and found 3 issues. yarn.log-aggregation-status.time-out.ms The default value is 600000, not 60000. yarn.nodemanager.webapp.https.address The default value is 0.0.0.0:8044 yarn.intermediate-data-encryption.enable The default value is false. I'm +1 if these are addressed.
          Hide
          rchiang Ray Chiang added a comment -

          Thanks Akira! I'm going to see about automating the default property comparison, if one exists, in the XML->YarnConfiguration direction. I should have thought of that earlier.

          Show
          rchiang Ray Chiang added a comment - Thanks Akira! I'm going to see about automating the default property comparison, if one exists, in the XML->YarnConfiguration direction. I should have thought of that earlier.
          Hide
          rchiang Ray Chiang added a comment -

          I found one more important mismatch in the existing file.

          XML Property: yarn.scheduler.maximum-allocation-vcores
          XML Value: 32
          Config Name: DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES
          Config Value: 4

          The Config value comes from YARN-193 and the default xml property comes from YARN-2. Should we keep it this way or should one of the values get updated?

          Show
          rchiang Ray Chiang added a comment - I found one more important mismatch in the existing file. XML Property: yarn.scheduler.maximum-allocation-vcores XML Value: 32 Config Name: DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES Config Value: 4 The Config value comes from YARN-193 and the default xml property comes from YARN-2 . Should we keep it this way or should one of the values get updated?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Nice catch! I'm thinking we can discuss the issue about the mismatch in a separate jira.

          Show
          ajisakaa Akira Ajisaka added a comment - Nice catch! I'm thinking we can discuss the issue about the mismatch in a separate jira.
          Hide
          rchiang Ray Chiang added a comment -

          Created YARN-3823.

          Show
          rchiang Ray Chiang added a comment - Created YARN-3823 .
          Hide
          rchiang Ray Chiang added a comment -

          Wrote up the code for automatic checking at HADOOP-12101 for automatic default checking. Ran automatic checking with the following results:

          • yarn-default.xml has 15 properties that do not match the default Config value
            • Filed one bug filed at YARN-3823
            • Remaining 14 are due to variable references like $ {yarn.resourcemanager.hostname}

              or a documented -1 value like yarn.nodemanager.resource.memory-mb.

          • Configuration(s) have 67 properties with no corresponding default member variable. These will need to be verified manually.
            • Will document as a separate comment.
          • yarn-default.xml has 6 properties with empty values
            • Nothing to compare
          • yarn-default.xml has 135 properties which match a corresponding Config variable
            • No need to compare
          Show
          rchiang Ray Chiang added a comment - Wrote up the code for automatic checking at HADOOP-12101 for automatic default checking. Ran automatic checking with the following results: yarn-default.xml has 15 properties that do not match the default Config value Filed one bug filed at YARN-3823 Remaining 14 are due to variable references like $ {yarn.resourcemanager.hostname} or a documented -1 value like yarn.nodemanager.resource.memory-mb. Configuration(s) have 67 properties with no corresponding default member variable. These will need to be verified manually. Will document as a separate comment. yarn-default.xml has 6 properties with empty values Nothing to compare yarn-default.xml has 135 properties which match a corresponding Config variable No need to compare
          Hide
          rchiang Ray Chiang added a comment -

          Most of the manual verification were in the following categories:

          • Hardcoded value
          • Not using DEFAULT_FOO for FOO member variable naming convention
          • No default value at all
          • Variable is used indirectly

          Manual verification specifics:

          CLIENT_FAILOVER_MAX_ATTEMPTS

          • Hardcoded default to -1 in RMProxy

          CLIENT_FAILOVER_SLEEPTIME_BASE_MS
          CLIENT_FAILOVER_SLEEPTIME_MAX_MS

          • Defaults to RESOURCEMANAGER_CONNECT_RETRY_INTERVAL_MS or
            DEFAULT_RESOURCEMANAGER_CONNECT_RETRY_INTERVAL_MS

          DEBUG_NM_DELETE_DELAY_SEC

          • Hardcoded default to 0 in DeletionService

          FS_NODE_LABELS_STORE_ROOT_DIR

          • Defaults to FileSystemNodeLabelsStore#getDefaultFSNodeLabelsRootDir() return value

          FS_RM_STATE_STORE_URI

          • No default value anywhere

          IS_MINI_YARN_CLUSTER

          • Hardcoded to false in Client, MRApps, ResourceManager

          NM_AUX_SERVICES

          • No default value anywhere. Maybe whatever Configuration#getStringCollection() returns.

          NM_BIND_HOST

          • No default value anywhere

          NM_CONTAINER_EXECUTOR

          • Hardcoded to DefaultContainerExecutor.class in NodeManager

          NM_CONTAINER_LOCALIZER_JAVA_OPTS_KEY

          • Defaults to YarnConfiguration.NM_CONTAINER_LOCALIZER_JAVA_OPTS_DEFAULT in ContainerLocalizer

          NM_CONTAINER_MON_PROCESS_TREE
          NM_CONTAINER_MON_RESOURCE_CALCULATOR

          • Hardcoded to null in ContainersMonitorImpl

          NM_DISK_HEALTH_CHECK_ENABLE

          • Hardcoded to true in LocalDirsHanderService

          NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME

          • Defaults to unconventional name YarnConfiguration.NM_DEFAULT_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME

          NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME

          • No default value anywhere

          NM_HEALTH_CHECK_SCRIPT_OPTS

          • Defaults to empty String array in NodeManager

          NM_HEALTH_CHECK_SCRIPT_PATH

          • No default value anywhere

          NM_KEYTAB

          • Defaults to YarnConfiguration.NM_PRINCIPAL

          NM_LINUX_CONTAINER_CGROUPS_HIERARCHY

          • Hardcoded to "/hadoop-yarn" in CGroupsHandlerImpl and CgroupsLCEResourcesHandler

          NM_LINUX_CONTAINER_CGROUPS_MOUNT

          • Hardcoded to false in CGroupsHandlerImpl and CgroupsLCEResourcesHandler

          NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH

          • Hardcoded to null in CGroupsHandlerImpl and CgroupsLCEResourcesHandler

          NM_LINUX_CONTAINER_EXECUTOR_PATH

          • Defaults to internal variable defaultPath (which looks to be based off HADOOP_YARN_HOME environment)

          NM_LINUX_CONTAINER_GROUP

          • Not used anywhere

          NM_LINUX_CONTAINER_RESOURCES_HANDLER

          • Hardcoded to DefaultLCEResourcesHandler.class in LinuxContainerExecutor

          NM_LOG_DELETION_THREADS_COUNT

          • Defaults to unconventional name YarnConfiguration.DEFAULT_NM_LOG_DELETE_THREAD_COUNT

          NM_NONSECURE_MODE_LOCAL_USER_KEY

          • Defaults to unconventional name YarnConfiguration.DEFAULT_NM_NONSECURE_MODE_LOCAL_USER

          NM_NONSECURE_MODE_USER_PATTERN_KEY

          • Defaults to unconventional name YarnConfiguration.DEFAULT_NM_NONSECURE_MODE_USER_PATTERN

          NM_PRINCIPAL

          • Is the default value for YarnConfiguration.NM_KEYTAB

          NM_RECOVERY_DIR

          • No default value anywhere

          NM_SYSTEM_RESERVED_PMEM_MB

          • Hardcoded to -1 in NodeManagerHardwareUtils

          NM_WEBAPP_SPNEGO_KEYTAB_FILE_KEY
          NM_WEBAPP_SPNEGO_USER_NAME_KEY

          • No default value anywhere

          NM_WINDOWS_SECURE_CONTAINER_GROUP

          • No default value anywhere

          PROXY_KEYTAB
          PROXY_PRINCIPAL

          • No default value anywhere

          RECOVERY_ENABLED

          • Defaults to YarnConfiguration.DEFAULT_NM_NONSECURE_MODE_USER_PATTERN in ResourceManager

          RM_BIND_HOST

          • No default value anywhere

          RM_CLUSTER_ID

          • No default value anywhere

          RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY

          • Defaults to YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT in RMSecretManagerService

          RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY

          • Defaults to YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT in RMSecretManagerService

          RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY

          • Defaults to YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT in RMSecretManagerService

          RM_HA_ID

          • Defaults to values from RM_HA_IDS

          RM_HA_IDS

          • No default value, but gets validation in HAUtil#verifyAndSetRMHAIdsList()

          RM_HOSTNAME

          • Defaults to internal variable RMId in HAUtils

          RM_KEYTAB

          • Defaults to YarnConfiguration.RM_PRINCIPAL

          RM_LEVELDB_STORE_PATH

          • No default value anywhere

          RM_PRINCIPAL

          • Default value for RM_KEYTAB

          RM_PROXY_USER_PRIVILEGES_ENABLED

          • Defaults to YarnConfiguration.DEFAULT_RM_PROXY_USER_PRIVILEGES_ENABLED. Needs final keyword added.

          RM_RESERVATION_SYSTEM_CLASS

          • Defaults to AbstractReservationSystem#getDefaultReservationSystem(scheduler)

          RM_RESERVATION_SYSTEM_PLAN_FOLLOWER

          • Defaults to AbstractReservationSystem.getDefaultPlanFollower()

          RM_SCHEDULER_INCLUDE_PORT_IN_NODE_NAME

          • Unconventional default YarnConfiguration.DEFAULT_RM_SCHEDULER_USE_PORT_FOR_NODE_NAME

          RM_SCHEDULER_MONITOR_POLICIES

          • Defaults to an SchedulingEditPolicy.class as an Interface

          RM_STORE

          • Hardcoded to MemoryRMStateStore.class in RMStateStoreFactory

          RM_WEBAPP_SPNEGO_KEYTAB_FILE_KEY
          RM_WEBAPP_SPNEGO_USER_NAME_KEY

          • No default value anywhere

          RM_ZK_ADDRESS

          • No default value anywhere

          RM_ZK_AUTH

          • No default value anywhere

          RM_ZK_NUM_RETRIES

          • Unconventional default YarnConfiguration.DEFAULT_ZK_RM_NUM_RETRIES

          YARN_APPLICATION_CLASSPATH

          • Defaults to String array YarnConfiguration.DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH

          YARN_AUTHORIZATION_PROVIDER

          • Hardcoded to ConfiguredYarnAuthorizer.class in YarnAuthorizationProvider

          YARN_HTTP_POLICY_KEY

          • Unconventional default of YarnConfiguration.YARN_HTTP_POLICY_DEFAULT

          YARN_INTERMEDIATE_DATA_ENCRYPTION

          • Non-primitive, non-String default in YarnConfiguration.YARN_INTERMEDIATE_DATA_ENCRYPTION

          YARN_LOG_SERVER_URL

          • No default value anywhere

          YARN_SECURITY_SERVICE_AUTHORIZATION_APPLICATIONHISTORY_PROTOCOL

          • Hardcoded to ApplicationMasterProtocolPB.class in RMPolicyProvider

          YARN_TRACKING_URL_GENERATOR

          • Hardcoded to TrackingUriPlugin.class in WebAppProxyServlet

          ZK_RM_STATE_STORE_ROOT_NODE_ACL

          • Passed into HAUtil.getConfValueForRMInstance
          Show
          rchiang Ray Chiang added a comment - Most of the manual verification were in the following categories: Hardcoded value Not using DEFAULT_FOO for FOO member variable naming convention No default value at all Variable is used indirectly Manual verification specifics: CLIENT_FAILOVER_MAX_ATTEMPTS Hardcoded default to -1 in RMProxy CLIENT_FAILOVER_SLEEPTIME_BASE_MS CLIENT_FAILOVER_SLEEPTIME_MAX_MS Defaults to RESOURCEMANAGER_CONNECT_RETRY_INTERVAL_MS or DEFAULT_RESOURCEMANAGER_CONNECT_RETRY_INTERVAL_MS DEBUG_NM_DELETE_DELAY_SEC Hardcoded default to 0 in DeletionService FS_NODE_LABELS_STORE_ROOT_DIR Defaults to FileSystemNodeLabelsStore#getDefaultFSNodeLabelsRootDir() return value FS_RM_STATE_STORE_URI No default value anywhere IS_MINI_YARN_CLUSTER Hardcoded to false in Client, MRApps, ResourceManager NM_AUX_SERVICES No default value anywhere. Maybe whatever Configuration#getStringCollection() returns. NM_BIND_HOST No default value anywhere NM_CONTAINER_EXECUTOR Hardcoded to DefaultContainerExecutor.class in NodeManager NM_CONTAINER_LOCALIZER_JAVA_OPTS_KEY Defaults to YarnConfiguration.NM_CONTAINER_LOCALIZER_JAVA_OPTS_DEFAULT in ContainerLocalizer NM_CONTAINER_MON_PROCESS_TREE NM_CONTAINER_MON_RESOURCE_CALCULATOR Hardcoded to null in ContainersMonitorImpl NM_DISK_HEALTH_CHECK_ENABLE Hardcoded to true in LocalDirsHanderService NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME Defaults to unconventional name YarnConfiguration.NM_DEFAULT_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME No default value anywhere NM_HEALTH_CHECK_SCRIPT_OPTS Defaults to empty String array in NodeManager NM_HEALTH_CHECK_SCRIPT_PATH No default value anywhere NM_KEYTAB Defaults to YarnConfiguration.NM_PRINCIPAL NM_LINUX_CONTAINER_CGROUPS_HIERARCHY Hardcoded to "/hadoop-yarn" in CGroupsHandlerImpl and CgroupsLCEResourcesHandler NM_LINUX_CONTAINER_CGROUPS_MOUNT Hardcoded to false in CGroupsHandlerImpl and CgroupsLCEResourcesHandler NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH Hardcoded to null in CGroupsHandlerImpl and CgroupsLCEResourcesHandler NM_LINUX_CONTAINER_EXECUTOR_PATH Defaults to internal variable defaultPath (which looks to be based off HADOOP_YARN_HOME environment) NM_LINUX_CONTAINER_GROUP Not used anywhere NM_LINUX_CONTAINER_RESOURCES_HANDLER Hardcoded to DefaultLCEResourcesHandler.class in LinuxContainerExecutor NM_LOG_DELETION_THREADS_COUNT Defaults to unconventional name YarnConfiguration.DEFAULT_NM_LOG_DELETE_THREAD_COUNT NM_NONSECURE_MODE_LOCAL_USER_KEY Defaults to unconventional name YarnConfiguration.DEFAULT_NM_NONSECURE_MODE_LOCAL_USER NM_NONSECURE_MODE_USER_PATTERN_KEY Defaults to unconventional name YarnConfiguration.DEFAULT_NM_NONSECURE_MODE_USER_PATTERN NM_PRINCIPAL Is the default value for YarnConfiguration.NM_KEYTAB NM_RECOVERY_DIR No default value anywhere NM_SYSTEM_RESERVED_PMEM_MB Hardcoded to -1 in NodeManagerHardwareUtils NM_WEBAPP_SPNEGO_KEYTAB_FILE_KEY NM_WEBAPP_SPNEGO_USER_NAME_KEY No default value anywhere NM_WINDOWS_SECURE_CONTAINER_GROUP No default value anywhere PROXY_KEYTAB PROXY_PRINCIPAL No default value anywhere RECOVERY_ENABLED Defaults to YarnConfiguration.DEFAULT_NM_NONSECURE_MODE_USER_PATTERN in ResourceManager RM_BIND_HOST No default value anywhere RM_CLUSTER_ID No default value anywhere RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY Defaults to YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT in RMSecretManagerService RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY Defaults to YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT in RMSecretManagerService RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY Defaults to YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT in RMSecretManagerService RM_HA_ID Defaults to values from RM_HA_IDS RM_HA_IDS No default value, but gets validation in HAUtil#verifyAndSetRMHAIdsList() RM_HOSTNAME Defaults to internal variable RMId in HAUtils RM_KEYTAB Defaults to YarnConfiguration.RM_PRINCIPAL RM_LEVELDB_STORE_PATH No default value anywhere RM_PRINCIPAL Default value for RM_KEYTAB RM_PROXY_USER_PRIVILEGES_ENABLED Defaults to YarnConfiguration.DEFAULT_RM_PROXY_USER_PRIVILEGES_ENABLED. Needs final keyword added. RM_RESERVATION_SYSTEM_CLASS Defaults to AbstractReservationSystem#getDefaultReservationSystem(scheduler) RM_RESERVATION_SYSTEM_PLAN_FOLLOWER Defaults to AbstractReservationSystem.getDefaultPlanFollower() RM_SCHEDULER_INCLUDE_PORT_IN_NODE_NAME Unconventional default YarnConfiguration.DEFAULT_RM_SCHEDULER_USE_PORT_FOR_NODE_NAME RM_SCHEDULER_MONITOR_POLICIES Defaults to an SchedulingEditPolicy.class as an Interface RM_STORE Hardcoded to MemoryRMStateStore.class in RMStateStoreFactory RM_WEBAPP_SPNEGO_KEYTAB_FILE_KEY RM_WEBAPP_SPNEGO_USER_NAME_KEY No default value anywhere RM_ZK_ADDRESS No default value anywhere RM_ZK_AUTH No default value anywhere RM_ZK_NUM_RETRIES Unconventional default YarnConfiguration.DEFAULT_ZK_RM_NUM_RETRIES YARN_APPLICATION_CLASSPATH Defaults to String array YarnConfiguration.DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH YARN_AUTHORIZATION_PROVIDER Hardcoded to ConfiguredYarnAuthorizer.class in YarnAuthorizationProvider YARN_HTTP_POLICY_KEY Unconventional default of YarnConfiguration.YARN_HTTP_POLICY_DEFAULT YARN_INTERMEDIATE_DATA_ENCRYPTION Non-primitive, non-String default in YarnConfiguration.YARN_INTERMEDIATE_DATA_ENCRYPTION YARN_LOG_SERVER_URL No default value anywhere YARN_SECURITY_SERVICE_AUTHORIZATION_APPLICATIONHISTORY_PROTOCOL Hardcoded to ApplicationMasterProtocolPB.class in RMPolicyProvider YARN_TRACKING_URL_GENERATOR Hardcoded to TrackingUriPlugin.class in WebAppProxyServlet ZK_RM_STATE_STORE_ROOT_NODE_ACL Passed into HAUtil.getConfValueForRMInstance
          Hide
          rchiang Ray Chiang added a comment -

          Forgot to mention the above two comments are for the .012 patch, coming next.

          Show
          rchiang Ray Chiang added a comment - Forgot to mention the above two comments are for the .012 patch, coming next.
          Hide
          rchiang Ray Chiang added a comment -
          • Implement Akira's last 3 comments
          • First version including fixes from HADOOP-12101
            • Fix default in yarn.nodemanager.env-whitelist to match
            • Fix spacing in two other properties to match
          Show
          rchiang Ray Chiang added a comment - Implement Akira's last 3 comments First version including fixes from HADOOP-12101 Fix default in yarn.nodemanager.env-whitelist to match Fix spacing in two other properties to match
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 21m 24s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 50s There were no new javac warning messages.
          +1 javadoc 9m 52s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 4s Site still builds.
          +1 checkstyle 2m 3s There were no new checkstyle issues.
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 44s Tests passed in hadoop-common.
          +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.
              74m 56s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740304/YARN-3069.012.patch
          Optional Tests site javadoc javac unit findbugs checkstyle
          git revision trunk / 295d678
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8282/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8282/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8282/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8282/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8282/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 21m 24s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 9m 52s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 site 3m 4s Site still builds. +1 checkstyle 2m 3s There were no new checkstyle issues. -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 44s Tests passed in hadoop-common. +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.     74m 56s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740304/YARN-3069.012.patch Optional Tests site javadoc javac unit findbugs checkstyle git revision trunk / 295d678 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8282/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8282/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8282/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8282/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8282/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -
          • Fix whitespace
          • Update against trunk
          Show
          rchiang Ray Chiang added a comment - Fix whitespace Update against trunk
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 21m 25s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 48s There were no new javac warning messages.
          +1 javadoc 9m 52s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 4s Site still builds.
          +1 checkstyle 2m 4s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 11s Tests passed in hadoop-common.
          +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.
              75m 23s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12741380/YARN-3069.013.patch
          Optional Tests site javadoc javac unit findbugs checkstyle
          git revision trunk / 122cad6
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8328/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8328/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8328/testReport/
          Java 1.7.0_55
          uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8328/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 21m 25s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 48s There were no new javac warning messages. +1 javadoc 9m 52s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 site 3m 4s Site still builds. +1 checkstyle 2m 4s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 11s Tests passed in hadoop-common. +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.     75m 23s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12741380/YARN-3069.013.patch Optional Tests site javadoc javac unit findbugs checkstyle git revision trunk / 122cad6 hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8328/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8328/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8328/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8328/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1, looks good to me. Thanks Ray Chiang for updating the patch. I'll commit it on July 13 JST if there are no objections.

          Show
          ajisakaa Akira Ajisaka added a comment - +1, looks good to me. Thanks Ray Chiang for updating the patch. I'll commit it on July 13 JST if there are no objections.
          Hide
          rchiang Ray Chiang added a comment -

          Thanks Akira! I'll be happy to see one of these XML verifiers pushed all the way through.

          Show
          rchiang Ray Chiang added a comment - Thanks Akira! I'll be happy to see one of these XML verifiers pushed all the way through.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8153 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8153/)
          YARN-3069. Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad)

          • hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8153 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8153/ ) YARN-3069 . Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad) hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed the latest patch to trunk and branch-2. Thanks Ray Chiang for the continuous work!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed the latest patch to trunk and branch-2. Thanks Ray Chiang for the continuous work!
          Hide
          rchiang Ray Chiang added a comment -

          Thanks for the sharp eye on the review and the commit!

          Show
          rchiang Ray Chiang added a comment - Thanks for the sharp eye on the review and the commit!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #255 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/255/)
          YARN-3069. Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad)

          • hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #255 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/255/ ) YARN-3069 . Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad) hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #985 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/985/)
          YARN-3069. Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #985 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/985/ ) YARN-3069 . Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/243/)
          YARN-3069. Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad)

          • hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/243/ ) YARN-3069 . Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad) hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2182 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2182/)
          YARN-3069. Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad)

          • hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2182 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2182/ ) YARN-3069 . Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad) hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2201 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2201/)
          YARN-3069. Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2201 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2201/ ) YARN-3069 . Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #253 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/253/)
          YARN-3069. Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad)

          • hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #253 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/253/ ) YARN-3069 . Document missing properties in yarn-default.xml. Contributed by Ray Chiang. (aajisaka: rev d6675606dc5f141c9af4f76a37128f8de4cfedad) hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt

            People

            • Assignee:
              rchiang Ray Chiang
              Reporter:
              rchiang Ray Chiang
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development