Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6057

Remove obsolete entries from mapred-default.xml

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 3.0.0
    • Component/s: None
    • Labels:

      Description

      The following properties are defined in mapred-default.xml but no longer exist in MRJobConfig.

      map.sort.class
      mapred.child.env
      mapred.child.java.opts
      mapreduce.app-submission.cross-platform
      mapreduce.client.completion.pollinterval
      mapreduce.client.output.filter
      mapreduce.client.progressmonitor.pollinterval
      mapreduce.client.submit.file.replication
      mapreduce.cluster.acls.enabled
      mapreduce.cluster.local.dir
      mapreduce.framework.name
      mapreduce.ifile.readahead
      mapreduce.ifile.readahead.bytes
      mapreduce.input.fileinputformat.list-status.num-threads
      mapreduce.input.fileinputformat.split.minsize
      mapreduce.input.lineinputformat.linespermap
      mapreduce.job.counters.limit
      mapreduce.job.max.split.locations
      mapreduce.job.reduce.shuffle.consumer.plugin.class
      mapreduce.jobhistory.address
      mapreduce.jobhistory.admin.acl
      mapreduce.jobhistory.admin.address
      mapreduce.jobhistory.cleaner.enable
      mapreduce.jobhistory.cleaner.interval-ms
      mapreduce.jobhistory.client.thread-count
      mapreduce.jobhistory.datestring.cache.size
      mapreduce.jobhistory.done-dir
      mapreduce.jobhistory.http.policy
      mapreduce.jobhistory.intermediate-done-dir
      mapreduce.jobhistory.joblist.cache.size
      mapreduce.jobhistory.keytab
      mapreduce.jobhistory.loadedjobs.cache.size
      mapreduce.jobhistory.max-age-ms
      mapreduce.jobhistory.minicluster.fixed.ports
      mapreduce.jobhistory.move.interval-ms
      mapreduce.jobhistory.move.thread-count
      mapreduce.jobhistory.principal
      mapreduce.jobhistory.recovery.enable
      mapreduce.jobhistory.recovery.store.class
      mapreduce.jobhistory.recovery.store.fs.uri
      mapreduce.jobhistory.store.class
      mapreduce.jobhistory.webapp.address
      mapreduce.local.clientfactory.class.name
      mapreduce.map.skip.proc.count.autoincr
      mapreduce.output.fileoutputformat.compress
      mapreduce.output.fileoutputformat.compress.codec
      mapreduce.output.fileoutputformat.compress.type
      mapreduce.reduce.skip.proc.count.autoincr
      mapreduce.shuffle.connection-keep-alive.enable
      mapreduce.shuffle.connection-keep-alive.timeout
      mapreduce.shuffle.max.connections
      mapreduce.shuffle.max.threads
      mapreduce.shuffle.port
      mapreduce.shuffle.ssl.enabled
      mapreduce.shuffle.ssl.file.buffer.size
      mapreduce.shuffle.transfer.buffer.size
      mapreduce.shuffle.transferTo.allowed
      yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts

      Submitting bug for comment/feedback about which properties should be kept in mapred-default.xml.

      1. MAPREDUCE-6057-03.patch
        12 kB
        Ray Chiang
      2. MAPREDUCE-6057-02.patch
        11 kB
        Ray Chiang
      3. MAPREDUCE-6057-01.patch
        11 kB
        Ray Chiang
      4. MAPREDUCE-6057.004.patch
        12 kB
        Ray Chiang

        Issue Links

          Activity

          Hide
          Ray Chiang added a comment -

          Thanks for commit. Getting closer getting those pesky files cleaned up.

          Show
          Ray Chiang added a comment - Thanks for commit. Getting closer getting those pesky files cleaned up.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2126 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2126/)
          MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2126 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2126/ ) MAPREDUCE-6057 . Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #177 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/177/)
          MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #177 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/177/ ) MAPREDUCE-6057 . Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml hadoop-mapreduce-project/CHANGES.txt hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #176 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/176/)
          MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          • hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #176 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/176/ ) MAPREDUCE-6057 . Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #910 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/910/)
          MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #910 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/910/ ) MAPREDUCE-6057 . Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #167 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/167/)
          MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #167 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/167/ ) MAPREDUCE-6057 . Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2108 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2108/)
          MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2108 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2108/ ) MAPREDUCE-6057 . Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml hadoop-mapreduce-project/CHANGES.txt
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7683 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7683/)
          MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7683 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7683/ ) MAPREDUCE-6057 . Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) (aw: rev 618ba707f0f2ddc353414dbd0eee0ab9e83b8013) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml hadoop-mapreduce-project/CHANGES.txt hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json
          Hide
          Allen Wittenauer added a comment -

          +1 committing to trunk.

          Thanks!

          Show
          Allen Wittenauer added a comment - +1 committing to trunk. Thanks!
          Hide
          Brahma Reddy Battula added a comment -

          Above findbugs are addressed in HADOOP-11821..

          Show
          Brahma Reddy Battula added a comment - Above findbugs are addressed in HADOOP-11821 ..
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 28s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          -1 release audit 0m 22s The applied patch generated 1 release audit warnings.
          -1 checkstyle 4m 0s The applied patch generated 1 additional checkstyle issues.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 2m 39s The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 1m 44s Tests passed in hadoop-mapreduce-client-core.
          +1 mapreduce tests 5m 42s Tests passed in hadoop-mapreduce-client-hs.
          +1 tools/hadoop tests 0m 51s Tests passed in hadoop-sls.
              49m 10s  



          Reason Tests
          FindBugs module:hadoop-sls
            Found reliance on default encoding in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String, String):in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String, String): new java.io.FileReader(String) At RumenToSLSConverter.java:[line 122]
            Found reliance on default encoding in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String, String):in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String, String): new java.io.FileWriter(String) At RumenToSLSConverter.java:[line 124]
            Found reliance on default encoding in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSNodeFile(String):in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSNodeFile(String): new java.io.FileWriter(String) At RumenToSLSConverter.java:[line 145]
            Found reliance on default encoding in org.apache.hadoop.yarn.sls.SLSRunner.startAMFromSLSTraces(Resource, int):in org.apache.hadoop.yarn.sls.SLSRunner.startAMFromSLSTraces(Resource, int): new java.io.FileReader(String) At SLSRunner.java:[line 280]
            Found reliance on default encoding in org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.initMetrics():in org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.initMetrics(): new java.io.FileWriter(String) At ResourceSchedulerWrapper.java:[line 490]
            Found reliance on default encoding in new org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper$MetricsLogRunnable(ResourceSchedulerWrapper):in new org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper$MetricsLogRunnable(ResourceSchedulerWrapper): new java.io.FileWriter(String) At ResourceSchedulerWrapper.java:[line 695]
            Found reliance on default encoding in org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.initMetrics():in org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.initMetrics(): new java.io.FileWriter(String) At SLSCapacityScheduler.java:[line 493]
            Found reliance on default encoding in new org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler$MetricsLogRunnable(SLSCapacityScheduler):in new org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler$MetricsLogRunnable(SLSCapacityScheduler): new java.io.FileWriter(String) At SLSCapacityScheduler.java:[line 698]
            Found reliance on default encoding in org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromNodeFile(String):in org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromNodeFile(String): new java.io.FileReader(String) At SLSUtils.java:[line 119]
            Found reliance on default encoding in org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromSLSTrace(String):in org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromSLSTrace(String): new java.io.FileReader(String) At SLSUtils.java:[line 92]
            Class org.apache.hadoop.yarn.sls.web.SLSWebApp defines non-transient non-serializable instance field handleOperTimecostHistogramMap In SLSWebApp.java:instance field handleOperTimecostHistogramMap In SLSWebApp.java
            Class org.apache.hadoop.yarn.sls.web.SLSWebApp defines non-transient non-serializable instance field queueAllocatedMemoryCounterMap In SLSWebApp.java:instance field queueAllocatedMemoryCounterMap In SLSWebApp.java
            Class org.apache.hadoop.yarn.sls.web.SLSWebApp defines non-transient non-serializable instance field queueAllocatedVCoresCounterMap In SLSWebApp.java:instance field queueAllocatedVCoresCounterMap In SLSWebApp.java



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12693506/MAPREDUCE-6057.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8847777
          Release Audit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/checkstyle-result-diff.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/newPatchFindbugsWarningshadoop-sls.html
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-hs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
          hadoop-sls test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/testrun_hadoop-sls.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. -1 release audit 0m 22s The applied patch generated 1 release audit warnings. -1 checkstyle 4m 0s The applied patch generated 1 additional checkstyle issues. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 2m 39s The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 44s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 5m 42s Tests passed in hadoop-mapreduce-client-hs. +1 tools/hadoop tests 0m 51s Tests passed in hadoop-sls.     49m 10s   Reason Tests FindBugs module:hadoop-sls   Found reliance on default encoding in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String, String):in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String, String): new java.io.FileReader(String) At RumenToSLSConverter.java: [line 122]   Found reliance on default encoding in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String, String):in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String, String): new java.io.FileWriter(String) At RumenToSLSConverter.java: [line 124]   Found reliance on default encoding in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSNodeFile(String):in org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSNodeFile(String): new java.io.FileWriter(String) At RumenToSLSConverter.java: [line 145]   Found reliance on default encoding in org.apache.hadoop.yarn.sls.SLSRunner.startAMFromSLSTraces(Resource, int):in org.apache.hadoop.yarn.sls.SLSRunner.startAMFromSLSTraces(Resource, int): new java.io.FileReader(String) At SLSRunner.java: [line 280]   Found reliance on default encoding in org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.initMetrics():in org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.initMetrics(): new java.io.FileWriter(String) At ResourceSchedulerWrapper.java: [line 490]   Found reliance on default encoding in new org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper$MetricsLogRunnable(ResourceSchedulerWrapper):in new org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper$MetricsLogRunnable(ResourceSchedulerWrapper): new java.io.FileWriter(String) At ResourceSchedulerWrapper.java: [line 695]   Found reliance on default encoding in org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.initMetrics():in org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.initMetrics(): new java.io.FileWriter(String) At SLSCapacityScheduler.java: [line 493]   Found reliance on default encoding in new org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler$MetricsLogRunnable(SLSCapacityScheduler):in new org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler$MetricsLogRunnable(SLSCapacityScheduler): new java.io.FileWriter(String) At SLSCapacityScheduler.java: [line 698]   Found reliance on default encoding in org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromNodeFile(String):in org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromNodeFile(String): new java.io.FileReader(String) At SLSUtils.java: [line 119]   Found reliance on default encoding in org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromSLSTrace(String):in org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromSLSTrace(String): new java.io.FileReader(String) At SLSUtils.java: [line 92]   Class org.apache.hadoop.yarn.sls.web.SLSWebApp defines non-transient non-serializable instance field handleOperTimecostHistogramMap In SLSWebApp.java:instance field handleOperTimecostHistogramMap In SLSWebApp.java   Class org.apache.hadoop.yarn.sls.web.SLSWebApp defines non-transient non-serializable instance field queueAllocatedMemoryCounterMap In SLSWebApp.java:instance field queueAllocatedMemoryCounterMap In SLSWebApp.java   Class org.apache.hadoop.yarn.sls.web.SLSWebApp defines non-transient non-serializable instance field queueAllocatedVCoresCounterMap In SLSWebApp.java:instance field queueAllocatedVCoresCounterMap In SLSWebApp.java Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12693506/MAPREDUCE-6057.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8847777 Release Audit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/checkstyle-result-diff.txt Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/newPatchFindbugsWarningshadoop-sls.html hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-hs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt hadoop-sls test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/artifact/patchprocess/testrun_hadoop-sls.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5452/console This message was automatically generated.
          Hide
          Ray Chiang added a comment -

          RE: findbugs

          No code changes in the files mentioned.

          Show
          Ray Chiang added a comment - RE: findbugs No code changes in the files mentioned.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12693506/MAPREDUCE-6057.004.patch
          against trunk revision a122278.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 26 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-sls.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5116//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5116//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5116//artifact/patchprocess/newPatchFindbugsWarningshadoop-sls.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5116//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693506/MAPREDUCE-6057.004.patch against trunk revision a122278. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 26 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-sls. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5116//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5116//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5116//artifact/patchprocess/newPatchFindbugsWarningshadoop-sls.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5116//console This message is automatically generated.
          Hide
          Ray Chiang added a comment -

          Update against trunk. Resubmit for testing.

          Show
          Ray Chiang added a comment - Update against trunk. Resubmit for testing.
          Hide
          Ray Chiang added a comment -

          RE: findbugs. All of these bugs aren't in the files changed by this patch.

          Show
          Ray Chiang added a comment - RE: findbugs. All of these bugs aren't in the files changed by this patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12685819/MAPREDUCE-6057-03.patch
          against trunk revision ffe942b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 100 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-sls.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5068//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5068//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5068//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5068//artifact/patchprocess/newPatchFindbugsWarningshadoop-sls.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5068//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685819/MAPREDUCE-6057-03.patch against trunk revision ffe942b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 100 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-sls. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5068//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5068//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5068//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5068//artifact/patchprocess/newPatchFindbugsWarningshadoop-sls.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5068//console This message is automatically generated.
          Hide
          Ray Chiang added a comment -

          Update patch against trunk.

          Show
          Ray Chiang added a comment - Update patch against trunk.
          Hide
          Ray Chiang added a comment -

          Any comments on this patch from anyone?

          Show
          Ray Chiang added a comment - Any comments on this patch from anyone?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12666402/MAPREDUCE-6057-02.patch
          against trunk revision f4caedf.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4847//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4847//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666402/MAPREDUCE-6057-02.patch against trunk revision f4caedf. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4847//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4847//console This message is automatically generated.
          Hide
          Ray Chiang added a comment -

          Re-upload same patch to see if that forces testing.

          Show
          Ray Chiang added a comment - Re-upload same patch to see if that forces testing.
          Hide
          Ray Chiang added a comment -

          Tests all pass in my tree. Resubmitting for testing.

          Show
          Ray Chiang added a comment - Tests all pass in my tree. Resubmitting for testing.
          Hide
          Ray Chiang added a comment -

          Will retry submit for testing. Would like to get 100% clean run.

          Show
          Ray Chiang added a comment - Will retry submit for testing. Would like to get 100% clean run.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12666135/MAPREDUCE-6057-01.patch
          against trunk revision 08a9ac7.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.TestLargeSort

          The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.TestMRIntermediateDataRunning
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler

          The test build failed in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4841//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4841//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666135/MAPREDUCE-6057-01.patch against trunk revision 08a9ac7. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.TestLargeSort The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestMRIntermediateDataRunning org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler The test build failed in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4841//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4841//console This message is automatically generated.
          Hide
          Ray Chiang added a comment -

          First version to submit for testing.

          Show
          Ray Chiang added a comment - First version to submit for testing.
          Hide
          Ray Chiang added a comment -

          1) Remove as obsolete:

          mapreduce.map.skip.proc.count.autoincr
          mapreduce.reduce.skip.proc.count.autoincr
          mapreduce.local.clientfactory.class.name

          2) Move to a proper member variable in MRJobConfig

          map.sort.class

          Show
          Ray Chiang added a comment - 1) Remove as obsolete: mapreduce.map.skip.proc.count.autoincr mapreduce.reduce.skip.proc.count.autoincr mapreduce.local.clientfactory.class.name 2) Move to a proper member variable in MRJobConfig map.sort.class
          Hide
          Ray Chiang added a comment -

          For map.sort.class, I see two options:

          1) Keep it as is

          2) Add it to MRJobConfig as a public static final variable and fix the reference in MapTask.java to use the new variable rather than a hardcoded string.

          Any thoughts?

          Show
          Ray Chiang added a comment - For map.sort.class, I see two options: 1) Keep it as is 2) Add it to MRJobConfig as a public static final variable and fix the reference in MapTask.java to use the new variable rather than a hardcoded string. Any thoughts?
          Hide
          Ray Chiang added a comment -

          The file

          hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml

          is used by the tests

          hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEntities.java
          hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/TestJobInfo.java

          The file

          hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json

          is used by the test

          hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/TestJobInfo.java

          If I remove the variables:

          mapreduce.map.skip.proc.count.autoincr
          mapreduce.reduce.skip.proc.count.autoincr
          mapreduce.local.clientfactory.class.name

          The tests still pass.

          Show
          Ray Chiang added a comment - The file hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml is used by the tests hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEntities.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/TestJobInfo.java The file hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json is used by the test hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/TestJobInfo.java If I remove the variables: mapreduce.map.skip.proc.count.autoincr mapreduce.reduce.skip.proc.count.autoincr mapreduce.local.clientfactory.class.name The tests still pass.
          Hide
          Ray Chiang added a comment -

          I did a more thorough searching through the files and I've found the following references (outside of mapred-default.xml):

          1) mapreduce.map.skip.proc.count.autoincr
          mapreduce.reduce.skip.proc.count.autoincr

          found in

          hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json

          2) map.sort.class

          found in

          hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json

          3) mapreduce.local.clientfactory.class.name

          found in

          hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json

          If anyone has any comments about the uses for these files or suggestion for modifications, it would be much appreciated.

          Show
          Ray Chiang added a comment - I did a more thorough searching through the files and I've found the following references (outside of mapred-default.xml): 1) mapreduce.map.skip.proc.count.autoincr mapreduce.reduce.skip.proc.count.autoincr found in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json 2) map.sort.class found in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json 3) mapreduce.local.clientfactory.class.name found in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json If anyone has any comments about the uses for these files or suggestion for modifications, it would be much appreciated.
          Hide
          Ray Chiang added a comment -

          Okay, I've got the list down to:

          map.sort.class

          • Can't find anywhere
            mapreduce.local.clientfactory.class.name
          • Can't find anywhere
            mapreduce.map.skip.proc.count.autoincr
          • Can't find anywhere
            mapreduce.reduce.skip.proc.count.autoincr
          • Can't find anywhere

          Any further feedback?

          Show
          Ray Chiang added a comment - Okay, I've got the list down to: map.sort.class Can't find anywhere mapreduce.local.clientfactory.class.name Can't find anywhere mapreduce.map.skip.proc.count.autoincr Can't find anywhere mapreduce.reduce.skip.proc.count.autoincr Can't find anywhere Any further feedback?
          Hide
          Ray Chiang added a comment -

          Thanks for all the feedback.

          I'll update the list once I update the code in YARN-2284 and re-run against all three classes. My intention was to use the properties found as a starting point.

          Show
          Ray Chiang added a comment - Thanks for all the feedback. I'll update the list once I update the code in YARN-2284 and re-run against all three classes. My intention was to use the properties found as a starting point.
          Hide
          Ray Chiang added a comment -

          Note: All the mapreduce.shuffle.* properties should be compared against ShuffleHandler class.

          Show
          Ray Chiang added a comment - Note: All the mapreduce.shuffle.* properties should be compared against ShuffleHandler class.
          Hide
          Ray Chiang added a comment -

          Note: All the mapreduce.jobhistory.* properties should be compared against JHAdminConfig class, not MRJobConfig.

          Show
          Ray Chiang added a comment - Note: All the mapreduce.jobhistory.* properties should be compared against JHAdminConfig class, not MRJobConfig.
          Hide
          Jason Lowe added a comment -

          Please be careful with this, as the entries in mapred-default.xml serve two valuable purposes:

          • Provide default values for cases where the code isn't providing a hardcoded default value
          • Documentation of property values and defaults

          So we should not remove anything that is currently being used by the code, and not everything being used is in MRJobConfig. For example, the mapreduce.jobhistory.* properties are covered by JHAdminConfig, the shufflehandler properties are covered by ShuffleHandler, etc.

          Before removing any of these properties the code base should be searched to ensure they are truly not being referenced.

          Show
          Jason Lowe added a comment - Please be careful with this, as the entries in mapred-default.xml serve two valuable purposes: Provide default values for cases where the code isn't providing a hardcoded default value Documentation of property values and defaults So we should not remove anything that is currently being used by the code, and not everything being used is in MRJobConfig. For example, the mapreduce.jobhistory.* properties are covered by JHAdminConfig, the shufflehandler properties are covered by ShuffleHandler, etc. Before removing any of these properties the code base should be searched to ensure they are truly not being referenced.

            People

            • Assignee:
              Ray Chiang
              Reporter:
              Ray Chiang
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development