Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4871

AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize

        Details

        • Type: Bug
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 0.23.3, 2.0.2-alpha
        • Fix Version/s: 0.23.7, 2.1.0-beta
        • Component/s: mrv2
        • Labels:
          None

          Description

          When the user needs to configure a larger split metainfo file size, mapred-default.xml points to the mapreduce.job.split.metainfo.maxsize property. However the ApplicationMaster actually uses the mapreduce.jobtracker.split.metainfo.maxsize property when determining the largest allowed size. This leads to much confusion on the part of end-users trying to increase the allowed limit.

            Issue Links

            relates to

            Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4926 Change documentation to use mapreduce.jobtracker.split.metainfo.maxsize

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-2760 mapreduce.jobtracker.split.metainfo.maxsize typoed in mapred-default.xml

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              jlowe Jason Lowe added a comment -

              This is similar to MAPREDUCE-2760, but it appears that change was undone when a merge cleanup from MAPREDUCE-279 was integrated.

              At this point there isn't a jobtracker, so I think we should deprecate mapreduce.jobtracker.split.metainfo.maxsize to mapreduce.job.split.metainfo.maxsize so we can continue to honor both methods of setting the max metainfo size.

              Show
              jlowe Jason Lowe added a comment - This is similar to MAPREDUCE-2760 , but it appears that change was undone when a merge cleanup from MAPREDUCE-279 was integrated. At this point there isn't a jobtracker, so I think we should deprecate mapreduce.jobtracker.split.metainfo.maxsize to mapreduce.job.split.metainfo.maxsize so we can continue to honor both methods of setting the max metainfo size.
              Hide
              Permalink
              jlowe Jason Lowe added a comment -

              Patch to deprecate mapreduce.jobtracker.split.metainfo.maxsize to mapreduce.job.split.metainfo.maxsize and update AM to use mapreduce.job.split.metainfo.maxsize.

              Show
              jlowe Jason Lowe added a comment - Patch to deprecate mapreduce.jobtracker.split.metainfo.maxsize to mapreduce.job.split.metainfo.maxsize and update AM to use mapreduce.job.split.metainfo.maxsize.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12563838/MAPREDUCE-4871.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3214//testReport/
              Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3214//console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563838/MAPREDUCE-4871.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3214//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3214//console This message is automatically generated.
              Hide
              Permalink
              lohit Lohit Vijayarenu added a comment -

              Changes look good to me. One thing is the change will make config incompatible. I assume that is intentional.

              Show
              lohit Lohit Vijayarenu added a comment - Changes look good to me. One thing is the change will make config incompatible. I assume that is intentional.
              Hide
              Permalink
              jlowe Jason Lowe added a comment -

              mapreduce.jobtracker.split.metainfo.splitsize is still honored, but deprecated. Using the old property name will internally set mapreduce.job.split.metainfo.maxsize and emit a deprecation warning, but otherwise the job should run as it did before. I manually verified that using the old property name the AM was looking for still works.

              Show
              jlowe Jason Lowe added a comment - mapreduce.jobtracker.split.metainfo.splitsize is still honored, but deprecated. Using the old property name will internally set mapreduce.job.split.metainfo.maxsize and emit a deprecation warning, but otherwise the job should run as it did before. I manually verified that using the old property name the AM was looking for still works.
              Hide
              Permalink
              jeagles Jonathan Eagles added a comment -

              +1 lgtm

              Show
              jeagles Jonathan Eagles added a comment - +1 lgtm
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-trunk-Commit #3397 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3397/)
              MAPREDUCE-4871. AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via jeagles) (Revision 1451318)

              Result = SUCCESS
              jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451318
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-trunk-Commit #3397 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3397/ ) MAPREDUCE-4871 . AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via jeagles) (Revision 1451318) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451318 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Yarn-trunk #142 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/142/)
              MAPREDUCE-4871. AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via jeagles) (Revision 1451318)

              Result = SUCCESS
              jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451318
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-Yarn-trunk #142 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/142/ ) MAPREDUCE-4871 . AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via jeagles) (Revision 1451318) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451318 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Hdfs-0.23-Build #540 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/540/)
              MAPREDUCE-4871. AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via jeagles) (Revision 1451321)

              Result = SUCCESS
              jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451321
              Files :

              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java
              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #540 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/540/ ) MAPREDUCE-4871 . AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via jeagles) (Revision 1451321) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451321 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Hdfs-trunk #1331 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1331/)
              MAPREDUCE-4871. AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via jeagles) (Revision 1451318)

              Result = FAILURE
              jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451318
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1331 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1331/ ) MAPREDUCE-4871 . AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via jeagles) (Revision 1451318) Result = FAILURE jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451318 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-Mapreduce-trunk #1359 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1359/)
              MAPREDUCE-4871. AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via jeagles) (Revision 1451318)

              Result = SUCCESS
              jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451318
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
              Show
              hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1359 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1359/ ) MAPREDUCE-4871 . AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via jeagles) (Revision 1451318) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451318 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/SplitMetaInfoReader.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java

                People

                • Assignee:
                  jlowe Jason Lowe
                  Reporter:
                  jlowe Jason Lowe
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  8 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development