Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5762

Port MAPREDUCE-3223 and MAPREDUCE-4695 (Remove MRv1 config from mapred-default.xml) to branch-2

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.8.0
    • Component/s: documentation
    • Labels:
      None
    • Target Version/s:

      Description

      MRv1 configs are removed in trunk, but they are not removed in branch-2.

      1. MAPREDUCE-5762.003.branch-2.patch
        24 kB
        Akira Ajisaka
      2. MAPREDUCE-5762-branch-2.03.patch
        24 kB
        Akira Ajisaka
      3. MAPREDUCE-5762-branch-2.patch
        24 kB
        Akira Ajisaka
      4. MAPREDUCE-5762-branch-2-002.patch
        23 kB
        Akira Ajisaka

        Issue Links

          Activity

          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch for branch-2.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch for branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12630415/MAPREDUCE-5762-branch-2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4369//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12630415/MAPREDUCE-5762-branch-2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4369//console This message is automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          The test will fail because the patch is for branch-2. I think the patch is ready to merge in.

          Show
          ajisakaa Akira Ajisaka added a comment - The test will fail because the patch is for branch-2. I think the patch is ready to merge in.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Rebased for the latest branch-2.

          Show
          ajisakaa Akira Ajisaka added a comment - Rebased for the latest branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707722/MAPREDUCE-5762-branch-2-002.patch
          against trunk revision af618f2.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5341//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707722/MAPREDUCE-5762-branch-2-002.patch against trunk revision af618f2. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5341//console This message is automatically generated.
          Hide
          qwertymaniac Harsh J added a comment -

          Went over the entire property list, and the removals look fine to me.

          +1, please commit. Many thanks for the work!

          Show
          qwertymaniac Harsh J added a comment - Went over the entire property list, and the removals look fine to me. +1, please commit. Many thanks for the work!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed v2 patch to branch-2. Thanks Harsh J for review!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed v2 patch to branch-2. Thanks Harsh J for review!
          Hide
          mingma Ming Ma added a comment -

          Akira Ajisaka it looks like mapreduce.cluster.local.dir was removed as part of this and caused some branch-2 unit tests to fail.

          Show
          mingma Ming Ma added a comment - Akira Ajisaka it looks like mapreduce.cluster.local.dir was removed as part of this and caused some branch-2 unit tests to fail.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Ming Ma for pointing out. What unit tests are failing in branch-2? Are the tests fail in trunk?
          Anyway I'm +1 for adding "mapreduce.cluster.local.dir".

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Ming Ma for pointing out. What unit tests are failing in branch-2? Are the tests fail in trunk? Anyway I'm +1 for adding "mapreduce.cluster.local.dir".
          Hide
          mingma Ming Ma added a comment -

          Akira Ajisaka, here is an example of test failures. The property is defined in trunk.

          Running org.apache.hadoop.mapreduce.task.reduce.TestMerger
          Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.625 sec <<< FAILURE! - in org.apache.hadoop.mapreduce.task.reduce.TestMerger
          testEncryptedMerger(org.apache.hadoop.mapreduce.task.reduce.TestMerger)  Time elapsed: 0.092 sec  <<< ERROR!
          java.lang.NullPointerException: null
          	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
          	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
          	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
          	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
          	at org.apache.hadoop.mapred.MROutputFiles.getInputFileForWrite(MROutputFiles.java:206)
          	at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$InMemoryMerger.merge(MergeManagerImpl.java:459)
          	at org.apache.hadoop.mapreduce.task.reduce.TestMerger.testInMemoryAndOnDiskMerger(TestMerger.java:136)
          	at org.apache.hadoop.mapreduce.task.reduce.TestMerger.testEncryptedMerger(TestMerger.java:92)
          
          Show
          mingma Ming Ma added a comment - Akira Ajisaka , here is an example of test failures. The property is defined in trunk. Running org.apache.hadoop.mapreduce.task.reduce.TestMerger Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.625 sec <<< FAILURE! - in org.apache.hadoop.mapreduce.task.reduce.TestMerger testEncryptedMerger(org.apache.hadoop.mapreduce.task.reduce.TestMerger) Time elapsed: 0.092 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.mapred.MROutputFiles.getInputFileForWrite(MROutputFiles.java:206) at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$InMemoryMerger.merge(MergeManagerImpl.java:459) at org.apache.hadoop.mapreduce.task.reduce.TestMerger.testInMemoryAndOnDiskMerger(TestMerger.java:136) at org.apache.hadoop.mapreduce.task.reduce.TestMerger.testEncryptedMerger(TestMerger.java:92)
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Ming Ma for the information. I forgot to port MAPREDUCE-4695.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Ming Ma for the information. I forgot to port MAPREDUCE-4695 .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I'll revert this commit and attach a new patch.

          Show
          ajisakaa Akira Ajisaka added a comment - I'll revert this commit and attach a new patch.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Harsh J and Ming Ma, would you review the new patch?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Harsh J and Ming Ma , would you review the new patch?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732480/MAPREDUCE-5762.003.branch-2.patch
          Optional Tests javadoc javac unit
          git revision trunk / fcd0702
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5722/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732480/MAPREDUCE-5762.003.branch-2.patch Optional Tests javadoc javac unit git revision trunk / fcd0702 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5722/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Renamed the patch for jenkins to test in branch-2.

          Show
          ajisakaa Akira Ajisaka added a comment - Renamed the patch for jenkins to test in branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 33s Pre-patch branch-2 compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 5m 49s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 18s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 mapreduce tests 1m 31s Tests passed in hadoop-mapreduce-client-core.
              33m 44s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732489/MAPREDUCE-5762-branch-2.03.patch
          Optional Tests javadoc javac unit
          git revision branch-2 / 345e62c
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5723/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5723/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5723/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 33s Pre-patch branch-2 compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 5m 49s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 18s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 mapreduce tests 1m 31s Tests passed in hadoop-mapreduce-client-core.     33m 44s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732489/MAPREDUCE-5762-branch-2.03.patch Optional Tests javadoc javac unit git revision branch-2 / 345e62c hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5723/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5723/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5723/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          LGTM.

          Show
          mingma Ming Ma added a comment - LGTM.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Ming Ma for the review. Could you give +1 for the patch?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Ming Ma for the review. Could you give +1 for the patch?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          After this issue is fixed, I'd like to port MAPREDUCE-6057 to branch-2 to remove unused MRv1 config completely.

          Show
          ajisakaa Akira Ajisaka added a comment - After this issue is fixed, I'd like to port MAPREDUCE-6057 to branch-2 to remove unused MRv1 config completely.
          Hide
          mingma Ming Ma added a comment -

          +1. Thanks Akira Ajisaka.

          Show
          mingma Ming Ma added a comment - +1. Thanks Akira Ajisaka .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed the latest patch to branch-2. Thanks Ming Ma!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed the latest patch to branch-2. Thanks Ming Ma !

            People

            • Assignee:
              ajisakaa Akira Ajisaka
              Reporter:
              ajisakaa Akira Ajisaka
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development