Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3955

Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.2
    • Fix Version/s: 2.0.0-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common.

      1. MAPREDUCE-3955-trunk.patch
        125 kB
        Jitendra Nath Pandey
      2. MAPREDUCE-3955-trunk.patch
        130 kB
        Jitendra Nath Pandey
      3. MAPREDUCE-3955-trunk.patch
        133 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          10h 49m 2 Jitendra Nath Pandey 21/Mar/12 07:23
          Open Open Patch Available Patch Available
          19d 1h 25m 3 Jitendra Nath Pandey 21/Mar/12 07:25
          Patch Available Patch Available Resolved Resolved
          7d 18h 39m 1 Siddharth Seth 29/Mar/12 03:04
          Resolved Resolved Closed Closed
          55d 18h 23m 1 Arun C Murthy 23/May/12 21:28
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Siddharth Seth made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.0.0 [ 12320354 ]
          Resolution Fixed [ 1 ]
          Hide
          Siddharth Seth added a comment -

          Committed to trunk and branch-2. Thanks Jitendra.

          Show
          Siddharth Seth added a comment - Committed to trunk and branch-2. Thanks Jitendra.
          Siddharth Seth made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Affects Version/s 0.23.2 [ 12319851 ]
          Target Version/s 0.23.3, 0.24.0 [ 12320060, 12317654 ] 2.0.0 [ 12320354 ]
          Component/s mrv2 [ 12314301 ]
          Hide
          Siddharth Seth added a comment -

          +1. The patch looks good. YarnRemoteExceptionPBImpl.unwrapAndThrowException should really be in a separate utility class. I'll take care of that in MAPREDUCE-4067 - which should remove YarnRemoteException anyway.

          Will commit this shortly.

          Show
          Siddharth Seth added a comment - +1. The patch looks good. YarnRemoteExceptionPBImpl.unwrapAndThrowException should really be in a separate utility class. I'll take care of that in MAPREDUCE-4067 - which should remove YarnRemoteException anyway. Will commit this shortly.
          Jitendra Nath Pandey made changes -
          Link This issue blocks MAPREDUCE-4067 [ MAPREDUCE-4067 ]
          Jitendra Nath Pandey made changes -
          Target Version/s 0.23.3, 0.24.0 [ 12320060, 12317654 ] 0.24.0, 0.23.3 [ 12317654, 12320060 ]
          Priority Major [ 3 ] Blocker [ 1 ]
          Hide
          Jitendra Nath Pandey added a comment -

          Marked as blocker for 0.23.3.

          Show
          Jitendra Nath Pandey added a comment - Marked as blocker for 0.23.3.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519182/MAPREDUCE-3955-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2095//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2095//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519182/MAPREDUCE-3955-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2095//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2095//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519182/MAPREDUCE-3955-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2079//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2079//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519182/MAPREDUCE-3955-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2079//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2079//console This message is automatically generated.
          Jitendra Nath Pandey made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.3, 0.24.0 [ 12320060, 12317654 ] 0.24.0, 0.23.3 [ 12317654, 12320060 ]
          Hide
          Jitendra Nath Pandey added a comment -

          Fixed TestMRJobsWithHistoryService.

          Show
          Jitendra Nath Pandey added a comment - Fixed TestMRJobsWithHistoryService.
          Jitendra Nath Pandey made changes -
          Attachment MAPREDUCE-3955-trunk.patch [ 12519182 ]
          Jitendra Nath Pandey made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 0.23.3, 0.24.0 [ 12320060, 12317654 ] 0.24.0, 0.23.3 [ 12317654, 12320060 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519177/MAPREDUCE-3955-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2078//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2078//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519177/MAPREDUCE-3955-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2078//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2078//console This message is automatically generated.
          Jitendra Nath Pandey made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.3, 0.24.0 [ 12320060, 12317654 ] 0.24.0, 0.23.3 [ 12317654, 12320060 ]
          Jitendra Nath Pandey made changes -
          Attachment MAPREDUCE-3955-trunk.patch [ 12519177 ]
          Hide
          Jitendra Nath Pandey added a comment -

          New patch fixing the test cases.

          Show
          Jitendra Nath Pandey added a comment - New patch fixing the test cases.
          Jitendra Nath Pandey made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 0.23.3, 0.24.0 [ 12320060, 12317654 ] 0.24.0, 0.23.3 [ 12317654, 12320060 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519118/MAPREDUCE-3955-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.mapred.TestMiniMRBringup
          org.apache.hadoop.mapred.TestMiniMRChildTask
          org.apache.hadoop.mapred.TestReduceFetch
          org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
          org.apache.hadoop.mapred.TestJobCounters
          org.apache.hadoop.mapreduce.TestChild
          org.apache.hadoop.mapred.TestMiniMRClientCluster
          org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
          org.apache.hadoop.mapred.TestClientRedirect
          org.apache.hadoop.mapred.TestLazyOutput
          org.apache.hadoop.mapred.TestJobCleanup
          org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
          org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
          org.apache.hadoop.mapreduce.v2.TestRMNMInfo
          org.apache.hadoop.mapreduce.v2.TestNonExistentJob
          org.apache.hadoop.mapred.TestJobSysDirWithDFS
          org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
          org.apache.hadoop.mapreduce.security.TestJHSSecurity

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2076//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2076//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519118/MAPREDUCE-3955-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2076//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2076//console This message is automatically generated.
          Jitendra Nath Pandey made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.3, 0.24.0 [ 12320060, 12317654 ] 0.24.0, 0.23.3 [ 12317654, 12320060 ]
          Jitendra Nath Pandey made changes -
          Attachment MAPREDUCE-3955-trunk.patch [ 12519118 ]
          Jitendra Nath Pandey made changes -
          Field Original Value New Value
          Target Version/s 0.24.0, 0.23.3 [ 12317654, 12320060 ]
          Jitendra Nath Pandey created issue -

            People

            • Assignee:
              Jitendra Nath Pandey
              Reporter:
              Jitendra Nath Pandey
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development