Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    1. hadoop-10044.patch
      13 kB
      Sanjay Radia
    2. HADOOP-10044.20131014.patch
      13 kB
      Jing Zhao

      Activity

      Hide
      sanjay.radia Sanjay Radia added a comment -

      The hadoop rpc code especially the code in Server.java is fairly complicated and poorly documented. Everytime I make changes there or try and debug an issue, I have relearn parts of the code. The javadoc needs to be improved.

      Show
      sanjay.radia Sanjay Radia added a comment - The hadoop rpc code especially the code in Server.java is fairly complicated and poorly documented. Everytime I make changes there or try and debug an issue, I have relearn parts of the code. The javadoc needs to be improved.
      Hide
      hadoopqa Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12608119/hadoop-10044.patch
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 eclipse:eclipse. The patch built with eclipse:eclipse.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3209//testReport/
      Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3209//console

      This message is automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12608119/hadoop-10044.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3209//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3209//console This message is automatically generated.
      Hide
      stevel@apache.org Steve Loughran added a comment -

      +1 to a javadoc patch, but as someone who has recently done some RPC work, I think we need something that gives an overall view of how to implement RPC services.

      Particular troublespots for me

      • basics of registering a service, client proxy
      • why turning Simple auth on is a good feature during dev
      • moving to protobuf as a payload (why, how)
      • moving to protbuf RPC with the various relay proxies. Here the why really has to come before the how, as having done this once I think you really need a good reason to invest the effort.
      • security, delegation tokens, auth: basic services, then under YARN
      • failure modes, testing

      YARN is going to enable many more people to write distributed apps, which means more people trying to understand the RPC mechanism

      Show
      stevel@apache.org Steve Loughran added a comment - +1 to a javadoc patch, but as someone who has recently done some RPC work, I think we need something that gives an overall view of how to implement RPC services. Particular troublespots for me basics of registering a service, client proxy why turning Simple auth on is a good feature during dev moving to protobuf as a payload (why, how) moving to protbuf RPC with the various relay proxies. Here the why really has to come before the how, as having done this once I think you really need a good reason to invest the effort. security, delegation tokens, auth: basic services, then under YARN failure modes, testing YARN is going to enable many more people to write distributed apps, which means more people trying to understand the RPC mechanism
      Hide
      jingzhao Jing Zhao added a comment -

      Thanks Sanjay! The javadoc is very helpful to understand the RPC code. Some minor comments about the current patch:

      1. In RpcConstants.java,
        /**
          * The Rpc-connection header is as follows 
          * +----------------------------------+
          * |  "hrpc" 4 bytes                  |      
          * +----------------------------------+
          * |  Version (1 byte)                |
          * +----------------------------------+
          * |  Service Class (1 byte)          |
          * +----------------------------------+
          * |  AuthProtocol (1 byte)           |      
          * +----------------------------------+
          */
        

        can be moved to somewhere in Server.java (e.g., as javadoc for Connection or Connection#readAndProcess) ?

      2. In Connection#saslProcess, the AccessCotnrolException thrown by
        processSaslMessage will be caught by the outside try-catch, and will be wrapped as a WrappedRpcServerException. Thus we do not need to add it to signature and javadoc.
      3. Do we need to javadoc the mechanism about how to create SASL server based on different auth methods?

      Besides there're some typos and I just modify them directly on the original patch. Upload a new patch fixing typos.

      Show
      jingzhao Jing Zhao added a comment - Thanks Sanjay! The javadoc is very helpful to understand the RPC code. Some minor comments about the current patch: In RpcConstants.java, /** * The Rpc-connection header is as follows * +----------------------------------+ * | "hrpc" 4 bytes | * +----------------------------------+ * | Version (1 byte ) | * +----------------------------------+ * | Service Class (1 byte ) | * +----------------------------------+ * | AuthProtocol (1 byte ) | * +----------------------------------+ */ can be moved to somewhere in Server.java (e.g., as javadoc for Connection or Connection#readAndProcess) ? In Connection#saslProcess, the AccessCotnrolException thrown by processSaslMessage will be caught by the outside try-catch, and will be wrapped as a WrappedRpcServerException. Thus we do not need to add it to signature and javadoc. Do we need to javadoc the mechanism about how to create SASL server based on different auth methods? Besides there're some typos and I just modify them directly on the original patch. Upload a new patch fixing typos.
      Hide
      hadoopqa Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12608377/HADOOP-10044.20131014.patch
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 eclipse:eclipse. The patch built with eclipse:eclipse.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3214//testReport/
      Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3214//console

      This message is automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12608377/HADOOP-10044.20131014.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3214//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3214//console This message is automatically generated.
      Hide
      stevel@apache.org Steve Loughran added a comment -

      If the new javadocs have preformatted code, it should be wrapped with <pre></pre> tags, so that the generated javadocs look OK too

      Show
      stevel@apache.org Steve Loughran added a comment - If the new javadocs have preformatted code, it should be wrapped with <pre></pre> tags, so that the generated javadocs look OK too
      Hide
      hadoopqa Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12608377/HADOOP-10044.20131014.patch
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 eclipse:eclipse. The patch built with eclipse:eclipse.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      -1 core tests. The following test timeouts occurred in hadoop-common-project/hadoop-common:

      org.apache.hadoop.http.TestHttpServer

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3356//testReport/
      Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3356//console

      This message is automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12608377/HADOOP-10044.20131014.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-common-project/hadoop-common: org.apache.hadoop.http.TestHttpServer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3356//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3356//console This message is automatically generated.
      Hide
      sanjay.radia Sanjay Radia added a comment -

      The failed test timeout is unrelated (and I also ran it successfully).
      Committed.

      Show
      sanjay.radia Sanjay Radia added a comment - The failed test timeout is unrelated (and I also ran it successfully). Committed.
      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Hadoop-trunk-Commit #4874 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4874/)
      HADOOP-10044 Improve the javadoc of rpc code (sanjay Radia) (sradia: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550486)

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4874 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4874/ ) HADOOP-10044 Improve the javadoc of rpc code (sanjay Radia) (sradia: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550486 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Yarn-trunk #420 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/420/)
      HADOOP-10044 Improve the javadoc of rpc code (sanjay Radia) (sradia: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550486)

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #420 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/420/ ) HADOOP-10044 Improve the javadoc of rpc code (sanjay Radia) (sradia: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550486 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Hadoop-Hdfs-trunk #1611 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1611/)
      HADOOP-10044 Improve the javadoc of rpc code (sanjay Radia) (sradia: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550486)

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1611 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1611/ ) HADOOP-10044 Improve the javadoc of rpc code (sanjay Radia) (sradia: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550486 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Mapreduce-trunk #1637 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1637/)
      HADOOP-10044 Improve the javadoc of rpc code (sanjay Radia) (sradia: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550486)

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1637 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1637/ ) HADOOP-10044 Improve the javadoc of rpc code (sanjay Radia) (sradia: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1550486 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcConstants.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
      Hide
      yzhangal Yongjun Zhang added a comment -

      Hi Sanjay Radia,

      Thanks for your earlier work on this. I found that the commit of this jira is missing from branch-2. May I know if there is any concern of putting this to branch-2?

      Thanks.

      Show
      yzhangal Yongjun Zhang added a comment - Hi Sanjay Radia , Thanks for your earlier work on this. I found that the commit of this jira is missing from branch-2. May I know if there is any concern of putting this to branch-2? Thanks.
      Hide
      vinodkv Vinod Kumar Vavilapalli added a comment -

      Closing old tickets that are already shipped in a release.

      Show
      vinodkv Vinod Kumar Vavilapalli added a comment - Closing old tickets that are already shipped in a release.

        People

        • Assignee:
          sanjay.radia Sanjay Radia
          Reporter:
          sanjay.radia Sanjay Radia
        • Votes:
          0 Vote for this issue
          Watchers:
          8 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development