Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1335

HDFS side of HADOOP-6904: first step towards inter-version communications between dfs client and NameNode

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: hdfs-client, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The idea is that for getProtocolVersion, NameNode checks if the client and server versions are compatible if the server version is greater than the client version. If no, throws a VersionIncompatible exception; otherwise, returns the server version.

      On the dfs client side, when creating a NameNode proxy, catches the VersionMismatch exception and then checks if the client version and the server version are compatible if the client version is greater than the server version. If not compatible, throws exception VersionIncomptible; otherwise, records the server version and continues.

      1. hdfsRPC.patch
        2 kB
        Hairong Kuang
      2. hdfsRpcVersion.patch
        5 kB
        Hairong Kuang

        Issue Links

          Activity

          Hide
          Hairong Kuang added a comment -

          Here is a patch for review.

          Show
          Hairong Kuang added a comment - Here is a patch for review.
          Hide
          Doug Cutting added a comment -

          If I understand the patch correctly, it still requires an exact version match by default. That seems good. What I don't understand is how you expect that to be altered. Do you expect folks to update the implementation of ProtocolCompatible as protocols evolve? Perhaps you can give some examples of how you expect this to work?

          Since there's more than one protocol in HDFS, do you expect to add more methods to ProtocolCompatible for each protocol?

          Show
          Doug Cutting added a comment - If I understand the patch correctly, it still requires an exact version match by default. That seems good. What I don't understand is how you expect that to be altered. Do you expect folks to update the implementation of ProtocolCompatible as protocols evolve? Perhaps you can give some examples of how you expect this to work? Since there's more than one protocol in HDFS, do you expect to add more methods to ProtocolCompatible for each protocol?
          Hide
          Hairong Kuang added a comment -

          > Do you expect folks to update the implementation of ProtocolCompatible as protocols evolve?
          Yes that's exactly what I expect developers to update.

          > Since there's more than one protocol in HDFS, do you expect to add more methods to ProtocolCompatible for each protocol?
          This is going to be done on need basis. For this jira, I intend to support only ClientProtocol (client & NameNode) compatibility.

          Show
          Hairong Kuang added a comment - > Do you expect folks to update the implementation of ProtocolCompatible as protocols evolve? Yes that's exactly what I expect developers to update. > Since there's more than one protocol in HDFS, do you expect to add more methods to ProtocolCompatible for each protocol? This is going to be done on need basis. For this jira, I intend to support only ClientProtocol (client & NameNode) compatibility.
          Hide
          Doug Cutting added a comment -

          Thanks for elaborating. This patch seems fine: it's simple and does not alone introduce significant risk.

          However subsequent changes to ProtocolCompatible will need to be carefully reviewed and tested. We don't yet have a testing framework to do that. So we're delaying that work (interoperability testing of different versions) until we actually have two versions that are meant to interoperate. This is a technical debt will be due in full the first time ProtocolCompatible is altered, I think.

          Show
          Doug Cutting added a comment - Thanks for elaborating. This patch seems fine: it's simple and does not alone introduce significant risk. However subsequent changes to ProtocolCompatible will need to be carefully reviewed and tested. We don't yet have a testing framework to do that. So we're delaying that work (interoperability testing of different versions) until we actually have two versions that are meant to interoperate. This is a technical debt will be due in full the first time ProtocolCompatible is altered, I think.
          Hide
          dhruba borthakur added a comment -

          This approach sounds good. Do we need to implement the same mechanisms for DatanodeProtocol and NameNodeProtocol as well?

          Show
          dhruba borthakur added a comment - This approach sounds good. Do we need to implement the same mechanisms for DatanodeProtocol and NameNodeProtocol as well?
          Hide
          Hairong Kuang added a comment -

          Here is the patch that makes HDFS to work with new method-based RPC compatibility protocol.

          Show
          Hairong Kuang added a comment - Here is the patch that makes HDFS to work with new method-based RPC compatibility protocol.
          Hide
          dhruba borthakur added a comment -

          +1, code looks great!

          Show
          dhruba borthakur added a comment - +1, code looks great!
          Hide
          Hairong Kuang added a comment -

          Since HADOOP-6904 breaks the compilation of HDFS, the patch can not go through hudson. I just committed the patch.

          Show
          Hairong Kuang added a comment - Since HADOOP-6904 breaks the compilation of HDFS, the patch can not go through hudson. I just committed the patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #539 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/539/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #539 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/539/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
          Hide
          Denny Ye added a comment -

          Does Hadoop-0.23 contains this patch?

          Show
          Denny Ye added a comment - Does Hadoop-0.23 contains this patch?

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development