Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3453

HDFS does not use ClientProtocol in a backward-compatible way

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      HDFS-617 was brought into branch-0.20-security/branch-1 to support non-recursive create, along with HADOOP-6840 and HADOOP-6886. However, the changes in HDFS was done in an incompatible way, making the client unusable against older clusters, even when plain old create() is called. This is because DFS now internally calls create() through the newly introduced method. By simply changing how the methods are wired internally, we can remove this limitation. We may eventually switch back to the approach in HDFS-617 when the majority of users adopt branch-1 based releases.

      1. hdfs-3453-branch-1.patch
        1.0 kB
        Kihwal Lee
      2. hdfs-3453-branch-1.patch
        1 kB
        Kihwal Lee

        Activity

        Hide
        Kihwal Lee added a comment -

        The attached patch changes how namenode.create() is called. If createParent is false (regular create), the old create() method is used. This only changes how things are internally wired and does not introduce any incompatibility.

        Show
        Kihwal Lee added a comment - The attached patch changes how namenode.create() is called. If createParent is false (regular create), the old create() method is used. This only changes how things are internally wired and does not introduce any incompatibility.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528595/hdfs-3453-branch-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2507//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528595/hdfs-3453-branch-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2507//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        I successfully ran smoke and commit on my desktop. No test is included.

        Show
        Kihwal Lee added a comment - I successfully ran smoke and commit on my desktop. No test is included.
        Hide
        Kihwal Lee added a comment -

        Manually ran test-patch

             [exec] -1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
             [exec]                         Please justify why no tests are needed for this patch.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     -1 findbugs.  The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.
        
        • test : No test was added since there is no functional change within the build/release.
        • findbugs : Contrary to the report, this change does not introduce any additional findbug warnings.
        Show
        Kihwal Lee added a comment - Manually ran test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. test : No test was added since there is no functional change within the build/release. findbugs : Contrary to the report, this change does not introduce any additional findbug warnings.
        Hide
        Kihwal Lee added a comment -

        A bit more on the incompatibility:

        ClientProtocol and namenode still have the original method and implementation. So pre-1.0 clients can still talk to 1.0.x namenodes. The 1.0.x client, on the other hand, internally calls only the new method; The original create method is never called. Since pre-1.0 namenode does not understand this new method, 1.0 clients cannot create files on pre-1.0 clusters. The patch makes client call the new method only when createParent is set to false. All others will be fulfilled by the original method.

        The protocol-level incompatibility still exists, but that only manifests when the new file system level method, FileSystem#createNonRecursive(), is explicitly called. Normal use cases within hadoop do not use this method. Currently only SequenceFile#createWriter() calls it when caller specifically asks for it. This was added to support HBase.

        In summary, 1.0.x and pre-1.0 are incompatible, but become interoperable with this patch for most use cases.

        Show
        Kihwal Lee added a comment - A bit more on the incompatibility: ClientProtocol and namenode still have the original method and implementation. So pre-1.0 clients can still talk to 1.0.x namenodes. The 1.0.x client, on the other hand, internally calls only the new method; The original create method is never called. Since pre-1.0 namenode does not understand this new method, 1.0 clients cannot create files on pre-1.0 clusters. The patch makes client call the new method only when createParent is set to false. All others will be fulfilled by the original method. The protocol-level incompatibility still exists, but that only manifests when the new file system level method, FileSystem#createNonRecursive(), is explicitly called. Normal use cases within hadoop do not use this method. Currently only SequenceFile#createWriter() calls it when caller specifically asks for it. This was added to support HBase. In summary, 1.0.x and pre-1.0 are incompatible, but become interoperable with this patch for most use cases.
        Hide
        Kihwal Lee added a comment -

        We re-tested a build with this patch and the previously broken cases (writing to 0.20.205.0 using 1.0.x client) are working now. All other core tests also ran successfully. This patch does not introduce any incompatibility to previous 1.0.x releases.

        Show
        Kihwal Lee added a comment - We re-tested a build with this patch and the previously broken cases (writing to 0.20.205.0 using 1.0.x client) are working now. All other core tests also ran successfully. This patch does not introduce any incompatibility to previous 1.0.x releases.
        Hide
        Suresh Srinivas added a comment -

        Finally got some time spend on this and understand the issue. This is a good change.

        +1 for the patch.

        A small comment. Can add to the comment Make sure the regular create() is done through the old create(), why this is being done. Say Ensure create() is done using old create() without createFlag. This allows newer clients that support createFlag to be interop with older server implementation that do not createFlag in create().

        Show
        Suresh Srinivas added a comment - Finally got some time spend on this and understand the issue. This is a good change. +1 for the patch. A small comment. Can add to the comment Make sure the regular create() is done through the old create() , why this is being done. Say Ensure create() is done using old create() without createFlag. This allows newer clients that support createFlag to be interop with older server implementation that do not createFlag in create().
        Hide
        Kihwal Lee added a comment -

        New patch: comments added as Suresh suggested.

        Show
        Kihwal Lee added a comment - New patch: comments added as Suresh suggested.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12530042/hdfs-3453-branch-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2531//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530042/hdfs-3453-branch-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2531//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        I committed the patch. Thank you Kihwal.

        Show
        Suresh Srinivas added a comment - I committed the patch. Thank you Kihwal.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Merged to branch-1.1.

        Show
        Tsz Wo Nicholas Sze added a comment - Merged to branch-1.1.
        Hide
        Matt Foley added a comment -

        Closed upon release of Hadoop-1.1.0.

        Show
        Matt Foley added a comment - Closed upon release of Hadoop-1.1.0.

          People

          • Assignee:
            Kihwal Lee
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development