Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-195

Need to handle access token expiration when re-establishing the pipeline for dfs write

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      Original access token may have expired when re-establishing the pipeline within processDatanodeError().

      1. 6048-09.patch
        14 kB
        Kan Zhang
      2. 6048-11.patch
        15 kB
        Kan Zhang
      3. 6048-12.patch
        11 kB
        Kan Zhang
      4. 6048-14.patch
        11 kB
        Kan Zhang
      5. 6048-15.patch
        11 kB
        Kan Zhang
      6. 6048-16.patch
        11 kB
        Kan Zhang
      7. 6048-17.patch
        12 kB
        Kan Zhang
      8. 6048-18.patch
        11 kB
        Kan Zhang
      9. HDFS-195.patch
        11 kB
        Raghu Angadi
      10. HDFS-195.patch
        11 kB
        Raghu Angadi
      11. HDFS-195-0_20.1.patch
        12 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          Hide
          Jitendra Nath Pandey added a comment -

          Patch for hadoop 20 added.

          Show
          Jitendra Nath Pandey added a comment - Patch for hadoop 20 added.
          Hide
          Raghu Angadi added a comment -

          I just committed this. Thanks Kan.

          Show
          Raghu Angadi added a comment - I just committed this. Thanks Kan.
          Hide
          Raghu Angadi added a comment -

          Updated patch for new trunk. The previous one was incorrect.

          Show
          Raghu Angadi added a comment - Updated patch for new trunk. The previous one was incorrect.
          Hide
          Raghu Angadi added a comment -

          +1 for the patch.

          I am attaching the hdfs trunk version of the patch.

          Show
          Raghu Angadi added a comment - +1 for the patch. I am attaching the hdfs trunk version of the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12411172/6048-18.patch
          against trunk revision 786278.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12411172/6048-18.patch against trunk revision 786278. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/console This message is automatically generated.
          Hide
          Kan Zhang added a comment -

          attached a new patch, which bumps up the ClientDatanodeProtocol version to signal the fact that now recoverBlock() never returns null and always returns a newly generated access token as part of the return value. Also, reverted the changes made in previous patches that changed the way how lastException is set by sync() and createBlockOutputStream().

          Show
          Kan Zhang added a comment - attached a new patch, which bumps up the ClientDatanodeProtocol version to signal the fact that now recoverBlock() never returns null and always returns a newly generated access token as part of the return value. Also, reverted the changes made in previous patches that changed the way how lastException is set by sync() and createBlockOutputStream().
          Hide
          Kan Zhang added a comment -

          attached a new patch. removed the testing for InvalidAccessTokenException in processDatanodeError() since it can never happen. processDatanodeError() is called in 2 places. 1) when an error happens while writing. since the pipeline has already been set up in this case, the error can't be access token error. 2) when initializing for append, in which case, there is no previous error yet.

          Show
          Kan Zhang added a comment - attached a new patch. removed the testing for InvalidAccessTokenException in processDatanodeError() since it can never happen. processDatanodeError() is called in 2 places. 1) when an error happens while writing. since the pipeline has already been set up in this case, the error can't be access token error. 2) when initializing for append, in which case, there is no previous error yet.
          Hide
          Kan Zhang added a comment -

          one cosmetic change.

          Show
          Kan Zhang added a comment - one cosmetic change.
          Hide
          Kan Zhang added a comment -

          To be clear to the reader, the last patch removed null checking on the return value from recoverBlock() since it always returns a LocatedBlock object or throw an exception.

          Show
          Kan Zhang added a comment - To be clear to the reader, the last patch removed null checking on the return value from recoverBlock() since it always returns a LocatedBlock object or throw an exception.
          Hide
          Kan Zhang added a comment -

          minor updates

          Show
          Kan Zhang added a comment - minor updates
          Hide
          Kan Zhang added a comment -

          attaching a new patch, which relies on block recovery to always return a newly generated access token, so that there is no need for adding a new method to ClientProtocol to get access token.

          Show
          Kan Zhang added a comment - attaching a new patch, which relies on block recovery to always return a newly generated access token, so that there is no need for adding a new method to ClientProtocol to get access token.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The is a re-design of block recovery in HADOOP-5744. Adding a link.

          Show
          Tsz Wo Nicholas Sze added a comment - The is a re-design of block recovery in HADOOP-5744 . Adding a link.
          Hide
          Kan Zhang added a comment -

          updated ClientProtocol version.

          Show
          Kan Zhang added a comment - updated ClientProtocol version.
          Hide
          Kan Zhang added a comment -

          this patch does the following.
          1. when the primary datanode recovers a block, it sends back a newly generated access token along a new generation stamp if any.
          2. add a namenode method so that a DFSClient can call to get a new access token when access token becomes expired during datanode error recovery.
          3. To differentiate access token error from other errors, datanode now needs to send back a status code together with firstBadLink during pipeline setup for dfs write (only for DFSClients, not for other datanodes).

          Show
          Kan Zhang added a comment - this patch does the following. 1. when the primary datanode recovers a block, it sends back a newly generated access token along a new generation stamp if any. 2. add a namenode method so that a DFSClient can call to get a new access token when access token becomes expired during datanode error recovery. 3. To differentiate access token error from other errors, datanode now needs to send back a status code together with firstBadLink during pipeline setup for dfs write (only for DFSClients, not for other datanodes).

            People

            • Assignee:
              Kan Zhang
              Reporter:
              Kan Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development