Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: None
    • Labels:
      None

      Description

      HDFS-3835 does not cleanly merge into branch-0.23. This is to port it over.

      1. HDFS-3841.txt
        4 kB
        Robert Joseph Evans
      2. HDFS-3841.txt
        4 kB
        Robert Joseph Evans
      3. HDFS-3841.txt
        7 kB
        Robert Joseph Evans

        Activity

        Robert Joseph Evans created issue -
        Hide
        Robert Joseph Evans added a comment -

        This patch only applies to branch-0.23. The main difference between this patch and HDFS-3835 is that the DelegationTokenSecretManager is in a different location so FSImage was modified to use the new location. Also The tests do not compile because HDFS-2579 is not part of 0.23 so DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY does not work. In response I removed the test. It seemed more risky to try and pull out DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY support then to simply remove the test.

        Show
        Robert Joseph Evans added a comment - This patch only applies to branch-0.23. The main difference between this patch and HDFS-3835 is that the DelegationTokenSecretManager is in a different location so FSImage was modified to use the new location. Also The tests do not compile because HDFS-2579 is not part of 0.23 so DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY does not work. In response I removed the test. It seemed more risky to try and pull out DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY support then to simply remove the test.
        Robert Joseph Evans made changes -
        Field Original Value New Value
        Attachment HDFS-3841.txt [ 12542028 ]
        Hide
        Robert Joseph Evans added a comment -

        test-patch is not going to work. I have run several of the HDFS tests manually without any failures. I will update the JIRA once my test run completes.

        Show
        Robert Joseph Evans added a comment - test-patch is not going to work. I have run several of the HDFS tests manually without any failures. I will update the JIRA once my test run completes.
        Robert Joseph Evans made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Eli Collins added a comment -

        This patch modifies TestCheckpoint, intentional? Sounded like from your above comment you mean to remove that diff.

        Show
        Eli Collins added a comment - This patch modifies TestCheckpoint, intentional? Sounded like from your above comment you mean to remove that diff.
        Hide
        Robert Joseph Evans added a comment -

        You are correct, by bad. I commented it out to validate that it runs, and I forgot to remove it. I'll upload a new patch. Thanks for the catch.

        Show
        Robert Joseph Evans added a comment - You are correct, by bad. I commented it out to validate that it runs, and I forgot to remove it. I'll upload a new patch. Thanks for the catch.
        Hide
        Robert Joseph Evans added a comment -

        Patch updated without test.

        Show
        Robert Joseph Evans added a comment - Patch updated without test.
        Robert Joseph Evans made changes -
        Attachment HDFS-3841.txt [ 12542030 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12542030/HDFS-3841.txt
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3069//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542030/HDFS-3841.txt against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3069//console This message is automatically generated.
        Hide
        Daryn Sharp added a comment -

        +1 But please put a space between the if and the paren.

        Show
        Daryn Sharp added a comment - +1 But please put a space between the if and the paren.
        Hide
        Robert Joseph Evans added a comment -

        Patch with space between if and (

        Show
        Robert Joseph Evans added a comment - Patch with space between if and (
        Robert Joseph Evans made changes -
        Attachment HDFS-3841.txt [ 12542032 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12542032/HDFS-3841.txt
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3070//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542032/HDFS-3841.txt against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3070//console This message is automatically generated.
        Eli Collins made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Eli Collins added a comment -

        +1 latest patch looks good

        Show
        Eli Collins added a comment - +1 latest patch looks good
        Hide
        Aaron T. Myers added a comment -

        +1, the patch looks good to me.

        Note that it's pretty easy to test this manually, if you wanted to, using a secure pseudo-cluster:

        1. Start up the NN and 2NN, with the checkpoint period set low.
        2. Get a DT using `hdfs fetchdt'
        3. Make sure the 2NN performs a checkpoint.
        4. Restart the NN, which on startup will merge the edits and fsimage.
        5. Without restarting the 2NN, make sure it successfully performs another checkpoint.
        Show
        Aaron T. Myers added a comment - +1, the patch looks good to me. Note that it's pretty easy to test this manually, if you wanted to, using a secure pseudo-cluster: Start up the NN and 2NN, with the checkpoint period set low. Get a DT using `hdfs fetchdt' Make sure the 2NN performs a checkpoint. Restart the NN, which on startup will merge the edits and fsimage. Without restarting the 2NN, make sure it successfully performs another checkpoint.
        Hide
        Robert Joseph Evans added a comment -

        Thanks for the reviews. The HDFS unit tests all pass. I asked Daryn to check it in when he gets a chance.

        Show
        Robert Joseph Evans added a comment - Thanks for the reviews. The HDFS unit tests all pass. I asked Daryn to check it in when he gets a chance.
        Hide
        Robert Joseph Evans added a comment -

        Daryn just checked this in to branch-0.23

        Show
        Robert Joseph Evans added a comment - Daryn just checked this in to branch-0.23
        Robert Joseph Evans made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.23.3 [ 12320052 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #353 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/353/)
        HDFS-3841. Port HDFS-3835 to branch-0.23 (bobby via daryn) (Revision 1376485)

        Result = UNSTABLE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376485
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #353 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/353/ ) HDFS-3841 . Port HDFS-3835 to branch-0.23 (bobby via daryn) (Revision 1376485) Result = UNSTABLE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376485 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322472 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Robert Joseph Evans
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development