Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: HDFS-5535 (Rolling upgrades)
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestRollingUpgrade#testSecondaryNameNode causes subsequent tests to fail. It seems to be caused by the terminate hook used by the test. Commenting out this test case makes other tests in the same class pass.

      1. h5963_20140218.patch
        20 kB
        Tsz Wo Nicholas Sze
      2. h5963_20140219.patch
        20 kB
        Tsz Wo Nicholas Sze

        Activity

        Arpit Agarwal created issue -
        Arpit Agarwal made changes -
        Field Original Value New Value
        Description {{TestRollingUpgrade#testSecondaryNameNode}} causes subsequent tests to fail. It seems to be caused by the terminate hook used by the test but I did not spend much time on it. Commenting out this test case makes other tests in the same class pass. {{TestRollingUpgrade#testSecondaryNameNode}} causes subsequent tests to fail. It seems to be caused by the terminate hook used by the test. Commenting out this test case makes other tests in the same class pass.
        Tsz Wo Nicholas Sze made changes -
        Assignee Tsz Wo (Nicholas), SZE [ szetszwo ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Since the edit log upgrade marker is changed to OP_ROLLING_UPGRADE_[START/FINALIZE], we could

        • remove RollingUpgradeStartupOption.STARTED;
        • split FSNamesystem.[start/finalize]RollingUpgrade to two methods for rpc and editlog processing; and
        • combine TestEditLogUpgradeMarker with TestRollingUpgrade.

        h5963_20140218.patch: moves testSecondaryNameNode to the end; fixes QJournalProtocolServerSideTranslatorPB and FileJournalManager; and also has the above changes.

        Show
        Tsz Wo Nicholas Sze added a comment - Since the edit log upgrade marker is changed to OP_ROLLING_UPGRADE_[START/FINALIZE], we could remove RollingUpgradeStartupOption.STARTED; split FSNamesystem.[start/finalize]RollingUpgrade to two methods for rpc and editlog processing; and combine TestEditLogUpgradeMarker with TestRollingUpgrade. h5963_20140218.patch: moves testSecondaryNameNode to the end; fixes QJournalProtocolServerSideTranslatorPB and FileJournalManager; and also has the above changes.
        Tsz Wo Nicholas Sze made changes -
        Attachment h5963_20140218.patch [ 12629718 ]
        Hide
        Arpit Agarwal added a comment -

        Thanks Nicholas. JDK7 could randomize the test case order so perhaps we need to put testSecondaryNameNode in a separate test class?

        Is this failure expected?

        Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.565 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestRollingUpgrade
        testRollback(org.apache.hadoop.hdfs.TestRollingUpgrade)  Time elapsed: 3.386 sec  <<< ERROR!
        java.io.IOException: There appears to be a gap in the edit log.  We expected txid 5, but got txid 8.
                at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)
                at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:203)
                at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:131)
        
        Show
        Arpit Agarwal added a comment - Thanks Nicholas. JDK7 could randomize the test case order so perhaps we need to put testSecondaryNameNode in a separate test class? Is this failure expected? Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.565 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestRollingUpgrade testRollback(org.apache.hadoop.hdfs.TestRollingUpgrade) Time elapsed: 3.386 sec <<< ERROR! java.io.IOException: There appears to be a gap in the edit log. We expected txid 5, but got txid 8. at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:203) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:131)
        Hide
        Tsz Wo Nicholas Sze added a comment -

        For testSecondaryNameNode, let's simply remove it since it is not very useful.

        Let me also fix the bug in rollback.

        Show
        Tsz Wo Nicholas Sze added a comment - For testSecondaryNameNode, let's simply remove it since it is not very useful. Let me also fix the bug in rollback.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > Let me also fix the bug in rollback.

        Talk to Jing Zhao, the rollback bug is quite involved so that we will fix it separately.

        h5963_20140219.patch: removes testSecondaryNameNode() and comments out restartNameNode() in testRollback().

        Show
        Tsz Wo Nicholas Sze added a comment - > Let me also fix the bug in rollback. Talk to Jing Zhao , the rollback bug is quite involved so that we will fix it separately. h5963_20140219.patch: removes testSecondaryNameNode() and comments out restartNameNode() in testRollback().
        Tsz Wo Nicholas Sze made changes -
        Attachment h5963_20140219.patch [ 12629913 ]
        Hide
        Arpit Agarwal added a comment -

        +1 for the patch. I committed it to the feature branch. Thanks Nicholas.

        Show
        Arpit Agarwal added a comment - +1 for the patch. I committed it to the feature branch. Thanks Nicholas.
        Arpit Agarwal made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s HDFS-5535 (Rolling upgrades) [ 12325754 ]
        Resolution Fixed [ 1 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        1d 2m 1 Arpit Agarwal 19/Feb/14 23:56

          People

          • Assignee:
            Tsz Wo Nicholas Sze
            Reporter:
            Arpit Agarwal
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development