Hadoop Common
  1. Hadoop Common
  2. HADOOP-4257

TestLeaseRecovery2.testBlockSynchronization failing.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.3
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Found this while running HADOOP-4173 through Hudson.

      HadoopQA output:

      org.apache.hadoop.hdfs.TestLeaseRecovery2.testBlockSynchronization
      Failing for the past 2 builds (Since Failed#3352 )
      Took 0 seconds.
      Error Message
      
      Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
      
      Stacktrace
      
      junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
      

      See http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3353/testReport/org.apache.hadoop.hdfs/TestLeaseRecovery2/testBlockSynchronization/ and http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3352/testReport/org.apache.hadoop.hdfs/TestLeaseRecovery2/testBlockSynchronization/

      1. 4257_20080929.patch
        4 kB
        Tsz Wo Nicholas Sze
      2. 4257_20080929b.patch
        4 kB
        Tsz Wo Nicholas Sze
      3. 4257_20080930.patch
        8 kB
        Tsz Wo Nicholas Sze
      4. 4257_20080930b.patch
        9 kB
        Tsz Wo Nicholas Sze
      5. 4257_20081001.patch
        11 kB
        Tsz Wo Nicholas Sze
      6. 4257_20081001b.patch
        11 kB
        Tsz Wo Nicholas Sze
      7. 4257_20081002_0.18.patch
        14 kB
        Tsz Wo Nicholas Sze
      8. 4257_20081002.patch
        11 kB
        Tsz Wo Nicholas Sze
      9. runtest
        0.4 kB
        Tsz Wo Nicholas Sze
      10. testClientTriggeredLeaseRecovery.htm
        490 kB
        Raghu Angadi
      11. TestLeaseRecovery2-1.txt
        391 kB
        Raghu Angadi

        Issue Links

          Activity

          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #677 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/677/ )
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Tested the patch locally. Everything works fine. I have committed this to 0.18.

          Show
          Tsz Wo Nicholas Sze added a comment - Tested the patch locally. Everything works fine. I have committed this to 0.18.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          4257_20081002_0.18.patch: the issue should also be committed to 0.18.

          Show
          Tsz Wo Nicholas Sze added a comment - 4257_20081002_0.18.patch: the issue should also be committed to 0.18.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          runtest: this is a simple script I used to run a single test repeatedly. I think someone else may find it useful.

          Show
          Tsz Wo Nicholas Sze added a comment - runtest: this is a simple script I used to run a single test repeatedly. I think someone else may find it useful.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #622 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/622/)
          . The DFS client should pick only one datanode as the candidate
          to initiate lease recovery. (Tsz Wo (Nicholas), SZE via cdouglas)

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #622 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/622/ ) . The DFS client should pick only one datanode as the candidate to initiate lease recovery. (Tsz Wo (Nicholas), SZE via cdouglas)
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Nicholas!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Nicholas!
          Hide
          dhruba borthakur added a comment -

          Code looks good. +1.

          Show
          dhruba borthakur added a comment - Code looks good. +1.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12391355/4257_20081002.patch
          against trunk revision 700997.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3424/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3424/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3424/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3424/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12391355/4257_20081002.patch against trunk revision 700997. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3424/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3424/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3424/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3424/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          4257_20081002.patch: fixed a bug in TestLeaseRecovery2: sleep too short + heartbeat too long => time out

          Passed the tests locally with same random number generator seed used in build #3421.

          Show
          Tsz Wo Nicholas Sze added a comment - 4257_20081002.patch: fixed a bug in TestLeaseRecovery2: sleep too short + heartbeat too long => time out Passed the tests locally with same random number generator seed used in build #3421.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12391340/4257_20081001b.patch
          against trunk revision 700997.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3421/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3421/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3421/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3421/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12391340/4257_20081001b.patch against trunk revision 700997. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3421/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3421/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3421/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3421/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          4257_20081001b.patch: changed some test log messages

          Show
          Tsz Wo Nicholas Sze added a comment - 4257_20081001b.patch: changed some test log messages
          Hide
          Tsz Wo Nicholas Sze added a comment -

          4257_20081001.patch: fixed another bug in BlockSender.sendChunks(...).

          Show
          Tsz Wo Nicholas Sze added a comment - 4257_20081001.patch: fixed another bug in BlockSender.sendChunks(...).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12391253/4257_20080930b.patch
          against trunk revision 700887.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3416/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3416/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3416/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3416/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12391253/4257_20080930b.patch against trunk revision 700887. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3416/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3416/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3416/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3416/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          4257_20080930b.patch: it passed TestLeaseRecovery2 around 20 times in my machine.

          Show
          Tsz Wo Nicholas Sze added a comment - 4257_20080930b.patch: it passed TestLeaseRecovery2 around 20 times in my machine.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          4257_20080930.patch: cleaned up TestLeaseRecovery2

          Show
          Tsz Wo Nicholas Sze added a comment - 4257_20080930.patch: cleaned up TestLeaseRecovery2
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12391175/4257_20080929b.patch
          against trunk revision 700322.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3403/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3403/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3403/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3403/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12391175/4257_20080929b.patch against trunk revision 700322. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3403/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3403/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3403/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3403/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -
               [exec] +1 overall.  
          
               [exec]     +1 @author.  The patch does not contain any @author tags.
          
               [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
          
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
          
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
          
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          

          Tested locally: only TestReduceFetch failed but it is not related to this issue. See HADOOP-4302. Submitting ...

          Show
          Tsz Wo Nicholas Sze added a comment - [exec] +1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. Tested locally: only TestReduceFetch failed but it is not related to this issue. See HADOOP-4302 . Submitting ...
          Hide
          dhruba borthakur added a comment -

          Great catch(es).

          +1. Code looks good.

          Show
          dhruba borthakur added a comment - Great catch(es). +1. Code looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          4257_20080929b.patch: forgot to reset the RANDOM_NUMBER_GENERATOR_SEED in my last patch.

          Show
          Tsz Wo Nicholas Sze added a comment - 4257_20080929b.patch: forgot to reset the RANDOM_NUMBER_GENERATOR_SEED in my last patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          4257_20080929.patch: fixed 2 bugs

          • Namenode should assign only one datanode as primary but not multiple datanodes
          • In FSDataset.updateBlock(...), renaming block meta file may fail because there may be multiple updateBlock(...) calls.
          Show
          Tsz Wo Nicholas Sze added a comment - 4257_20080929.patch: fixed 2 bugs Namenode should assign only one datanode as primary but not multiple datanodes In FSDataset.updateBlock(...), renaming block meta file may fail because there may be multiple updateBlock(...) calls.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          This seems related to HADOOP-3614.

          Show
          Tsz Wo Nicholas Sze added a comment - This seems related to HADOOP-3614 .
          Hide
          Raghu Angadi added a comment -

          > Edited comment for clarity.. not hijacked.
          I wanted to say "Edited title ..."

          Show
          Raghu Angadi added a comment - > Edited comment for clarity.. not hijacked. I wanted to say "Edited title ..."
          Hide
          Raghu Angadi added a comment -

          Edited comment for clarity.. not hijacked.

          Show
          Raghu Angadi added a comment - Edited comment for clarity.. not hijacked.
          Hide
          Raghu Angadi added a comment -

          Couple of lease recovery related failures :

          • TestLeaseRecovery2-1.txt : from nightly failure.
          • testClientTriggeredLeaseRecovery.htm : core-test failure from HADOOP-4173.
          Show
          Raghu Angadi added a comment - Couple of lease recovery related failures : TestLeaseRecovery2-1.txt : from nightly failure. testClientTriggeredLeaseRecovery.htm : core-test failure from HADOOP-4173 .
          Hide
          Raghu Angadi added a comment -

          There are quite a few failures for TestLeaseRecovery2. I will attach couple of examples.

          Show
          Raghu Angadi added a comment - There are quite a few failures for TestLeaseRecovery2. I will attach couple of examples.

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development