Hadoop Common
  1. Hadoop Common
  2. HADOOP-5384

DataNodeCluster should not create blocks with generationStamp == 1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.2, 0.20.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In DataNodeCluster.main(..), injected blocks are created with generationStamp == 1, which is a reserved value but not a valid generation stamp. As a consequence, NameNode may die when those blocks are reported.

      1. 5384_20090303.patch
        1 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          5384_20090303.patch: use Block.GRANDFATHER_GENERATION_STAMP instead of 1.

          Show
          Tsz Wo Nicholas Sze added a comment - 5384_20090303.patch: use Block.GRANDFATHER_GENERATION_STAMP instead of 1.
          Hide
          Hairong Kuang added a comment -

          Would it be less error-prone to provide a constructor Block(blk-id, blk-size), which sets the default generation stamp to be Block.GRANDFATHER_GENERATION_STAMP?

          Show
          Hairong Kuang added a comment - Would it be less error-prone to provide a constructor Block(blk-id, blk-size), which sets the default generation stamp to be Block.GRANDFATHER_GENERATION_STAMP?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Yes, it is good to add a new Block constructor. I think we better to refactor the constructors of Block so that the invalid values can be detected. How about we do it in a separated JIRA?

          Show
          Tsz Wo Nicholas Sze added a comment - Yes, it is good to add a new Block constructor. I think we better to refactor the constructors of Block so that the invalid values can be detected. How about we do it in a separated JIRA?
          Hide
          Hairong Kuang added a comment -

          > How about we do it in a separated JIRA?
          Sounds good to me.

          +1. The patch looks good to me.

          Show
          Hairong Kuang added a comment - > How about we do it in a separated JIRA? Sounds good to me. +1. The patch looks good to me.
          Hide
          Hairong Kuang added a comment -

          Marking this as a blocker so that simulated datanode could work with NameNode.

          Show
          Hairong Kuang added a comment - Marking this as a blocker so that simulated datanode could work with NameNode.
          Hide
          Raghu Angadi added a comment -

          NameNode can reject such blocks, but dieing does not seem correct.

          Show
          Raghu Angadi added a comment - NameNode can reject such blocks, but dieing does not seem correct.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > NameNode can reject such blocks, but dieing does not seem correct.
          Agree. This is going to be fixed in HADOOP-5399.

          Tested the patch manually. It fixes the problem with DataNodeCluster. Since the patch is simple, I won't add new unit tests.

          Show
          Tsz Wo Nicholas Sze added a comment - > NameNode can reject such blocks, but dieing does not seem correct. Agree. This is going to be fixed in HADOOP-5399 . Tested the patch manually. It fixes the problem with DataNodeCluster. Since the patch is simple, I won't add new unit tests.
          Hide
          Tsz Wo Nicholas Sze added a comment -
               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          Show
          Tsz Wo Nicholas Sze added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I committed this to 0.19 and above.

          Show
          Tsz Wo Nicholas Sze added a comment - I committed this to 0.19 and above.
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #778 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/ )

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development