Hadoop Common
  1. Hadoop Common
  2. HADOOP-5554

DataNodeCluster should create blocks with the same generation stamp as the blocks created in CreateEditsLog

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.2
    • Fix Version/s: 0.19.2, 0.20.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-5384 makes DataNodeCluster to create blocks with generation stamp Block#GRANDFATHER_GENERATION_STAMP(0) so simuated datanodes do not crash NameNode any more. But there is still a problem. CreateEditLogs creates blocks with generation stamp GenerationStamp#FIRST_VALID_STAMP (1000). Because of the generation stamp mismatch, all injected blocks are marked as invalid when NameNode processes block reports.

      1. blockGS-0.19.patch
        2 kB
        Hairong Kuang
      2. blockGS.patch
        2 kB
        Hairong Kuang

        Activity

        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Nigel Daley made changes -
        Fix Version/s 0.21.0 [ 12313563 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #789 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/789/)
        . DataNodeCluster and CreateEditsLog should create blocks with the same generation stamp value. (hairong via szetszwo)

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #789 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/789/ ) . DataNodeCluster and CreateEditsLog should create blocks with the same generation stamp value. (hairong via szetszwo)
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this to 0.19 and above. Thanks, Hairong!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this to 0.19 and above. Thanks, Hairong!
        Hairong Kuang made changes -
        Attachment blockGS-0.19.patch [ 12403471 ]
        Hide
        Hairong Kuang added a comment -

        This is a patch for 0.19.

        Show
        Hairong Kuang added a comment - This is a patch for 0.19.
        Hide
        Hairong Kuang added a comment -

        I have tested the patch manually.

        Show
        Hairong Kuang added a comment - I have tested the patch manually.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        This patch changes tools for manual tests. I think we don't have junit for these test tools. So, I suggest to do some manual tests instead of waiting for Hudson.

        Show
        Tsz Wo Nicholas Sze added a comment - This patch changes tools for manual tests. I think we don't have junit for these test tools. So, I suggest to do some manual tests instead of waiting for Hudson.
        Hairong Kuang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.19.2 [ 12313650 ]
        Affects Version/s 0.19.1 [ 12313473 ]
        Tsz Wo Nicholas Sze made changes -
        Hadoop Flags [Reviewed]
        Component/s test [ 12311440 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1

        Show
        Tsz Wo Nicholas Sze added a comment - +1
        Hairong Kuang made changes -
        Field Original Value New Value
        Attachment blockGS.patch [ 12403446 ]
        Hide
        Hairong Kuang added a comment -

        This patch makes both CreateEditsLog and DataNodeCluster to create blocks with generation stamp GenerationStamp#FIRST_VALID_STAMP.

        Show
        Hairong Kuang added a comment - This patch makes both CreateEditsLog and DataNodeCluster to create blocks with generation stamp GenerationStamp#FIRST_VALID_STAMP.
        Hairong Kuang created issue -

          People

          • Assignee:
            Hairong Kuang
            Reporter:
            Hairong Kuang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development