Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20-append, 0.20.205.0
    • Fix Version/s: 0.20.205.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      found that when the DN uses client verification of a block that is open for writing, it will add it to the DataBlockScanner prematurely.

      1. HDFS-1122-20-security.2.patch
        6 kB
        Jitendra Nath Pandey
      2. hdfs-1122-for-0.20.txt
        7 kB
        sam rash

        Activity

        Hide
        Eli Collins added a comment -

        Marking for 2.x since trunk still has this issue.

        Show
        Eli Collins added a comment - Marking for 2.x since trunk still has this issue.
        Hide
        Matt Foley added a comment -

        If no one chooses to port this patch to 0.20-append, perhaps we should go ahead and close this jira.

        Show
        Matt Foley added a comment - If no one chooses to port this patch to 0.20-append, perhaps we should go ahead and close this jira.
        Hide
        Jitendra Nath Pandey added a comment -

        Committed to branch-0.20-security. Thanks to Sam.

        Show
        Jitendra Nath Pandey added a comment - Committed to branch-0.20-security. Thanks to Sam.
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch.

        Show
        Suresh Srinivas added a comment - +1 for the patch.
        Hide
        Jitendra Nath Pandey added a comment -

        The attached patch is Sam's original patch with following changes
        1) Rebased against 20-security.
        2) Changed testcase to junit4.
        3) Addressed Todd's comments.

        Show
        Jitendra Nath Pandey added a comment - The attached patch is Sam's original patch with following changes 1) Rebased against 20-security. 2) Changed testcase to junit4. 3) Addressed Todd's comments.
        Hide
        sam rash added a comment -

        it looks it does exist--has the same problem that if a BlockSender sends all the data for an open block, it gets added to the block scanner.

        oddly, in retrospect, this doesn't produce an error since the DataBlockScanner itself is just another concurrent reader
        (it does produce the warnings of adding a block more than once)

        Show
        sam rash added a comment - it looks it does exist--has the same problem that if a BlockSender sends all the data for an open block, it gets added to the block scanner. oddly, in retrospect, this doesn't produce an error since the DataBlockScanner itself is just another concurrent reader (it does produce the warnings of adding a block more than once)
        Hide
        Todd Lipcon added a comment -

        Hey Sam, do you think this bug exists on trunk's implementation?

        Show
        Todd Lipcon added a comment - Hey Sam, do you think this bug exists on trunk's implementation?
        Hide
        Nicolas Spiegelberg added a comment -

        This should be pulled into the 0.20-append branch.

        Show
        Nicolas Spiegelberg added a comment - This should be pulled into the 0.20-append branch.
        Hide
        Todd Lipcon added a comment -

        20 patch looks good to me. Small nits:

        • assertTrue("/ should be a directory"...) check at the top of testPrematureDataBlockScannerAdd seems out of place.
        • The javadoc at the top of TestDataBlockScanner is misplaced
        • You may also want to add a bit of javadoc to the test case itself that explains what it's checking for
        Show
        Todd Lipcon added a comment - 20 patch looks good to me. Small nits: assertTrue("/ should be a directory"...) check at the top of testPrematureDataBlockScannerAdd seems out of place. The javadoc at the top of TestDataBlockScanner is misplaced You may also want to add a bit of javadoc to the test case itself that explains what it's checking for
        Hide
        sam rash added a comment -

        patch that works on 0.20

        Show
        sam rash added a comment - patch that works on 0.20
        Hide
        sam rash added a comment -

        This results in these log messages:

        2010-04-21 13:06:30,951 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_6423942125821562308_117574
        2010-04-21 12:59:47,054 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_-1890060265487773738_117566
        2010-04-21 12:56:26,831 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_-8254097362836825914_117561
        2010-04-21 12:53:03,386 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_8946894423251690136_117557
        2010-04-21 12:49:43,148 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_-5467425469535997066_117553
        2010-04-21 12:46:22,613 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_-3020378094937646676_117549

        and its possible the block scanner could mark the blocks corrupt since they are being written to. I have a test + 0.20 patch I will upload shortly (crux of patch is that client verifications can only update the DataBlockScanner, not add new blocks).

        Show
        sam rash added a comment - This results in these log messages: 2010-04-21 13:06:30,951 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_6423942125821562308_117574 2010-04-21 12:59:47,054 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_-1890060265487773738_117566 2010-04-21 12:56:26,831 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_-8254097362836825914_117561 2010-04-21 12:53:03,386 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_8946894423251690136_117557 2010-04-21 12:49:43,148 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_-5467425469535997066_117553 2010-04-21 12:46:22,613 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Adding an already existing block blk_-3020378094937646676_117549 and its possible the block scanner could mark the blocks corrupt since they are being written to. I have a test + 0.20 patch I will upload shortly (crux of patch is that client verifications can only update the DataBlockScanner, not add new blocks).

          People

          • Assignee:
            sam rash
            Reporter:
            sam rash
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development