Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2282

Semi-harmless race between block reports and block invalidation

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not A Problem
    • Affects Version/s: 0.20.203.0, 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:

      Description

      In the 0.20.203 codebase, block reports are not synchronized in any way against mutations to the actual file structure on disk. If a file is removed from a directory while the block report is scanning that directory, it will be mistakenly reported as existing with a length of 0, since File.length() on a non-existent file returns 0.

      This results in an error being logged on the DataNode when the NN sends it a second block deletion request for the already-deleted block. I believe it to be harmless, but the error message can concern users.

      This was fixed in the 0.20 code line in HDFS-2379. This jira remains open to track the port to 0.24.

      1. hdfs-2282-20.txt
        0.8 kB
        Todd Lipcon
      2. hdfs-2282-20.txt
        0.9 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Here's a patch for 20. Before resolving we should write a unit test and also double check that this bug doesn't show up in trunk.

          Show
          Todd Lipcon added a comment - Here's a patch for 20. Before resolving we should write a unit test and also double check that this bug doesn't show up in trunk.
          Hide
          Todd Lipcon added a comment -

          oops, correcting slight error in previous upload

          Show
          Todd Lipcon added a comment - oops, correcting slight error in previous upload
          Hide
          Eli Collins added a comment -

          Looks like this was fixed in HDFS-2379.

          Show
          Eli Collins added a comment - Looks like this was fixed in HDFS-2379 .
          Hide
          Todd Lipcon added a comment -

          Yea, fixed it in 20 with HDFS-2379, but might still need a small fix in trunk.

          Show
          Todd Lipcon added a comment - Yea, fixed it in 20 with HDFS-2379 , but might still need a small fix in trunk.
          Hide
          Eli Collins added a comment -

          Cool.. using our new fancy fix/target scheme to indicate this was fixed in 206 but needs a fix in 204.

          Show
          Eli Collins added a comment - Cool.. using our new fancy fix/target scheme to indicate this was fixed in 206 but needs a fix in 204.
          Hide
          Matt Foley added a comment -

          Since this patch was not applied (from this jira) to 1.0, changing "fix version" usage to clean up release notes for 1.1.0.

          Show
          Matt Foley added a comment - Since this patch was not applied (from this jira) to 1.0, changing "fix version" usage to clean up release notes for 1.1.0.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I believe this is "Not A Problem" anymore. Please feel free to reopen it if it is not the case.

          Show
          Tsz Wo Nicholas Sze added a comment - I believe this is "Not A Problem" anymore. Please feel free to reopen it if it is not the case.

            People

            • Assignee:
              Unassigned
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development