Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2282

Semi-harmless race between block reports and block invalidation

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not A Problem
    • Affects Version/s: 0.20.203.0, 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:

      Description

      In the 0.20.203 codebase, block reports are not synchronized in any way against mutations to the actual file structure on disk. If a file is removed from a directory while the block report is scanning that directory, it will be mistakenly reported as existing with a length of 0, since File.length() on a non-existent file returns 0.

      This results in an error being logged on the DataNode when the NN sends it a second block deletion request for the already-deleted block. I believe it to be harmless, but the error message can concern users.

      This was fixed in the 0.20 code line in HDFS-2379. This jira remains open to track the port to 0.24.

      1. hdfs-2282-20.txt
        0.8 kB
        Todd Lipcon
      2. hdfs-2282-20.txt
        0.9 kB
        Todd Lipcon

        Issue Links

          Activity

          Allen Wittenauer made changes -
          Affects Version/s 2.0.0-alpha [ 12320353 ]
          Affects Version/s 0.24.0 [ 12317653 ]
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Not A Problem [ 8 ]
          Matt Foley made changes -
          Link This issue is related to HDFS-2379 [ HDFS-2379 ]
          Matt Foley made changes -
          Fix Version/s 1.1.0 [ 12317959 ]
          Affects Version/s 0.24.0 [ 12317653 ]
          Description In the 0.20 codebase, block reports are not synchronized in any way against mutations to the actual file structure on disk. If a file is removed from a directory while the block report is scanning that directory, it will be mistakenly reported as existing with a length of 0, since File.length() on a non-existent file returns 0.

          This results in an error being logged on the DataNode when the NN sends it a second block deletion request for the already-deleted block. I believe it to be harmless, but the error message can concern users.
          In the 0.20.203 codebase, block reports are not synchronized in any way against mutations to the actual file structure on disk. If a file is removed from a directory while the block report is scanning that directory, it will be mistakenly reported as existing with a length of 0, since File.length() on a non-existent file returns 0.

          This results in an error being logged on the DataNode when the NN sends it a second block deletion request for the already-deleted block. I believe it to be harmless, but the error message can concern users.

          This was fixed in the 0.20 code line in HDFS-2379. This jira remains open to track the port to 0.24.
          Eli Collins made changes -
          Fix Version/s 0.20.206.0 [ 12317959 ]
          Eli Collins made changes -
          Target Version/s 0.24.0 [ 12317653 ]
          Todd Lipcon made changes -
          Attachment hdfs-2282-20.txt [ 12491438 ]
          Todd Lipcon made changes -
          Field Original Value New Value
          Attachment hdfs-2282-20.txt [ 12491437 ]
          Todd Lipcon created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development