Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8150

Make getFileChecksum fail for blocks under construction

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We have seen the cases of validating data copy using checksum then the content of target changing. It turns out the target wasn't closed successfully, so it was still under-construction. One hour later, a lease recovery kicked in and truncated the block.

      Although this can be prevented in many ways, if there is no valid use case for getting file checksum from under-construction blocks, can it be disabled? E.g. Datanode can throw an exception if the replica is not in the finalized state.

        Attachments

        1. HDFS-8150.3.patch
          6 kB
          J.Andreina
        2. HDFS-8150.2.patch
          5 kB
          J.Andreina
        3. HDFS-8150.1.patch
          3 kB
          J.Andreina

          Activity

            People

            • Assignee:
              andreina J.Andreina
              Reporter:
              kihwal Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: