Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7281

Missing block is marked as corrupted block

    Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The patch improves the reporting around missing blocks and corrupted blocks.
       
      1. A block is missing if and only if all DNs of its expected replicas are dead.
      2. A block is corrupted if and only if all its available replicas are corrupted. So if a block has 3 replicas; one of the DN is dead, the other two replicas are corrupted; it will be marked as corrupted.
      3. A new line is added to fsck output to display the corrupt block size per file.
      4. A new line is added to fsck output to display the number of missing blocks in the summary section.
      Show
      The patch improves the reporting around missing blocks and corrupted blocks.   1. A block is missing if and only if all DNs of its expected replicas are dead. 2. A block is corrupted if and only if all its available replicas are corrupted. So if a block has 3 replicas; one of the DN is dead, the other two replicas are corrupted; it will be marked as corrupted. 3. A new line is added to fsck output to display the corrupt block size per file. 4. A new line is added to fsck output to display the number of missing blocks in the summary section.

      Description

      In the situation where the block lost all its replicas, fsck shows the block is missing as well as corrupted. Perhaps it is better not to mark the block corrupted in this case. The reason it is marked as corrupted is numCorruptNodes == numNodes == 0 in the following code.

      BlockManager
          final boolean isCorrupt = numCorruptNodes == numNodes;
      

      Would like to clarify if it is the intent to mark missing block as corrupted or it is just a bug.

        Attachments

        1. HDFS-7281-6.patch
          9 kB
          Ming Ma
        2. HDFS-7281-5.patch
          9 kB
          Ming Ma
        3. HDFS-7281-4.patch
          8 kB
          Ming Ma
        4. HDFS-7281-3.patch
          5 kB
          Ming Ma
        5. HDFS-7281-2.patch
          5 kB
          Ming Ma
        6. HDFS-7281.patch
          5 kB
          Ming Ma

          Activity

            People

            • Assignee:
              mingma Ming Ma
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: