Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12182

BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: hdfs
    • Labels:

      Description

      Currently, BlockManager.metaSave method (which is called by "-metasave" dfs CLI command) reports both "under replicated" and "missing" blocks under same metric Metasave: Blocks waiting for reconstruction: as shown on below code snippet:

         synchronized (neededReconstruction) {
            out.println("Metasave: Blocks waiting for reconstruction: "
                + neededReconstruction.size());
            for (Block block : neededReconstruction) {
              dumpBlockMeta(block, out);
            }
          }
      

      neededReconstruction is an instance of LowRedundancyBlocks, which actually wraps 5 priority queues currently. 4 of these queues store different under replicated scenarios, but the 5th one is dedicated for corrupt/missing blocks.

      Thus, metasave report may suggest some corrupt blocks are just under replicated. This can be misleading for admins and operators trying to track block missing/corruption issues, and/or other issues related to BlockManager metrics.

      I would like to propose a patch with trivial changes that would report corrupt blocks separately.

        Attachments

        1. HDFS-12182.001.patch
          2 kB
          Wellington Chevreuil
        2. HDFS-12182.002.patch
          6 kB
          Wellington Chevreuil
        3. HDFS-12182.003.patch
          6 kB
          Wellington Chevreuil
        4. HDFS-12182.004.patch
          6 kB
          Wellington Chevreuil
        5. HDFS-12182-branch-2.001.patch
          7 kB
          Wellington Chevreuil
        6. HDFS-12182-branch-2.002.patch
          7 kB
          Wellington Chevreuil

          Activity

            People

            • Assignee:
              wchevreuil Wellington Chevreuil
              Reporter:
              wchevreuil Wellington Chevreuil
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: