Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13818

Extend OIV to detect FSImage corruption

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: hdfs
    • Labels:
      None

      Description

      A follow-up Jira for HDFS-13031: an improvement of the OIV is suggested for detecting corruptions like HDFS-13101 in an offline way.

      The reasoning is the following. Apart from a NN startup throwing the error, there is nothing in the customer's hand that could reassure him/her that the FSImages is good or corrupted.

      Although real full checking of the FSImage is only possible by the NN, for stack traces associated with the observed corruption cases the solution of putting up a tertiary NN is a little bit of overkill. The OIV would be a handy choice, already having functionality like loading the fsimage and constructing the folder structure, we just have to add the option of detecting the null INodes. For e.g. the Delimited OIV processor can already use in disk MetadataMap, which reduces memory consumption. Also there may be a window for parallelizing: iterating through INodes for e.g. could be done distributed, increasing efficiency, and we wouldn't need a high mem-high CPU setup for just checking the FSImage.

      The suggestion is to add a --detectCorruption option to the OIV which would check the FSImage for consistency.

        Attachments

        1. HDFS-13818.009.patch
          62 kB
          Adam Antal
        2. HDFS-13818.008.patch
          62 kB
          Adam Antal
        3. HDFS-13818.007.patch
          62 kB
          Adam Antal
        4. HDFS-13818.006.patch
          64 kB
          Adam Antal
        5. HDFS-13818.005.patch
          63 kB
          Adam Antal
        6. HDFS-13818.004.patch
          63 kB
          Adam Antal
        7. HDFS-13818.003.patch
          54 kB
          Adam Antal
        8. OIV_CorruptionDetector_processor.002.pdf
          38 kB
          Adam Antal
        9. HDFS-13818.003.patch
          54 kB
          Adam Antal
        10. HDFS-13818.002.patch
          52 kB
          Adam Antal
        11. OIV_CorruptionDetector_processor.001.pdf
          38 kB
          Adam Antal
        12. HDFS-13818.001.patch
          20 kB
          Adam Antal

          Issue Links

            Activity

              People

              • Assignee:
                adam.antal Adam Antal
                Reporter:
                adam.antal Adam Antal
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: