Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3049

During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.23.0
    • 2.0.3-alpha
    • namenode
    • None

    Description

      During the NameNode startup process, we load an image, and then apply edit logs to it until we believe that we have all the latest changes. Unfortunately, if there is an I/O error while reading any of these files, in most cases, we simply abort the startup process. We should try harder to locate a readable edit log and/or image file.

      There are three main use cases for this feature:
      1. If the operating system does not honor fsync (usually due to a misconfiguration), a file may end up in an inconsistent state.
      2. In certain older releases where we did not use fallocate() or similar to pre-reserve blocks, a disk full condition may cause a truncated log in one edit directory.
      3. There may be a bug in HDFS which results in some of the data directories receiving corrupt data, but not all. This is the least likely use case.

      Proposed changes to normal NN startup

      • We should try a different FSImage if we can't load the first one we try.
      • We should examine other FSEditLogs if we can't load the first one(s) we try.
      • We should fail if we can't find EditLogs that would bring us up to what we believe is the latest transaction ID.

      Proposed changes to recovery mode NN startup:
      we should list out all the available storage directories and allow the operator to select which one he wants to use.
      Something like this:

      Multiple storage directories found.
      1. /foo/bar
          edits__curent__XYZ          size:213421345       md5:2345345
          image                                  size:213421345       md5:2345345
      2. /foo/baz
          edits__curent__XYZ          size:213421345       md5:2345345345
          image                                  size:213421345       md5:2345345
      Which one would you like to use? (1/2)
      

      As usual in recovery mode, we want to be flexible about error handling. In this case, this means that we should NOT fail if we can't find EditLogs that would bring us up to what we believe is the latest transaction ID.

      Not addressed by this feature
      This feature will not address the case where an attempt to access the NameNode name directory or directories hangs because of an I/O error. This may happen, for example, when trying to load an image from a hard-mounted NFS directory, when the NFS server has gone away. Just as now, the operator will have to notice this problem and take steps to correct it.

      Attachments

        1. HDFS-3049.001.patch
          22 kB
          Colin McCabe
        2. HDFS-3049.002.patch
          23 kB
          Colin McCabe
        3. HDFS-3049.003.patch
          23 kB
          Colin McCabe
        4. HDFS-3049.005.against3335.patch
          53 kB
          Colin McCabe
        5. HDFS-3049.006.against3335.patch
          69 kB
          Colin McCabe
        6. HDFS-3049.007.against3335.patch
          72 kB
          Colin McCabe
        7. HDFS-3049.010.patch
          79 kB
          Colin McCabe
        8. HDFS-3049.011.patch
          79 kB
          Colin McCabe
        9. HDFS-3049.012.patch
          88 kB
          Colin McCabe
        10. HDFS-3049.013.patch
          88 kB
          Colin McCabe
        11. HDFS-3049.015.patch
          114 kB
          Colin McCabe
        12. HDFS-3049.017.patch
          117 kB
          Colin McCabe
        13. HDFS-3049.018.patch
          118 kB
          Colin McCabe
        14. HDFS-3049.021.patch
          87 kB
          Colin McCabe
        15. HDFS-3049.023.patch
          37 kB
          Colin McCabe
        16. HDFS-3049.025.patch
          39 kB
          Colin McCabe
        17. HDFS-3049.026.patch
          39 kB
          Colin McCabe
        18. HDFS-3049.027.patch
          41 kB
          Colin McCabe
        19. HDFS-3049.028.patch
          40 kB
          Colin McCabe
        20. HDFS-3049.028.patch
          40 kB
          Colin McCabe
        21. HDFS-3049.028.patch
          40 kB
          Colin McCabe
        22. hdfs-3049-branch-2.txt
          28 kB
          Todd Lipcon

        Issue Links

          Activity

            People

              cmccabe Colin McCabe
              cmccabe Colin McCabe
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: