Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8965

Harden edit log reading code against out of memory errors

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0-alpha
    • 2.8.0, 3.0.0-alpha1
    • None
    • None

    Description

      We should harden the edit log reading code against out of memory errors. Now that each op has a length prefix and a checksum, we can validate the checksum before trying to load the Op data. This should avoid out of memory errors when trying to load garbage data as Op data.

      Attachments

        1. HDFS-8965.001.patch
          18 kB
          Colin McCabe
        2. HDFS-8965.002.patch
          18 kB
          Colin McCabe
        3. HDFS-8965.003.patch
          19 kB
          Colin McCabe
        4. HDFS-8965.004.patch
          24 kB
          Colin McCabe
        5. HDFS-8965.005.patch
          25 kB
          Colin McCabe
        6. HDFS-8965.006.patch
          25 kB
          Colin McCabe
        7. HDFS-8965.007.patch
          25 kB
          Colin McCabe

        Issue Links

          relates to

          Test - A new unit, integration or system test. HDFS-5995 TestFSEditLogLoader#testValidateEditLogWithCorruptBody gets OutOfMemoryError and dumps heap.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              cmccabe Colin McCabe
              cmccabe Colin McCabe
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: