Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8965

Harden edit log reading code against out of memory errors

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      We should harden the edit log reading code against out of memory errors. Now that each op has a length prefix and a checksum, we can validate the checksum before trying to load the Op data. This should avoid out of memory errors when trying to load garbage data as Op data.

        Attachments

        1. HDFS-8965.001.patch
          18 kB
          Colin McCabe
        2. HDFS-8965.002.patch
          18 kB
          Colin McCabe
        3. HDFS-8965.003.patch
          19 kB
          Colin McCabe
        4. HDFS-8965.004.patch
          24 kB
          Colin McCabe
        5. HDFS-8965.005.patch
          25 kB
          Colin McCabe
        6. HDFS-8965.006.patch
          25 kB
          Colin McCabe
        7. HDFS-8965.007.patch
          25 kB
          Colin McCabe

          Issue Links

          relates to

          Test - A new unit, integration or system test. HDFS-5995 TestFSEditLogLoader#testValidateEditLogWithCorruptBody gets OutOfMemoryError and dumps heap.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

            Activity

              People

              • Assignee:
                cmccabe Colin McCabe
                Reporter:
                cmccabe Colin McCabe
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: