Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3134

Harden edit log loader against malformed or malicious input

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, the edit log loader does not handle bad or malicious input sensibly.

      We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated.

      It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations.

      The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input.

      1. HDFS-3134.001.patch
        5 kB
        Colin Patrick McCabe
      2. HDFS-3134.002.patch
        8 kB
        Colin Patrick McCabe
      3. HDFS-3134.003.patch
        3 kB
        Colin Patrick McCabe
      4. HDFS-3134.004.patch
        7 kB
        Colin Patrick McCabe
      5. HDFS-3134.005.patch
        4 kB
        Colin Patrick McCabe
      6. HDFS-3134.006.patch
        7 kB
        Colin Patrick McCabe
      7. HDFS-3134.007.patch
        4 kB
        Colin Patrick McCabe
      8. HDFS-3134.009.patch
        7 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Gavin made changes -
          Link This issue depends upon HADOOP-8275 [ HADOOP-8275 ]
          Gavin made changes -
          Link This issue depends on HADOOP-8275 [ HADOOP-8275 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322472 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Todd Lipcon made changes -
          Fix Version/s 2.0.1 [ 12321440 ]
          Fix Version/s 2.0.0 [ 12320353 ]
          Colin Patrick McCabe made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Eli Collins made changes -
          Summary harden edit log loader against malformed or malicious input Harden edit log loader against malformed or malicious input
          Hadoop Flags Reviewed [ 10343 ]
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          Fix Version/s 2.0.0 [ 12320353 ]
          Target Version/s 2.0.0 [ 12320353 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3134.009.patch [ 12526231 ]
          Colin Patrick McCabe made changes -
          Link This issue is blocked by HADOOP-8361 [ HADOOP-8361 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HDFS-3346 [ HDFS-3346 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3134.007.patch [ 12525248 ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3134.006.patch [ 12525230 ]
          Colin Patrick McCabe made changes -
          Fix Version/s 2.0.0 [ 12320353 ]
          Target Version/s 2.0.0 [ 12320353 ]
          Colin Patrick McCabe made changes -
          Fix Version/s 2.0.0 [ 12320353 ]
          Affects Version/s 0.23.0 [ 12315571 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3134.005.patch [ 12522629 ]
          Colin Patrick McCabe made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Colin Patrick McCabe made changes -
          Link This issue depends on HADOOP-8275 [ HADOOP-8275 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3134.004.patch [ 12522464 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3134.003.patch [ 12522365 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3134.002.patch [ 12521726 ]
          Tsz Wo Nicholas Sze made changes -
          Component/s name-node [ 12312926 ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3134.001.patch [ 12521708 ]
          Colin Patrick McCabe made changes -
          Field Original Value New Value
          Link This issue relates to HDFS-3004 [ HDFS-3004 ]
          Colin Patrick McCabe created issue -

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development