Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6125

The application attempt's diagnostic message should have a maximum size

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: resourcemanager
    • Labels:
      None

      Description

      We've found through experience that the diagnostic message can grow unbounded. I've seen attempts that have diagnostic messages over 1MB. Since the message is stored in the state store, it's a bad idea to allow the message to grow unbounded. Instead, there should be a property that sets a maximum size on the message.

      I suspect that some of the ZK state store issues we've seen in the past were due to the size of the diagnostic messages and not to the size of the classpath, as is the current prevailing opinion.

      An open question is how best to prune the message once it grows too large. Should we

      1. truncate the tail,
      2. truncate the head,
      3. truncate the middle,
      4. add another property to make the behavior selectable, or
      5. none of the above?

        Attachments

        1. YARN-6125.000.patch
          18 kB
          Andras Piros
        2. YARN-6125.001.patch
          17 kB
          Andras Piros
        3. YARN-6125.002.patch
          20 kB
          Andras Piros
        4. YARN-6125.003.patch
          20 kB
          Andras Piros
        5. YARN-6125.004.patch
          28 kB
          Andras Piros
        6. YARN-6125.005.patch
          17 kB
          Andras Piros
        7. YARN-6125.006.patch
          17 kB
          Andras Piros
        8. YARN-6125.007.patch
          17 kB
          Andras Piros
        9. YARN-6125.008.patch
          17 kB
          Andras Piros
        10. YARN-6125.009.patch
          23 kB
          Andras Piros

          Activity

            People

            • Assignee:
              andras.piros Andras Piros
              Reporter:
              templedf Daniel Templeton
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: