Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-972

Add holistic memory monitoring for SamzaContainer

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: None
    • Labels:
      None

      Description

      This feature introduces physical memory monitoring in SamzaContainer.

      Context:
      Often memory used by the SamzaContainer process includes
      A. JVM Heap memory: This is where all JVM variables live.
      B. Native memory: This memory lives out of the JVM heap and is not visible to the JVM. Examples include used by RocksDb, native libraries that user code depends on etc.

      User jobs could be killed by Yarn if their total memory (A+B) exceeds the configured maximum of yarn.container.memory.mb.

      Currently, while our existing metrics provide visibility into [A] via JMX, we don't have visibility into [B]. (as it's totally external to the JVM).

        Attachments

        1. SAMZA-972-1.patch
          28 kB
          Yi Pan (Data Infrastructure)

          Activity

            People

            • Assignee:
              jagadish1989@gmail.com Jagadish
              Reporter:
              jagadish1989@gmail.com Jagadish
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: