Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-972

Add holistic memory monitoring for SamzaContainer

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: None
    • Labels:
      None

      Description

      This feature introduces physical memory monitoring in SamzaContainer.

      Context:
      Often memory used by the SamzaContainer process includes
      A. JVM Heap memory: This is where all JVM variables live.
      B. Native memory: This memory lives out of the JVM heap and is not visible to the JVM. Examples include used by RocksDb, native libraries that user code depends on etc.

      User jobs could be killed by Yarn if their total memory (A+B) exceeds the configured maximum of yarn.container.memory.mb.

      Currently, while our existing metrics provide visibility into [A] via JMX, we don't have visibility into [B]. (as it's totally external to the JVM).

      1. SAMZA-972-1.patch
        28 kB
        Yi Pan (Data Infrastructure)

        Activity

        Hide
        nickpan47 Yi Pan (Data Infrastructure) added a comment -

        +1. Merged and submitted.

        Show
        nickpan47 Yi Pan (Data Infrastructure) added a comment - +1. Merged and submitted.
        Hide
        nickpan47 Yi Pan (Data Infrastructure) added a comment -

        Rebased against latest master based on the RB

        Show
        nickpan47 Yi Pan (Data Infrastructure) added a comment - Rebased against latest master based on the RB
        Hide
        nickpan47 Yi Pan (Data Infrastructure) added a comment -

        Add the RB link here.

        Show
        nickpan47 Yi Pan (Data Infrastructure) added a comment - Add the RB link here .

          People

          • Assignee:
            jagadish1989@gmail.com Jagadish
            Reporter:
            jagadish1989@gmail.com Jagadish
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development