Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-617 YARN host affinity in Samza
  3. SAMZA-817

Add metrics to gauge effectiveness of host-affinity solution

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None

      Description

      We wanted to reduce to container start-up time for jobs that have large change logs. There is no metric to measure this.
      Additionally, Yarn host-affinity makes a best-effort to run a container in the requested host. There is no metric to measure this either.

      Proposal:
      Since task store restoration takes up most of the time during a container start-up time, we can measure the time to restore task stores. It may also help identify data distribution among the task stores. To measure "host-affinity" itself on Yarn, we can add a gauge for the ratio of container requests that is successfully matched to the requested host to the total container requests.

        Attachments

        1. SAMZA-817-1.patch
          11 kB
          Navina Ramesh
        2. SAMZA-817-0.patch
          7 kB
          Navina Ramesh

          Activity

            People

            • Assignee:
              navina Navina Ramesh
              Reporter:
              navina Navina Ramesh
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: