Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-617

YARN host affinity in Samza

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Today in Samza we do not guarantee that a container gets deployed on the same machine upon a job upgrade/restart. Hence, the co-located data needs to restored every time a container restarts. Restoring data each time can be expensive, especially for applications that have a large data set.

      If we can enable restarting containers on the same machine, we can re-use available local state.

      Attachments

        1. DESIGN-SAMZA-617-0.md
          14 kB
          Navina Ramesh
        2. DESIGN-SAMZA-617-0.pdf
          211 kB
          Navina Ramesh
        3. DESIGN-SAMZA-617-1.md
          15 kB
          Navina Ramesh
        4. DESIGN-SAMZA-617-1.pdf
          221 kB
          Navina Ramesh
        5. DESIGN-SAMZA-617-2.md
          19 kB
          Navina Ramesh
        6. DESIGN-SAMZA-617-2.pdf
          405 kB
          Navina Ramesh

        Issue Links

          depends upon

          Task - A task that needs to be done. SAMZA-618 Add container-to-host mapping messages to config stream

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. SAMZA-619 Modify Samza AppMaster to enable host-affinity

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. SAMZA-656 Script to periodically clean-up store directories that are persisted outside YARN

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. SAMZA-668 Document YARN host affinity in the website

          • Major - Major loss of function.
          • Resolved
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SAMZA-335 Locality for Samza containers

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-598 Provide a tool to debug state stores

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SAMZA-670 Allow easier access to JMX port

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              navina Navina Ramesh
              navina Navina Ramesh
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated: