Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-617

YARN host affinity in Samza

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Today in Samza we do not guarantee that a container gets deployed on the same machine upon a job upgrade/restart. Hence, the co-located data needs to restored every time a container restarts. Restoring data each time can be expensive, especially for applications that have a large data set.

      If we can enable restarting containers on the same machine, we can re-use available local state.

        Attachments

        1. DESIGN-SAMZA-617-0.md
          14 kB
          Navina Ramesh
        2. DESIGN-SAMZA-617-0.pdf
          211 kB
          Navina Ramesh
        3. DESIGN-SAMZA-617-1.md
          15 kB
          Navina Ramesh
        4. DESIGN-SAMZA-617-1.pdf
          221 kB
          Navina Ramesh
        5. DESIGN-SAMZA-617-2.md
          19 kB
          Navina Ramesh
        6. DESIGN-SAMZA-617-2.pdf
          405 kB
          Navina Ramesh

          Issue Links

          depends upon

          Task - A task that needs to be done. SAMZA-618 Add container-to-host mapping messages to config stream

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. SAMZA-619 Modify Samza AppMaster to enable host-affinity

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. SAMZA-656 Script to periodically clean-up store directories that are persisted outside YARN

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. SAMZA-668 Document YARN host affinity in the website

          • Major - Major loss of function.
          • Resolved
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SAMZA-335 Locality for Samza containers

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-598 Provide a tool to debug state stores

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SAMZA-670 Allow easier access to JMX port

          • Major - Major loss of function.
          • Resolved

            Activity

              People

              • Assignee:
                navina Navina Ramesh
                Reporter:
                navina Navina Ramesh
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated: