Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-617

YARN host affinity in Samza

          Details

          • Type: Bug
          • Status: Open
          • Priority: Major
          • Resolution: Unresolved
          • Affects Version/s: None
          • Fix Version/s: None
          • Component/s: None
          • Labels:
            None

            Description

            Today in Samza we do not guarantee that a container gets deployed on the same machine upon a job upgrade/restart. Hence, the co-located data needs to restored every time a container restarts. Restoring data each time can be expensive, especially for applications that have a large data set.

            If we can enable restarting containers on the same machine, we can re-use available local state.

              Attachments

              1. DESIGN-SAMZA-617-0.md
                14 kB
                Navina Ramesh
              2. DESIGN-SAMZA-617-0.pdf
                211 kB
                Navina Ramesh
              3. DESIGN-SAMZA-617-1.md
                15 kB
                Navina Ramesh
              4. DESIGN-SAMZA-617-1.pdf
                221 kB
                Navina Ramesh
              5. DESIGN-SAMZA-617-2.md
                19 kB
                Navina Ramesh
              6. DESIGN-SAMZA-617-2.pdf
                405 kB
                Navina Ramesh

                Issue Links

                depends upon

                Task - A task that needs to be done. SAMZA-618 Add container-to-host mapping messages to config stream

                • Major - Major loss of function.
                • Resolved

                Task - A task that needs to be done. SAMZA-619 Modify Samza AppMaster to enable host-affinity

                • Major - Major loss of function.
                • Resolved

                Task - A task that needs to be done. SAMZA-656 Script to periodically clean-up store directories that are persisted outside YARN

                • Major - Major loss of function.
                • Resolved

                Task - A task that needs to be done. SAMZA-668 Document YARN host affinity in the website

                • Major - Major loss of function.
                • Resolved
                duplicates

                Improvement - An improvement or enhancement to an existing feature or task. SAMZA-335 Locality for Samza containers

                • Major - Major loss of function.
                • Resolved
                relates to

                Sub-task - The sub-task of the issue SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

                • Major - Major loss of function.
                • Resolved

                Bug - A problem which impairs or prevents the functions of the product. SAMZA-598 Provide a tool to debug state stores

                • Major - Major loss of function.
                • Resolved

                Improvement - An improvement or enhancement to an existing feature or task. SAMZA-670 Allow easier access to JMX port

                • Major - Major loss of function.
                • Resolved

                  Activity

                    People

                    • Assignee:
                      navina Navina Ramesh
                      Reporter:
                      navina Navina Ramesh
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      11 Start watching this issue

                      Dates

                      • Created:
                        Updated: