Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13378

Common features between YARN and HDFS Router-based federation

        Details

        • Type: New Feature
        • Status: Open
        • Priority: Major
        • Resolution: Unresolved
        • Affects Version/s: None
        • Fix Version/s: None
        • Component/s: None
        • Labels:
          None

          Description

          HDFS-10467 uses a similar architecture to the one proposed in YARN-2915. This JIRA tries to identify what is common between these two efforts and try to build a common framework.

            Issue Links

            is related to

            New Feature - A new feature of the product, which has yet to be developed. YARN-2915 Enable YARN RM scale out via federation using multiple RM's

            • Major - Major loss of function.
            • Open

            New Feature - A new feature of the product, which has yet to be developed. HDFS-10467 Router-based HDFS federation

            • Major - Major loss of function.
            • Reopened
            relates to

            Sub-task - The sub-task of the issue HDFS-10630 Federation State Store FS Implementation

            • Major - Major loss of function.
            • Resolved

              Activity

              Hide
              Permalink
              elgoiri Íñigo Goiri added a comment - - edited

              It looks like the main two candidates to be shared between YARN and HDFS Router-based federation are:

              1. Generic State Store: Right now, in HDFS-10647 we make use of a generic State Store that stores entities. Once we want to store a new piece of information, we just need to create a new serialization/deserialization method. I'm pretty sure we could abstract this even further and it would be fairly easy to use in YARN. Actually, I think this could be even extended to components like the RMStateStore.
              2. Membership maintenance: Our proposal in HDFS-10467 is to follow a pull model to avoid changes in the Namenode. However, this brings some additional problems like fault tolerance: the Router can die while the Namenode is still running. To mitigate this, we made Routers able to monitor multiple Namenodes. Then, when the Routers get the information from the State Store, they do use quorum to unify views. Not sure if it makes sense for YARN federation to use this approach but it's worth considering. The other challenge would be to make this service abstract enough.

              Subru Krishnan, Carlo Curino, Giovanni Matteo Fumarola, any thoughts on this?

              Show
              elgoiri Íñigo Goiri added a comment - - edited It looks like the main two candidates to be shared between YARN and HDFS Router-based federation are: Generic State Store: Right now, in HDFS-10647 we make use of a generic State Store that stores entities. Once we want to store a new piece of information, we just need to create a new serialization/deserialization method. I'm pretty sure we could abstract this even further and it would be fairly easy to use in YARN. Actually, I think this could be even extended to components like the RMStateStore . Membership maintenance: Our proposal in HDFS-10467 is to follow a pull model to avoid changes in the Namenode. However, this brings some additional problems like fault tolerance: the Router can die while the Namenode is still running. To mitigate this, we made Routers able to monitor multiple Namenodes. Then, when the Routers get the information from the State Store, they do use quorum to unify views. Not sure if it makes sense for YARN federation to use this approach but it's worth considering. The other challenge would be to make this service abstract enough. Subru Krishnan , Carlo Curino , Giovanni Matteo Fumarola , any thoughts on this?

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  elgoiri Íñigo Goiri
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  10 Start watching this issue

                  Dates

                  • Created:
                    Updated:

                    Development