Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13378

Common features between YARN and HDFS Router-based federation

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      HDFS-10467 uses a similar architecture to the one proposed in YARN-2915. This JIRA tries to identify what is common between these two efforts and try to build a common framework.

        Issue Links

          Activity

          Hide
          elgoiri Inigo Goiri added a comment - - edited

          It looks like the main two candidates to be shared between YARN and HDFS Router-based federation are:

          1. Generic State Store: Right now, in HDFS-10647 we make use of a generic State Store that stores entities. Once we want to store a new piece of information, we just need to create a new serialization/deserialization method. I'm pretty sure we could abstract this even further and it would be fairly easy to use in YARN. Actually, I think this could be even extended to components like the RMStateStore.
          2. Membership maintenance: Our proposal in HDFS-10467 is to follow a pull model to avoid changes in the Namenode. However, this brings some additional problems like fault tolerance: the Router can die while the Namenode is still running. To mitigate this, we made Routers able to monitor multiple Namenodes. Then, when the Routers get the information from the State Store, they do use quorum to unify views. Not sure if it makes sense for YARN federation to use this approach but it's worth considering. The other challenge would be to make this service abstract enough.

          Subru Krishnan, Carlo Curino, Giovanni Matteo Fumarola, any thoughts on this?

          Show
          elgoiri Inigo Goiri added a comment - - edited It looks like the main two candidates to be shared between YARN and HDFS Router-based federation are: Generic State Store: Right now, in HDFS-10647 we make use of a generic State Store that stores entities. Once we want to store a new piece of information, we just need to create a new serialization/deserialization method. I'm pretty sure we could abstract this even further and it would be fairly easy to use in YARN. Actually, I think this could be even extended to components like the RMStateStore . Membership maintenance: Our proposal in HDFS-10467 is to follow a pull model to avoid changes in the Namenode. However, this brings some additional problems like fault tolerance: the Router can die while the Namenode is still running. To mitigate this, we made Routers able to monitor multiple Namenodes. Then, when the Routers get the information from the State Store, they do use quorum to unify views. Not sure if it makes sense for YARN federation to use this approach but it's worth considering. The other challenge would be to make this service abstract enough. Subru Krishnan , Carlo Curino , Giovanni Matteo Fumarola , any thoughts on this?

            People

            • Assignee:
              Unassigned
              Reporter:
              elgoiri Inigo Goiri
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development