Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      MAPREDUCE-279 included bits and pieces of possible ZK integration for YARN's RM, but looks like it failed to complete it (for scalability reasons? etc?) and there seems to be no JIRA tracking this feature that has been already claimed publicly as a good part about YARN.

      If it did complete it, we should document how to use it. Setting the following only yields:

      <property>
      <name>yarn.resourcemanager.store.class</name>
      <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKStore</value>
      </property>
      <property>
      <name>yarn.resourcemanager.zookeeper-store.address</name>
      <value>test.vm:2181/yarn-recovery-store</value>
      </property>
      
      Error starting ResourceManager
      java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKStore.<init>()
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:128)
      at org.apache.hadoop.yarn.server.resourcemanager.recovery.StoreFactory.getStore(StoreFactory.java:32)
      at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:621)
      Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKStore.<init>()
      at java.lang.Class.getConstructor0(Class.java:2706)
      at java.lang.Class.getDeclaredConstructor(Class.java:1985)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:122)
      ... 2 more
      

      This JIRA is hence filed to track the addition/completion of recovery via ZK.

      1. MR-4343.1.patch
        17 kB
        Tsuyoshi OZAWA

        Issue Links

          Activity

          Hide
          Tsuyoshi OZAWA added a comment -

          Harsh,

          I think this problem should be split into 2 tickets as follows:

          1. Fix the runtime error of ZKStore by adding default constructor.
          2. Add ZK recovery support to the resource manager.

          In fact, I've already created the patch for fixing the 1st problem.
          Should I attach the file here or on the new ticket?

          Show
          Tsuyoshi OZAWA added a comment - Harsh, I think this problem should be split into 2 tickets as follows: 1. Fix the runtime error of ZKStore by adding default constructor. 2. Add ZK recovery support to the resource manager. In fact, I've already created the patch for fixing the 1st problem. Should I attach the file here or on the new ticket?
          Hide
          Harsh J added a comment -

          Hi,

          With (1) done in your local build, are you able to successfully validate that (2) is possible today already? In case it is (that the RM recovers its state properly, once ZKStore is fixed), we can just do (1) here and add some docs on how to use ZKStore (the configs) and resolve the ticket.

          Show
          Harsh J added a comment - Hi, With (1) done in your local build, are you able to successfully validate that (2) is possible today already? In case it is (that the RM recovers its state properly, once ZKStore is fixed), we can just do (1) here and add some docs on how to use ZKStore (the configs) and resolve the ticket.
          Hide
          Tsuyoshi OZAWA added a comment -

          I'm trying to solve the 2nd problem now, and I've not estimated how large the work is.
          I'll report the status before I go to sleep.

          Show
          Tsuyoshi OZAWA added a comment - I'm trying to solve the 2nd problem now, and I've not estimated how large the work is. I'll report the status before I go to sleep.
          Hide
          Tsuyoshi OZAWA added a comment -

          Harsh,

          The attached patch allows ResourceManager to use ZKStore.

          Show
          Tsuyoshi OZAWA added a comment - Harsh, The attached patch allows ResourceManager to use ZKStore.
          Hide
          Tsuyoshi OZAWA added a comment -

          It's very course patch but works on my environment, so could you review it?

          Show
          Tsuyoshi OZAWA added a comment - It's very course patch but works on my environment, so could you review it?
          Hide
          Tsuyoshi OZAWA added a comment -

          s/course/coarse/

          Show
          Tsuyoshi OZAWA added a comment - s/course/coarse/
          Hide
          Sharad Agarwal added a comment -

          There is already MAPREDUCE-2713 for this. Some ZK code may be lying around but it is not implemented as yet.

          can this be marked as duplicate ?

          Show
          Sharad Agarwal added a comment - There is already MAPREDUCE-2713 for this. Some ZK code may be lying around but it is not implemented as yet. can this be marked as duplicate ?
          Hide
          Tsuyoshi OZAWA added a comment -

          Sharad,

          Bikas marked MAPREDUCE-2713 as a duplicated task.

          Show
          Tsuyoshi OZAWA added a comment - Sharad, Bikas marked MAPREDUCE-2713 as a duplicated task.
          Hide
          Arun C Murthy added a comment -

          Duplicate of MAPREDUCE-4326.

          Show
          Arun C Murthy added a comment - Duplicate of MAPREDUCE-4326 .
          Hide
          Harsh J added a comment -

          Thanks Sharad and Arun. I missed out MAPREDUCE-2713 in my search (apparently missed searching for 'recover' (instead searched for availability)). Sorry for the noise.

          Thanks much for the immediate patch Tsuyoshi. It may be considered now in MAPREDUCE-4326 unless the whole thing is being revamped.

          Show
          Harsh J added a comment - Thanks Sharad and Arun. I missed out MAPREDUCE-2713 in my search (apparently missed searching for 'recover' (instead searched for availability)). Sorry for the noise. Thanks much for the immediate patch Tsuyoshi. It may be considered now in MAPREDUCE-4326 unless the whole thing is being revamped.

            People

            • Assignee:
              Unassigned
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development