Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3774

ZKRMStateStore should use Curator 3.0 and avail CuratorOp

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.8.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      YARN-2716 changes ZKRMStateStore to use Curator. Transactions added there are somewhat involved, and could be improved using CuratorOp introduced in Curator 3.0. Hadoop 3.0.0 would be a good time to upgrade the Curator version and make this change.

      Curator is considering shading guava through CURATOR-200. In Hadoop 3, we should upgrade to the next Curator version.

        Issue Links

          Activity

          Hide
          busbey Sean Busbey added a comment -

          Please make sure to document the impact of moving to Curator 3. The last time we updated the curator version (2.6.0 -> 2.7.1) they broke compatibility.

          Show
          busbey Sean Busbey added a comment - Please make sure to document the impact of moving to Curator 3. The last time we updated the curator version (2.6.0 -> 2.7.1) they broke compatibility.
          Hide
          kasha Karthik Kambatla added a comment -

          Fair point, Sean. We should hold off until Hadoop 3, primarily to handle potential compat issues.

          Show
          kasha Karthik Kambatla added a comment - Fair point, Sean. We should hold off until Hadoop 3, primarily to handle potential compat issues.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          ..and then there's the transitive guava dependency

          Show
          stevel@apache.org Steve Loughran added a comment - ..and then there's the transitive guava dependency
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          FYI: http://curator.apache.org/dependencies.html

          Curator 2.7.1 also depends on Guava 16.0.1 though Hadoop depends on Guava 11.0.2. IIUC, we have a possibility to face method missing error at runtime when we use specific features which is available only with Guava 16.0.1.
          Steve describes what happens with upgrading Curator version on HADOOP-11102.

          We have a chance to use maven-jarjar-plugin to avoid conflicts about Guava when we deal with HADOOP-11656.
          http://sonatype.github.io/jarjar-maven-plugin/

          Show
          ozawa Tsuyoshi Ozawa added a comment - FYI: http://curator.apache.org/dependencies.html Curator 2.7.1 also depends on Guava 16.0.1 though Hadoop depends on Guava 11.0.2. IIUC, we have a possibility to face method missing error at runtime when we use specific features which is available only with Guava 16.0.1. Steve describes what happens with upgrading Curator version on HADOOP-11102 . We have a chance to use maven-jarjar-plugin to avoid conflicts about Guava when we deal with HADOOP-11656 . http://sonatype.github.io/jarjar-maven-plugin/
          Hide
          randgalt Jordan Zimmerman added a comment -

          FYI - this will be fixed in the next release of Curator per CURATOR-200

          Show
          randgalt Jordan Zimmerman added a comment - FYI - this will be fixed in the next release of Curator per CURATOR-200
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks Jordan for the notification! I think we should use 3.3.0, 2.12.0 or later.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Thanks Jordan for the notification! I think we should use 3.3.0, 2.12.0 or later.

            People

            • Assignee:
              kasha Karthik Kambatla
              Reporter:
              kasha Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development