Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3774

ZKRMStateStore should use CuratorOp when we upgrade to Curator 3

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.8.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      YARN-2716 changes ZKRMStateStore to use Curator. Transactions added there are somewhat involved, and could be improved using CuratorOp introduced in Curator 3.0. Hadoop 3.0.0 would be a good time to upgrade the Curator version and make this change.

      Curator is considering shading guava through CURATOR-200. In Hadoop 3, we should upgrade to the next Curator version.

        Issue Links

          Activity

          Hide
          rkanter Robert Kanter added a comment -

          That seems reasonable. I've updated the JIRA.

          Show
          rkanter Robert Kanter added a comment - That seems reasonable. I've updated the JIRA.
          Hide
          templedf Daniel Templeton added a comment -

          Are we sure deleting the comment is the right thing? Isn't the comment still accurate? At some point in the future we may adopt Curator 3.0, at which time the comment and this JIRA become relevant again. I'd rather just put this JIRA back onto the back burner.

          Show
          templedf Daniel Templeton added a comment - Are we sure deleting the comment is the right thing? Isn't the comment still accurate? At some point in the future we may adopt Curator 3.0, at which time the comment and this JIRA become relevant again. I'd rather just put this JIRA back onto the back burner.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 14m 3s trunk passed
          +1 compile 0m 33s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 36s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 4s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 33s the patch passed
          +1 compile 0m 33s the patch passed
          +1 javac 0m 33s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 33s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 7s the patch passed
          +1 javadoc 0m 20s the patch passed
          -1 unit 38m 25s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          61m 21s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue YARN-3774
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863669/YARN-3774.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 30f642b5e001 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ac3cfdf
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15652/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15652/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15652/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 3s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 4s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 33s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 7s the patch passed +1 javadoc 0m 20s the patch passed -1 unit 38m 25s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 61m 21s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue YARN-3774 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863669/YARN-3774.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 30f642b5e001 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ac3cfdf Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/15652/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15652/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15652/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          I talked with Karthik about this and there isn't anything to do here other than remove the comment about it in the code. Curator 3.x has a newer Transaction API that looks a bit different, and has deprecated the one we're using, but we're going to stay on Curator 2.x to continue supporting ZooKeeper 3.4.x for now, so it's not deprecated. And we're already using the latest version of Curator 2.x (2.12.0).

          Show
          rkanter Robert Kanter added a comment - I talked with Karthik about this and there isn't anything to do here other than remove the comment about it in the code. Curator 3.x has a newer Transaction API that looks a bit different, and has deprecated the one we're using, but we're going to stay on Curator 2.x to continue supporting ZooKeeper 3.4.x for now, so it's not deprecated. And we're already using the latest version of Curator 2.x (2.12.0).
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks Robert for expressing interest in picking this up.

          Show
          kasha Karthik Kambatla added a comment - Thanks Robert for expressing interest in picking this up.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks Jordan for the notification! I think we should use 3.3.0, 2.12.0 or later.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Thanks Jordan for the notification! I think we should use 3.3.0, 2.12.0 or later.
          Hide
          randgalt Jordan Zimmerman added a comment -

          FYI - this will be fixed in the next release of Curator per CURATOR-200

          Show
          randgalt Jordan Zimmerman added a comment - FYI - this will be fixed in the next release of Curator per CURATOR-200
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          FYI: http://curator.apache.org/dependencies.html

          Curator 2.7.1 also depends on Guava 16.0.1 though Hadoop depends on Guava 11.0.2. IIUC, we have a possibility to face method missing error at runtime when we use specific features which is available only with Guava 16.0.1.
          Steve describes what happens with upgrading Curator version on HADOOP-11102.

          We have a chance to use maven-jarjar-plugin to avoid conflicts about Guava when we deal with HADOOP-11656.
          http://sonatype.github.io/jarjar-maven-plugin/

          Show
          ozawa Tsuyoshi Ozawa added a comment - FYI: http://curator.apache.org/dependencies.html Curator 2.7.1 also depends on Guava 16.0.1 though Hadoop depends on Guava 11.0.2. IIUC, we have a possibility to face method missing error at runtime when we use specific features which is available only with Guava 16.0.1. Steve describes what happens with upgrading Curator version on HADOOP-11102 . We have a chance to use maven-jarjar-plugin to avoid conflicts about Guava when we deal with HADOOP-11656 . http://sonatype.github.io/jarjar-maven-plugin/
          Hide
          stevel@apache.org Steve Loughran added a comment -

          ..and then there's the transitive guava dependency

          Show
          stevel@apache.org Steve Loughran added a comment - ..and then there's the transitive guava dependency
          Hide
          kasha Karthik Kambatla added a comment -

          Fair point, Sean. We should hold off until Hadoop 3, primarily to handle potential compat issues.

          Show
          kasha Karthik Kambatla added a comment - Fair point, Sean. We should hold off until Hadoop 3, primarily to handle potential compat issues.
          Hide
          busbey Sean Busbey added a comment -

          Please make sure to document the impact of moving to Curator 3. The last time we updated the curator version (2.6.0 -> 2.7.1) they broke compatibility.

          Show
          busbey Sean Busbey added a comment - Please make sure to document the impact of moving to Curator 3. The last time we updated the curator version (2.6.0 -> 2.7.1) they broke compatibility.

            People

            • Assignee:
              Unassigned
              Reporter:
              kasha Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development