Hadoop Common
  1. Hadoop Common
  2. HADOOP-9905

remove dependency of zookeeper for hadoop-client

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0, 2.1.0-beta, 2.0.6-alpha
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      zookeeper dependency was added for ZKFC, which will not be used by client.
      Better remove the dependency of zookeeper jar for hadoop-client

      1. HADOOP-9905.patch
        0.8 kB
        Vinayakumar B

        Issue Links

          Activity

          Hide
          Vinayakumar B added a comment -

          Attaching patch for exclusion

          Show
          Vinayakumar B added a comment - Attaching patch for exclusion
          Hide
          Alejandro Abdelnur added a comment -

          AFAIK HDFS HA client (and soon YARN HA client) uses ZK, thus it is required.

          Show
          Alejandro Abdelnur added a comment - AFAIK HDFS HA client (and soon YARN HA client) uses ZK, thus it is required.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12599942/HADOOP-9905.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-client.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3019//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3019//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12599942/HADOOP-9905.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-client. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3019//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3019//console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          I presume it also affects 2.0.6-alpha, right?

          Show
          Konstantin Boudnik added a comment - I presume it also affects 2.0.6-alpha, right?
          Hide
          Alejandro Abdelnur added a comment -

          I just checked with Aaron T. Myers and he said that I'm wrong, that the HDFS HA client does not use ZK. Please disregard my previous comment.

          Show
          Alejandro Abdelnur added a comment - I just checked with Aaron T. Myers and he said that I'm wrong, that the HDFS HA client does not use ZK. Please disregard my previous comment.
          Hide
          Konstantin Boudnik added a comment -

          Also, it seems that the patch actually adds the dependency to hadoop-client. Am I reading the patch wrong?

          Show
          Konstantin Boudnik added a comment - Also, it seems that the patch actually adds the dependency to hadoop-client. Am I reading the patch wrong?
          Hide
          Alejandro Abdelnur added a comment -

          the patch does an exclusion, it seems OK, still have to verify it.

          Show
          Alejandro Abdelnur added a comment - the patch does an exclusion, it seems OK, still have to verify it.
          Hide
          Konstantin Boudnik added a comment -

          Thanks, I knew I was misreading it... needs more coffee.

          Show
          Konstantin Boudnik added a comment - Thanks, I knew I was misreading it... needs more coffee.
          Hide
          Vinayakumar B added a comment -

          Any more updation required on this jira..?

          Show
          Vinayakumar B added a comment - Any more updation required on this jira..?
          Hide
          Steve Loughran added a comment -

          Konstantin, if yarn-client (& hence mapred-client) needs ZK, then it should be an explict dependency in those modules. We should warn them though

          Show
          Steve Loughran added a comment - Konstantin, if yarn-client (& hence mapred-client) needs ZK, then it should be an explict dependency in those modules. We should warn them though
          Hide
          Vinod Kumar Vavilapalli added a comment -

          if yarn-client (& hence mapred-client) needs ZK, then it should be an explict dependency in those modules.

          It's not the client, the server needs it. Hopefully removing it from client doesn't affect our setup for YARN servers.

          Show
          Vinod Kumar Vavilapalli added a comment - if yarn-client (& hence mapred-client) needs ZK, then it should be an explict dependency in those modules. It's not the client, the server needs it. Hopefully removing it from client doesn't affect our setup for YARN servers.
          Hide
          Vinayakumar B added a comment -

          Hi All,
          Any changes required on this jira..?
          If no changes required can push this in..?
          Thanks

          Show
          Vinayakumar B added a comment - Hi All, Any changes required on this jira..? If no changes required can push this in..? Thanks
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12599942/HADOOP-9905.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-client.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3530//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3530//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12599942/HADOOP-9905.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 2 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-client. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3530//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3530//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          we are past Hadoop 2 GA, removing this dependency in Hadoop 2 now would break backward compatibility for apps using ZK (and expecting Hadoop to provide it)

          Show
          Alejandro Abdelnur added a comment - we are past Hadoop 2 GA, removing this dependency in Hadoop 2 now would break backward compatibility for apps using ZK (and expecting Hadoop to provide it)
          Hide
          Steve Loughran added a comment -

          Alejandro, I'm still up for dropping the dependency on ZK

          We can't have a "no changes to POMs or dependencies in Hadoop 2" policy. To do so says "we can't ever upgrade any POM we depend on, in case transient dependencies change"

          1. we explicitly say "no guarantees about dependencies" in the compatibility document. That was not an accident.
          2. this only affects downstream apps that work via maven, not production code
          3. we can mark it as 'incompatible" in the changes.

          The patch isn't ready yet as we need to make sure YARN is up with it. But otherwise, I will eventually +1 a patch that drops ZK from hadoop-cliient

          Show
          Steve Loughran added a comment - Alejandro, I'm still up for dropping the dependency on ZK We can't have a "no changes to POMs or dependencies in Hadoop 2" policy. To do so says "we can't ever upgrade any POM we depend on, in case transient dependencies change" we explicitly say "no guarantees about dependencies" in the compatibility document. That was not an accident. this only affects downstream apps that work via maven, not production code we can mark it as 'incompatible" in the changes. The patch isn't ready yet as we need to make sure YARN is up with it. But otherwise, I will eventually +1 a patch that drops ZK from hadoop-cliient
          Hide
          Vinayakumar B added a comment -

          Thanks steve.
          As for as Yarn is concerned Vinod has already confirmed that zookeeper dependency required only for server. Removing it will not affect the clients.

          Show
          Vinayakumar B added a comment - Thanks steve. As for as Yarn is concerned Vinod has already confirmed that zookeeper dependency required only for server. Removing it will not affect the clients.
          Hide
          Steve Loughran added a comment -

          @Vinay -yes, but we need to make sure that the server POMs are set up to pull it in once the client POM drops it

          Show
          Steve Loughran added a comment - @Vinay -yes, but we need to make sure that the server POMs are set up to pull it in once the client POM drops it
          Hide
          Vinayakumar B added a comment -

          Defining in server pom may not be required. That will be inherited from hadoop common dependency. Excluding in hadoop client will not affect that.

          Show
          Vinayakumar B added a comment - Defining in server pom may not be required. That will be inherited from hadoop common dependency. Excluding in hadoop client will not affect that.

            People

            • Assignee:
              Vinayakumar B
              Reporter:
              Vinayakumar B
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development