Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.5.2, 3.6.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Remove rpm/deb/... packaging from our source repo. Now that BigTop is available and fully supporting ZK it's no longer necessary for us to attempt to include this.

      1. ZOOKEEPER-1604.patch
        51 kB
        Patrick Hunt
      2. ZOOKEEPER-1604.001.patch
        51 kB
        Chris Nauroth

        Issue Links

          Activity

          Hide
          Andrew Purtell added a comment -

          You are not talking about anything to do with zookeeper Eric Yang, especially since they're getting out of the self packaging business. Why don't you bring your opinions over to the Bigtop mailing lists since it's completely off topic to have the discussion you want here.

          Show
          Andrew Purtell added a comment - You are not talking about anything to do with zookeeper Eric Yang , especially since they're getting out of the self packaging business. Why don't you bring your opinions over to the Bigtop mailing lists since it's completely off topic to have the discussion you want here.
          Hide
          Eric Yang added a comment -

          Exim is only active if no other MTA programs are installed. This usually only happens on minimal install without sendmail. Redhat closes these type of bugs as not a problem because LSB specification enforces MTA dependency. The question is, does using ZooKeeper need to comply to LSB specification? I hope we don't tie up in ball and chains to use a light weight program like ZooKeeper. Of course, I could be wrong and people like to have a MTA on their ZooKeeper servers.

          Show
          Eric Yang added a comment - Exim is only active if no other MTA programs are installed. This usually only happens on minimal install without sendmail. Redhat closes these type of bugs as not a problem because LSB specification enforces MTA dependency. The question is, does using ZooKeeper need to comply to LSB specification? I hope we don't tie up in ball and chains to use a light weight program like ZooKeeper. Of course, I could be wrong and people like to have a MTA on their ZooKeeper servers.
          Hide
          Andrew Purtell added a comment -

          Eric Yang your complaint seems to be with RedHat as best as I can gather. Like Olaf Flebbe I'm not seeing exim installed, but this is a Linux OS issue at any rate having nothing to do with ZooKeeper or Bigtop. Suggest you file something on the RedHat bug tracker.

          Show
          Andrew Purtell added a comment - Eric Yang your complaint seems to be with RedHat as best as I can gather. Like Olaf Flebbe I'm not seeing exim installed, but this is a Linux OS issue at any rate having nothing to do with ZooKeeper or Bigtop. Suggest you file something on the RedHat bug tracker.
          Hide
          Eric Yang added a comment -

          Does every Linux box needs MTA? Government sector is one of the customers that ask for removal of MTA dependencies. People don't bind Java program to lower port and use firewall/reverse proxy to shield their system as solutions. I agree there are ways to work around problems. However, one less problem for novice is better than leaving problems behind for future generations. This is the reason that I am cool with removing the current rpm/deb packaging for ant build. However, if someone put patches forward for making rpm package with better tooling. I hope we don't stumble on them like this incidence.

          Show
          Eric Yang added a comment - Does every Linux box needs MTA? Government sector is one of the customers that ask for removal of MTA dependencies. People don't bind Java program to lower port and use firewall/reverse proxy to shield their system as solutions. I agree there are ways to work around problems. However, one less problem for novice is better than leaving problems behind for future generations. This is the reason that I am cool with removing the current rpm/deb packaging for ant build. However, if someone put patches forward for making rpm package with better tooling. I hope we don't stumble on them like this incidence.
          Hide
          Olaf Flebbe added a comment -

          Exim is not in any way installed by Bigtop on Centos/RHEL. At least not in Bigtop-1.0.0 or later.

          LSB-core happens to require a MTA, any one. And Centos/RHEL redhat-lsb-core chooses sendmail rather then exim at least in the bigtop default setup.

          rpm -q zookeeper-server redhat-lsb-core sendmail exim
          zookeeper-server-3.4.6-1.el6.x86_64
          redhat-lsb-core-4.0-7.el6.centos.x86_64
          sendmail-8.14.4-9.el6.x86_64
          package exim is not installed
          

          Either way, please consider java a huge target for remote exploits and avoid it. See for instance.
          https://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-19117/Oracle-JRE.html .

          Show
          Olaf Flebbe added a comment - Exim is not in any way installed by Bigtop on Centos/RHEL. At least not in Bigtop-1.0.0 or later. LSB-core happens to require a MTA, any one. And Centos/RHEL redhat-lsb-core chooses sendmail rather then exim at least in the bigtop default setup. rpm -q zookeeper-server redhat-lsb-core sendmail exim zookeeper-server-3.4.6-1.el6.x86_64 redhat-lsb-core-4.0-7.el6.centos.x86_64 sendmail-8.14.4-9.el6.x86_64 package exim is not installed Either way, please consider java a huge target for remote exploits and avoid it. See for instance. https://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-19117/Oracle-JRE.html .
          Hide
          Flavio Junqueira added a comment -

          Eric Yang at this point, I'm a bit confused about what you'd like to happen. You seem to be dissatisfied with BigTop and you seem to want this community to move in a different different direction with respect to packaging. I believe we are all trying to move forward as a community, so why don't why start a thread on the dev list explaining what is troubling you and what you'd like to happen? I'm suggesting it because the scope of this jira is too narrow for what you seem to be trying to convey. I also suggest that you try to be specific about what you'd like to happen and how you'd like to contribute for it to happen, and let's see how the community reacts. Is that fair?

          Show
          Flavio Junqueira added a comment - Eric Yang at this point, I'm a bit confused about what you'd like to happen. You seem to be dissatisfied with BigTop and you seem to want this community to move in a different different direction with respect to packaging. I believe we are all trying to move forward as a community, so why don't why start a thread on the dev list explaining what is troubling you and what you'd like to happen? I'm suggesting it because the scope of this jira is too narrow for what you seem to be trying to convey. I also suggest that you try to be specific about what you'd like to happen and how you'd like to contribute for it to happen, and let's see how the community reacts. Is that fair?
          Hide
          Eric Yang added a comment -

          Konstantin,

          Exim is a soft target for root exploit. I am only stating the obvious to help Apache projects.

          https://www.cvedetails.com/vulnerability-list/vendor_id-10919/product_id-19563/Exim-Exim.html

          What you do with this information is entirely up to you. The patch to remove redhat-lsb-core is located here:

          https://issues.apache.org/jira/browse/BIGTOP-1194

          Inverse patch it, then your system has one less threat.

          What the community does to remove rpm packages from the projects, that is community's choice. I respect that, but it is also irritating to see that projects are all tight up to a monolithic packaging project. BigTop copied packaging technique from HADOOP-6255 to enable support for packaging both deb and rpm packages for most of the projects surrounding Hadoop. While I appreciate the hard work that bigtop committers invested in Bigtop. It is just awkward to use for people that don't want to have the monolithic build system. It's like a zombie that put on a blue dress to imitate Elsa from Disney Frozen. The bigtop show just won't work for some parents.

          Show
          Eric Yang added a comment - Konstantin, Exim is a soft target for root exploit. I am only stating the obvious to help Apache projects. https://www.cvedetails.com/vulnerability-list/vendor_id-10919/product_id-19563/Exim-Exim.html What you do with this information is entirely up to you. The patch to remove redhat-lsb-core is located here: https://issues.apache.org/jira/browse/BIGTOP-1194 Inverse patch it, then your system has one less threat. What the community does to remove rpm packages from the projects, that is community's choice. I respect that, but it is also irritating to see that projects are all tight up to a monolithic packaging project. BigTop copied packaging technique from HADOOP-6255 to enable support for packaging both deb and rpm packages for most of the projects surrounding Hadoop. While I appreciate the hard work that bigtop committers invested in Bigtop. It is just awkward to use for people that don't want to have the monolithic build system. It's like a zombie that put on a blue dress to imitate Elsa from Disney Frozen. The bigtop show just won't work for some parents.
          Hide
          Konstantin Boudnik added a comment -

          It is quite fascinating to read this exchange. Especially, Eric Yang's laying it thick on everyone including LSB community.

          I see two ways to resolve your grudge: you can stop using Linux (or just LSB part of it, perhaps?) because of the commonly known exim vulnerability (and perhaps a few others). Or you can contribute a patch to Bigtop to fix the issue.

          Perhaps, we have stepped on your toe somewhere along the road. If we did - please accept my sincere apologies. And let me make it worth your while: once your patch resolving this alleged exim issue is submitted, I will personally make it my utmost priority to review and bring it in ASAP. Until then, please be considerate not to bad-mouth your fellow Apache contributors nor the project you haven't ever personally contributed to.

          Show
          Konstantin Boudnik added a comment - It is quite fascinating to read this exchange. Especially, Eric Yang 's laying it thick on everyone including LSB community. I see two ways to resolve your grudge: you can stop using Linux (or just LSB part of it, perhaps?) because of the commonly known exim vulnerability (and perhaps a few others). Or you can contribute a patch to Bigtop to fix the issue. Perhaps, we have stepped on your toe somewhere along the road. If we did - please accept my sincere apologies. And let me make it worth your while: once your patch resolving this alleged exim issue is submitted, I will personally make it my utmost priority to review and bring it in ASAP. Until then, please be considerate not to bad-mouth your fellow Apache contributors nor the project you haven't ever personally contributed to.
          Hide
          Patrick Hunt added a comment -

          Here are a couple things to also consider.

          Maybe the patches aren't getting reviewed because we (committers) don't have the expertise to review, nor the infrastructure to test the changes. (let's assume we have the time, although based on the PA load I don't think that's true). From what I saw folks were hesitant to move these forward w/o understanding and validating the impact.

          You can get ZooKeeper packages from Cloudera, certainly, but you can also get them from Hortonworks, or from Canonical (deb) http://packages.ubuntu.com/search?keywords=zookeeper or RPMs http://rpmfind.net/linux/rpm2html/search.php?query=zookeeper etc...

          The Hadoop ecosystem has moved forward. There just isn't the value in a project trying to maintain it's own packages when others are already doing so. Apache already has it's own packaging infrastructure that lots of folks are using and contributing to - Apache Bigtop. I'd rather see the investment go there.

          Show
          Patrick Hunt added a comment - Here are a couple things to also consider. Maybe the patches aren't getting reviewed because we (committers) don't have the expertise to review, nor the infrastructure to test the changes. (let's assume we have the time, although based on the PA load I don't think that's true). From what I saw folks were hesitant to move these forward w/o understanding and validating the impact. You can get ZooKeeper packages from Cloudera, certainly, but you can also get them from Hortonworks, or from Canonical (deb) http://packages.ubuntu.com/search?keywords=zookeeper or RPMs http://rpmfind.net/linux/rpm2html/search.php?query=zookeeper etc... The Hadoop ecosystem has moved forward. There just isn't the value in a project trying to maintain it's own packages when others are already doing so. Apache already has it's own packaging infrastructure that lots of folks are using and contributing to - Apache Bigtop. I'd rather see the investment go there.
          Hide
          Eric Yang added a comment -

          If ZOOKEEPER-2007 was committed, then ZOOKEEPER-2275 would not exist. Evidently, There are people that are willing to provide patch for packaging issues. Instead, ZOOKEEPER-2007 patch were left without review and code became stale. (Code did not apply due to pre-commit test did not verify patch depth, not the patch.) BigTop has some security issues, including non-uniform uid/gid generation on cluster, and including security vulnerable packages. I am only presenting facts for others to consider. It seems like it's a few steps backward because BigTop project create interlocks of open source projects instead of allow individual project to evolve on their own pace. Ex-Cloudera developers had made some poor choices in BigTop project direction, rather than allowing other projects to evolve. Granted that this is my opinion, and I respect yours. Good luck with BigTop.

          Show
          Eric Yang added a comment - If ZOOKEEPER-2007 was committed, then ZOOKEEPER-2275 would not exist. Evidently, There are people that are willing to provide patch for packaging issues. Instead, ZOOKEEPER-2007 patch were left without review and code became stale. (Code did not apply due to pre-commit test did not verify patch depth, not the patch.) BigTop has some security issues, including non-uniform uid/gid generation on cluster, and including security vulnerable packages. I am only presenting facts for others to consider. It seems like it's a few steps backward because BigTop project create interlocks of open source projects instead of allow individual project to evolve on their own pace. Ex-Cloudera developers had made some poor choices in BigTop project direction, rather than allowing other projects to evolve. Granted that this is my opinion, and I respect yours. Good luck with BigTop.
          Hide
          Patrick Hunt added a comment -

          Hi Eric Yang, I didn't mean to exclude anyone. Based on my reading of the recent comments it seemed like the questions had been answered, and we had reached consensus, e.g. Ralph's "I've come around on this issue" and comments from Chris and others. A patch had been posted (not by me) and no negative comments have come in in over six months.

          I also feel that your comments re Cloudera are unfair to me. My main goal here was just to clean things up and move forward. However you're entitled to your opinion.

          Additionally this is consistent with what a number of other components (Hadoop) have done over time. Our packaging was created before Bigtop existed, a number of issues have been reported but not resolved, and it seemed like it was causing more problems than it solved. e.g. Chris's comment "ZOOKEEPER-2275 is another new incoming issue where the presence of broken rpm packaging code in the build has caused confusion."

          EOD we just don't have community available to support packaging, we barely have enough folks to address issues found with the service itself, and while there may have been "interest" there unfortunately hasn't been enough "action".

          Wouldn't it make more sense to focus on the Bigtop packaging rather than duplicate it here, poorly?

          Show
          Patrick Hunt added a comment - Hi Eric Yang , I didn't mean to exclude anyone. Based on my reading of the recent comments it seemed like the questions had been answered, and we had reached consensus, e.g. Ralph's "I've come around on this issue" and comments from Chris and others. A patch had been posted (not by me) and no negative comments have come in in over six months. I also feel that your comments re Cloudera are unfair to me. My main goal here was just to clean things up and move forward. However you're entitled to your opinion. Additionally this is consistent with what a number of other components (Hadoop) have done over time. Our packaging was created before Bigtop existed, a number of issues have been reported but not resolved, and it seemed like it was causing more problems than it solved. e.g. Chris's comment " ZOOKEEPER-2275 is another new incoming issue where the presence of broken rpm packaging code in the build has caused confusion." EOD we just don't have community available to support packaging, we barely have enough folks to address issues found with the service itself, and while there may have been "interest" there unfortunately hasn't been enough "action". Wouldn't it make more sense to focus on the Bigtop packaging rather than duplicate it here, poorly?
          Hide
          Chris Nauroth added a comment -

          Eric Yang, I'm sorry to hear this change disappoints you. However, we did allow this issue to remain open for discussion for multiple years, with a lot of the discussion happening in the past 10 months. Based on past history, we have not had success maintaining packaging infrastructure within the ZooKeeper project itself. (It has been broken at various times without anyone noticing for a long time.) Many projects have externalized their packaging infrastructure to BigTop. We felt we had reached a consensus decision that the overall best choice for the project was to remove our own packaging infrastructure.

          Patrick and I and others are in agreement about this change, even though we don't share a common employer. We are trying our best to act in a way that benefits as much of the Apache community as possible. Unfortunately, any resolution of this issue is likely to be disappointing to someone. Sometimes compromise is necessary.

          I hope you can reconsider using BigTop. Would you consider raising your security concern with the BigTop community?

          Show
          Chris Nauroth added a comment - Eric Yang , I'm sorry to hear this change disappoints you. However, we did allow this issue to remain open for discussion for multiple years, with a lot of the discussion happening in the past 10 months. Based on past history, we have not had success maintaining packaging infrastructure within the ZooKeeper project itself. (It has been broken at various times without anyone noticing for a long time.) Many projects have externalized their packaging infrastructure to BigTop. We felt we had reached a consensus decision that the overall best choice for the project was to remove our own packaging infrastructure. Patrick and I and others are in agreement about this change, even though we don't share a common employer. We are trying our best to act in a way that benefits as much of the Apache community as possible. Unfortunately, any resolution of this issue is likely to be disappointing to someone. Sometimes compromise is necessary. I hope you can reconsider using BigTop. Would you consider raising your security concern with the BigTop community?
          Hide
          Eric Yang added a comment -

          There are 4 people expressed interest to keep this in source code. Yet, Cloudera push in changes regardless of people protest against this issue. This is a sad day for Apache community. In addition, Bigtop contains /lib/lsb/init-functions which will import redhat-lsb-core which imports exim. Exim is known for common root escalation vulnerability. If you value your cluster security, I would recommend to think twice before using BigTop.

          Show
          Eric Yang added a comment - There are 4 people expressed interest to keep this in source code. Yet, Cloudera push in changes regardless of people protest against this issue. This is a sad day for Apache community. In addition, Bigtop contains /lib/lsb/init-functions which will import redhat-lsb-core which imports exim. Exim is known for common root escalation vulnerability. If you value your cluster security, I would recommend to think twice before using BigTop.
          Hide
          Chris Nauroth added a comment -

          Patrick Hunt, thank you for reviewing and committing.

          Show
          Chris Nauroth added a comment - Patrick Hunt , thank you for reviewing and committing.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in ZooKeeper-trunk #2836 (See https://builds.apache.org/job/ZooKeeper-trunk/2836/)
          ZOOKEEPER-1604 remove rpm/deb/... packaging (cnauroth via phunt) (phunt: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1733489)

          • trunk/README_packaging.txt
          • trunk/build.xml
          • trunk/ivy.xml
          • trunk/src/contrib/build-contrib.xml
          • trunk/src/contrib/build.xml
          • trunk/src/contrib/zkpython/build.xml
          • trunk/src/contrib/zkpython/ivy.xml
          • trunk/src/contrib/zkpython/src/packages
          • trunk/src/packages
          • trunk/src/recipes/build-recipes.xml
          • trunk/src/recipes/build.xml
          Show
          Hudson added a comment - FAILURE: Integrated in ZooKeeper-trunk #2836 (See https://builds.apache.org/job/ZooKeeper-trunk/2836/ ) ZOOKEEPER-1604 remove rpm/deb/... packaging (cnauroth via phunt) (phunt: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1733489 ) trunk/README_packaging.txt trunk/build.xml trunk/ivy.xml trunk/src/contrib/build-contrib.xml trunk/src/contrib/build.xml trunk/src/contrib/zkpython/build.xml trunk/src/contrib/zkpython/ivy.xml trunk/src/contrib/zkpython/src/packages trunk/src/packages trunk/src/recipes/build-recipes.xml trunk/src/recipes/build.xml
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12791195/ZOOKEEPER-1604.patch
          against trunk revision 1733348.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3079//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3079//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3079//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12791195/ZOOKEEPER-1604.patch against trunk revision 1733348. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3079//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3079//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3079//console This message is automatically generated.
          Hide
          Patrick Hunt added a comment -

          Makes sense to remove this given Bigtop. Thanks Chris!

          Show
          Patrick Hunt added a comment - Makes sense to remove this given Bigtop. Thanks Chris!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12791195/ZOOKEEPER-1604.patch
          against trunk revision 1733348.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3078//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3078//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3078//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12791195/ZOOKEEPER-1604.patch against trunk revision 1733348. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3078//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3078//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3078//console This message is automatically generated.
          Hide
          Patrick Hunt added a comment -

          Updated the original patch to apply cleanly. Removed debian target as well.

          Show
          Patrick Hunt added a comment - Updated the original patch to apply cleanly. Removed debian target as well.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726926/ZOOKEEPER-1604.001.patch
          against trunk revision 1733348.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3077//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726926/ZOOKEEPER-1604.001.patch against trunk revision 1733348. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3077//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726926/ZOOKEEPER-1604.001.patch
          against trunk revision 1702378.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2875//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726926/ZOOKEEPER-1604.001.patch against trunk revision 1702378. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2875//console This message is automatically generated.
          Hide
          Chris Nauroth added a comment -

          ZOOKEEPER-2275 is another new incoming issue where the presence of broken rpm packaging code in the build has caused confusion. Can we try again to reach a decision on this issue? Shall we drop rpm/deb/etc. packaging support within the ZooKeeper build? Last time we discussed it, I came around to thinking it was best overall, so I posted a patch that does the removal.

          If instead the decision is to maintain it, then let's get ZOOKEEPER-2275 in so that it's not broken.

          Show
          Chris Nauroth added a comment - ZOOKEEPER-2275 is another new incoming issue where the presence of broken rpm packaging code in the build has caused confusion. Can we try again to reach a decision on this issue? Shall we drop rpm/deb/etc. packaging support within the ZooKeeper build? Last time we discussed it, I came around to thinking it was best overall, so I posted a patch that does the removal. If instead the decision is to maintain it, then let's get ZOOKEEPER-2275 in so that it's not broken.
          Hide
          Chris Nauroth added a comment -

          The Bigtop rpm/deb packages are stand-alone. Those are visible here:

          http://bigtop.s3.amazonaws.com/

          It's definitely possible to install just a Zookeeper package from Bigtop. The point was more about discussing whether or not it would be valuable to offer rpm/deb packages in ZooKeeper's own release cycle, independent of Bigtop. As Patrick points out, there is always the tarball if a user needs to deploy a patch urgently before Bigtop publishes a package.

          Show
          Chris Nauroth added a comment - The Bigtop rpm/deb packages are stand-alone. Those are visible here: http://bigtop.s3.amazonaws.com/ It's definitely possible to install just a Zookeeper package from Bigtop. The point was more about discussing whether or not it would be valuable to offer rpm/deb packages in ZooKeeper's own release cycle, independent of Bigtop. As Patrick points out, there is always the tarball if a user needs to deploy a patch urgently before Bigtop publishes a package.
          Hide
          Michi Mutsuzaki added a comment -

          How does Bigtop package ZooKeeper? It's not a standalone rpm/deb package?

          Show
          Michi Mutsuzaki added a comment - How does Bigtop package ZooKeeper? It's not a standalone rpm/deb package?
          Hide
          Chris Nauroth added a comment -

          -1 tests included. The patch doesn't appear to include any new or modified tests.

          There are no new tests, because this is a change in the build only. My manual test procedure was:

          1. Verify ant rpm does nothing/fails.
          2. Verify ant deb does nothing/fails.
          3. Verify ant package-native tar succeeds. Deploy the resulting zookeeper-<VERSION>.tar.gz and zookeeper-<VERSION>-lib.tar.gz. Run a single node and exercise some basic create/get/set commands.
          Show
          Chris Nauroth added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. There are no new tests, because this is a change in the build only. My manual test procedure was: Verify ant rpm does nothing/fails. Verify ant deb does nothing/fails. Verify ant package-native tar succeeds. Deploy the resulting zookeeper-<VERSION>.tar.gz and zookeeper-<VERSION>-lib.tar.gz. Run a single node and exercise some basic create/get/set commands.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726926/ZOOKEEPER-1604.001.patch
          against trunk revision 1672934.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2643//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2643//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2643//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726926/ZOOKEEPER-1604.001.patch against trunk revision 1672934. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2643//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2643//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2643//console This message is automatically generated.
          Hide
          Patrick Hunt added a comment -

          If you're rolling your own (not using Bigtop packaging) they you typically start from tarball anyway. Even our scripts are of limited use in most of those cases. What I've seen is folks typically start from class files (jar) and work their way up from that for their particular context.

          Show
          Patrick Hunt added a comment - If you're rolling your own (not using Bigtop packaging) they you typically start from tarball anyway. Even our scripts are of limited use in most of those cases. What I've seen is folks typically start from class files (jar) and work their way up from that for their particular context.
          Hide
          Chris Nauroth added a comment -

          Patrick Hunt, that makes a lot of sense. Speaking for myself, I can fumble through some packaging work, but I can't claim expertise in all of the variations that you mentioned.

          In Hadoop, we've made the decision to publish releases in tarball form only and defer all other packaging concerns to Bigtop. One possible additional consideration for ZooKeeper is that some users deploy it completely outside of the context of Hadoop. My own first use cases involved coordinating a distributed cache update and putting a distributed lock around some Cassandra schema modification operations that were not safe for concurrent execution at the time. Nothing else from the Bigtop ecosystem was relevant in this deployment. For those kinds of users, perhaps it's valuable for ZooKeeper to have its own release cadence with packaging support, decoupled from the Bigtop release cadence. It's definitely a trade-off though because of the maintenance costs.

          I'm attaching a patch that removes the ant rpm and deb targets and all supporting properties and files. Between this and ZOOKEEPER-2124, I think one will need to get committed and the other will be a no-fix. Alternatively, if this one goes in on trunk, then maybe ZOOKEEPER-2124 still goes into older maintenance release branches to minimize surprises for users on those branches.

          Thanks!

          Show
          Chris Nauroth added a comment - Patrick Hunt , that makes a lot of sense. Speaking for myself, I can fumble through some packaging work, but I can't claim expertise in all of the variations that you mentioned. In Hadoop, we've made the decision to publish releases in tarball form only and defer all other packaging concerns to Bigtop. One possible additional consideration for ZooKeeper is that some users deploy it completely outside of the context of Hadoop. My own first use cases involved coordinating a distributed cache update and putting a distributed lock around some Cassandra schema modification operations that were not safe for concurrent execution at the time. Nothing else from the Bigtop ecosystem was relevant in this deployment. For those kinds of users, perhaps it's valuable for ZooKeeper to have its own release cadence with packaging support, decoupled from the Bigtop release cadence. It's definitely a trade-off though because of the maintenance costs. I'm attaching a patch that removes the ant rpm and deb targets and all supporting properties and files. Between this and ZOOKEEPER-2124 , I think one will need to get committed and the other will be a no-fix. Alternatively, if this one goes in on trunk, then maybe ZOOKEEPER-2124 still goes into older maintenance release branches to minimize surprises for users on those branches. Thanks!
          Hide
          Patrick Hunt added a comment -

          I'd say the main issues were/are that the original packaging was added (prior to bigtop existing btw), it worked ok, but there were bugs. Maintaining the different package types, ensuring they work across all the distros, versions, etc... is not a simple job. Not many/any folks in this community really understand all the packaging issues. Bigtop does and their charter is to create packaging. So it would make more sense imo to handle packaging there. At this point the packaging is out of date, there are more issues and they haven't been resolved. My thought is that it would make more sense to spend the effort once, at Bigtop, rather than twice, once at Bigtop and once here.

          Show
          Patrick Hunt added a comment - I'd say the main issues were/are that the original packaging was added (prior to bigtop existing btw), it worked ok, but there were bugs. Maintaining the different package types, ensuring they work across all the distros, versions, etc... is not a simple job. Not many/any folks in this community really understand all the packaging issues. Bigtop does and their charter is to create packaging. So it would make more sense imo to handle packaging there. At this point the packaging is out of date, there are more issues and they haven't been resolved. My thought is that it would make more sense to spend the effort once, at Bigtop, rather than twice, once at Bigtop and once here.
          Hide
          Ralph Tice added a comment -

          There definitely wasn't a strong reason, and I've come around on this issue. I don't think I'm supposed to minus or plus things either, sorry.

          Show
          Ralph Tice added a comment - There definitely wasn't a strong reason, and I've come around on this issue. I don't think I'm supposed to minus or plus things either, sorry.
          Hide
          Flavio Junqueira added a comment -

          I don't think there is a resolution. Ralph Tice isn't in favor of relying only on BigTop, but frankly I don't understand if there is a strong reason behind it. If we are to maintain our own packaging, then I'd like to know how is interested in maintaining. In any case, I'll have a look at ZOOKEEPER-2124.

          Show
          Flavio Junqueira added a comment - I don't think there is a resolution. Ralph Tice isn't in favor of relying only on BigTop, but frankly I don't understand if there is a strong reason behind it. If we are to maintain our own packaging, then I'd like to know how is interested in maintaining. In any case, I'll have a look at ZOOKEEPER-2124 .
          Hide
          Chris Nauroth added a comment -

          I'm curious if there has been any resolution to this discussion. If there is a decision to maintain the packaging targets, then would someone please review ZOOKEEPER-2124? This would fix the rpm packaging for -alpha releases. (The hyphen character is illegal in an rpm version.) There were also some earlier rpm fixes discussed in an email thread that would need to be committed and credited to the original author.

          http://mail-archives.apache.org/mod_mbox/zookeeper-user/201212.mbox/%3C50D2D481.8010507@pt-consulting.eu%3E

          Show
          Chris Nauroth added a comment - I'm curious if there has been any resolution to this discussion. If there is a decision to maintain the packaging targets, then would someone please review ZOOKEEPER-2124 ? This would fix the rpm packaging for -alpha releases. (The hyphen character is illegal in an rpm version.) There were also some earlier rpm fixes discussed in an email thread that would need to be committed and credited to the original author. http://mail-archives.apache.org/mod_mbox/zookeeper-user/201212.mbox/%3C50D2D481.8010507@pt-consulting.eu%3E
          Hide
          Michi Mutsuzaki added a comment -

          Ok let's hear from other people to what their thoughts are.

          By the way, we don't actively monitor the github mirror for pull requests. Please follow the steps described here to submit your patches.

          http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

          Thanks!
          --Michi

          Show
          Michi Mutsuzaki added a comment - Ok let's hear from other people to what their thoughts are. By the way, we don't actively monitor the github mirror for pull requests. Please follow the steps described here to submit your patches. http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute Thanks! --Michi
          Hide
          Ralph Tice added a comment -

          I don't have anything against BigTop but the path to getting latest version of ZooKeeper packaged should be kept as simple as possible. Let BigTop mirror the packaging and merge in their own patches if there are Hadoop-specific use cases to support, but ideally the packaging in the ZooKeeper repository should be made generic enough to support any Hadoop use case and BigTop should simply consume the same packaging from upstream. IMO.

          Show
          Ralph Tice added a comment - I don't have anything against BigTop but the path to getting latest version of ZooKeeper packaged should be kept as simple as possible. Let BigTop mirror the packaging and merge in their own patches if there are Hadoop-specific use cases to support, but ideally the packaging in the ZooKeeper repository should be made generic enough to support any Hadoop use case and BigTop should simply consume the same packaging from upstream. IMO.
          Hide
          Michi Mutsuzaki added a comment -

          What's wrong with using BigTop?

          Show
          Michi Mutsuzaki added a comment - What's wrong with using BigTop?
          Hide
          Ralph Tice added a comment -

          -1 for this ticket and +1 for keeping packaging local to the ZK project. I would be interested in helping with the infrastructure work for automated testing of the debs/rpms.

          Show
          Ralph Tice added a comment - -1 for this ticket and +1 for keeping packaging local to the ZK project. I would be interested in helping with the infrastructure work for automated testing of the debs/rpms.
          Hide
          Johan Hillertz added a comment -

          As part of https://issues.apache.org/jira/browse/ZOOKEEPER-1708 i got pointed to this issue. I can not find the latest 3.4.5 in BigTop the latest version i find is 3.4.3. How would the BigTop be updated?

          If it is not updated at all release it can be good to keep the possibility to make a local package build.

          Show
          Johan Hillertz added a comment - As part of https://issues.apache.org/jira/browse/ZOOKEEPER-1708 i got pointed to this issue. I can not find the latest 3.4.5 in BigTop the latest version i find is 3.4.3. How would the BigTop be updated? If it is not updated at all release it can be good to keep the possibility to make a local package build.

            People

            • Assignee:
              Chris Nauroth
              Reporter:
              Patrick Hunt
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development