Bigtop
  1. Bigtop
  2. BIGTOP-1110

Define BOM for 0.8.0 release of Bigtop

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: general
    • Labels:
      None

      Description

      This JIRA is going to be a rolling list of our thinking on what BOM makes sense for Bigtop 0.8.0 singularity release.

        Issue Links

        1.
        Bump version of Hadoop to 2.2.0 Sub-task Closed Roman Shaposhnik
         
        2.
        bump hadoop version to 2.3.0 Sub-task Resolved Giridharan Kesavan
         
        3.
        bump jsvc to 1.0.15 Sub-task Resolved Konstantin Boudnik
         
        4.
        bump flume to 1.5.0.1 Sub-task Resolved Konstantin Boudnik
         
        5.
        bump pig version to 0.12.1 Sub-task Resolved Konstantin Boudnik
         
        6.
        bump hbase version to 0.98.2 Sub-task Resolved Konstantin Boudnik
         
        7.
        bump hive version to 0.12.0 Sub-task Resolved Konstantin Boudnik
         
        8.
        Include Hive 0.13 into the stack Sub-task Resolved Giridharan Kesavan
         
        9.
        bump Oozie version to 4.0.1 Sub-task Resolved Konstantin Boudnik
         
        10.
        bump Mahout version to 0.9 Sub-task Resolved Konstantin Boudnik
         
        11.
        bump Solr version to 4.6.0 Sub-task Resolved Konstantin Boudnik
         
        12.
        bump Spark version to 0.9.1 Sub-task Resolved Mark Grover
         
        13.
        Bump version of Phoenix to 4.0 Sub-task Resolved YoungWoo Kim
         
        14.
        Hbase build has to use hadoop-two.version sys. prop Sub-task Resolved Konstantin Boudnik
         
        15.
        bump Hue version to 3.6.0 Sub-task Resolved Roman Shaposhnik
         
        16.
        bump Giraph version to 1.1.0 Sub-task Resolved Roman Shaposhnik
         
        17.
        On a clean ~/.m2 hbase mvn install site will fail as install is executed first Sub-task Resolved Konstantin Boudnik
         
        18.
        Bump version of Crunch to 0.10.0 Sub-task Resolved Mark Grover
         
        19.
        bump version of Solr to 4.9.0 Sub-task Resolved Roman Shaposhnik
         
        20.
        Bump version of Hadoop to 2.4.1 Sub-task Resolved Roman Shaposhnik
         
        21.
        Update HBase version to 0.98.5 in the BOM Sub-task Resolved Andrew Purtell
         
        22.
        Fix Hive build after BIGTOP-1429 Sub-task Resolved Andrew Purtell
         
        23.
        Fix Crunch build after BIGTOP-1429 Sub-task Resolved Konstantin Boudnik
         

          Activity

          Hide
          Konstantin Boudnik added a comment -

          And now all BOM related tickets are fixed, hence the umbrella JIRA can be resolved. We have all-blue RC build here
          http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/4/

          Show
          Konstantin Boudnik added a comment - And now all BOM related tickets are fixed, hence the umbrella JIRA can be resolved. We have all-blue RC build here http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/4/
          Hide
          Andrew Purtell added a comment -

          Patches on BIGTOP-1420, BIGTOP-1421, and BIGTOP-1432. I am now testing a 'gradle deb' build after blowing away .m2/ on an ubuntu box with master plus these three patches applied.

          Show
          Andrew Purtell added a comment - Patches on BIGTOP-1420 , BIGTOP-1421 , and BIGTOP-1432 . I am now testing a 'gradle deb' build after blowing away .m2/ on an ubuntu box with master plus these three patches applied.
          Hide
          Konstantin Boudnik added a comment -

          Could you please provide all needed patches to resolve the issue? Thanks!

          Show
          Konstantin Boudnik added a comment - Could you please provide all needed patches to resolve the issue? Thanks!
          Hide
          Andrew Purtell added a comment -

          Sure, make it .5 instead of .6 and Phoenix 4.1.0 is still good.

          Show
          Andrew Purtell added a comment - Sure, make it .5 instead of .6 and Phoenix 4.1.0 is still good.
          Hide
          Konstantin Boudnik added a comment -

          I will start branch-0.8 to work on the RC today. If the agreement will be to bump the version higher - we'll commit it to the release branch and then merge the change back to master. This way no one is blocked on anything.

          Show
          Konstantin Boudnik added a comment - I will start branch-0.8 to work on the RC today. If the agreement will be to bump the version higher - we'll commit it to the release branch and then merge the change back to master. This way no one is blocked on anything.
          Hide
          Konstantin Boudnik added a comment -

          I converted BIGTOP-1432 to a subtask of this one to reflect the fact they are related.
          Will BIGTOP-1420 be solved with hbase 0.98.5? The reason I am asking is that 0.98.6 planned for Sep 10th and I want to make a release candidate for 0.8 over the weekend.

          Show
          Konstantin Boudnik added a comment - I converted BIGTOP-1432 to a subtask of this one to reflect the fact they are related. Will BIGTOP-1420 be solved with hbase 0.98.5? The reason I am asking is that 0.98.6 planned for Sep 10th and I want to make a release candidate for 0.8 over the weekend.
          Hide
          Andrew Purtell added a comment -

          Reopening.

          A third party filed BIGTOP-1420. If we accept this change, then we also need BIGTOP-1432. They change the BOM for 0.8.

          If the BOM for 0.8 is settled then we should push these issues to 0.9 and re-resolve this.

          If the changes proposed in those issues are ok for 0.8, getting them in should be relatively painless (famous last words).

          Show
          Andrew Purtell added a comment - Reopening. A third party filed BIGTOP-1420 . If we accept this change, then we also need BIGTOP-1432 . They change the BOM for 0.8. If the BOM for 0.8 is settled then we should push these issues to 0.9 and re-resolve this. If the changes proposed in those issues are ok for 0.8, getting them in should be relatively painless (famous last words).
          Hide
          Konstantin Boudnik added a comment -

          Closing as the stack is formed now.

          Show
          Konstantin Boudnik added a comment - Closing as the stack is formed now.
          Hide
          Konstantin Boudnik added a comment -

          All subtasks are resolved and we have a fully green build now, so I am going to close this ticket.

          Show
          Konstantin Boudnik added a comment - All subtasks are resolved and we have a fully green build now, so I am going to close this ticket.
          Hide
          Konstantin Boudnik added a comment -

          One more proposal for the BOM improvement: shall we get rid of Whirr? I don't see any evidences that it is being used actively or at all. And it causes nothing but troubles in the CI - last build was all blue, except that Whirr keeps falling on the PermGen insufficiency. We have bumped it recently, but this didn't help.

          So, anyone wants to keep Whirr in 0.8.0? If so - please fix the build.

          Show
          Konstantin Boudnik added a comment - One more proposal for the BOM improvement: shall we get rid of Whirr? I don't see any evidences that it is being used actively or at all. And it causes nothing but troubles in the CI - last build was all blue, except that Whirr keeps falling on the PermGen insufficiency. We have bumped it recently, but this didn't help. So, anyone wants to keep Whirr in 0.8.0? If so - please fix the build.
          Hide
          Konstantin Boudnik added a comment -

          Yeah, 2.3 was ok when we started 0.8.0, but because of the complexity of the task we have lagged quite a bit. Hence, I personally would be ok with update - let's try and see how bad the switch is Rollback is always easy.

          Show
          Konstantin Boudnik added a comment - Yeah, 2.3 was ok when we started 0.8.0, but because of the complexity of the task we have lagged quite a bit. Hence, I personally would be ok with update - let's try and see how bad the switch is Rollback is always easy.
          Hide
          Roman Shaposhnik added a comment -

          I apologize for re-opening this discussion, but I'd like to propose we bump Hadoop version to 2.4. I can volunteer to handle all the issue that that may entail. Worst case scenario we'll rollback to 2.3.

          Thoughts?

          Show
          Roman Shaposhnik added a comment - I apologize for re-opening this discussion, but I'd like to propose we bump Hadoop version to 2.4. I can volunteer to handle all the issue that that may entail. Worst case scenario we'll rollback to 2.3. Thoughts?
          Hide
          Jesse Hu added a comment -

          Hi, is there a ETA for the bigtop 0.8.0 release ? I see that the build version on http://bigtop.apache.org/ is changed from 0.8.0-snapshot back to 0.7.0. Thanks.

          Show
          Jesse Hu added a comment - Hi, is there a ETA for the bigtop 0.8.0 release ? I see that the build version on http://bigtop.apache.org/ is changed from 0.8.0-snapshot back to 0.7.0. Thanks.
          Hide
          Konstantin Boudnik added a comment -

          Everything HBase seems to be ok with jdk6u45. I have also came across a couple of trans.deps issues - HBASE-11422 and HBASE-11418 - but they do not directly affect the BOM in question.

          Show
          Konstantin Boudnik added a comment - Everything HBase seems to be ok with jdk6u45. I have also came across a couple of trans.deps issues - HBASE-11422 and HBASE-11418 - but they do not directly affect the BOM in question.
          Hide
          Andrew Purtell added a comment -
          apurtell@acer:~/src/hbase$ export JAVA_HOME=/usr/java/jdk1.6.0_45
          apurtell@acer:~/src/hbase$ export PATH=$JAVA_HOME/bin:$PATH
          apurtell@acer:~/src/hbase$ java -version
          java version "1.6.0_45"
          Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
          Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
          apurtell@acer:~/src/hbase$ git branch
            0.94
          * 0.98
            master
          apurtell@acer:~/src/hbase$ mvn -DskipTests clean install 
          [...]
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 01:09 min
          [INFO] Finished at: 2014-06-25T14:53:02-08:00
          [INFO] Final Memory: 76M/734M
          
          Show
          Andrew Purtell added a comment - apurtell@acer:~/src/hbase$ export JAVA_HOME=/usr/java/jdk1.6.0_45 apurtell@acer:~/src/hbase$ export PATH=$JAVA_HOME/bin:$PATH apurtell@acer:~/src/hbase$ java -version java version "1.6.0_45" Java(TM) SE Runtime Environment (build 1.6.0_45-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode) apurtell@acer:~/src/hbase$ git branch 0.94 * 0.98 master apurtell@acer:~/src/hbase$ mvn -DskipTests clean install [...] [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:09 min [INFO] Finished at: 2014-06-25T14:53:02-08:00 [INFO] Final Memory: 76M/734M
          Hide
          Andrew Purtell added a comment -

          Isn't the latest 6u45?

          Show
          Andrew Purtell added a comment - Isn't the latest 6u45?
          Hide
          Roman Shaposhnik added a comment -

          We're definitely using Oracle JDK 6u20. See the following line in the latest build log:

          JAVA_HOME=/mnt/jenkins/toolchain/JDK6u20-64bit
          

          http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-HBase/618/

          Show
          Roman Shaposhnik added a comment - We're definitely using Oracle JDK 6u20. See the following line in the latest build log: JAVA_HOME=/mnt/jenkins/toolchain/JDK6u20-64bit http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-HBase/618/
          Hide
          Andrew Purtell added a comment -
          Show
          Andrew Purtell added a comment - See HBASE-11297
          Hide
          Andrew Purtell added a comment -

          Ah. IIRC that's a bug that only appears in OpenJDK 6. If you use Oracle Java 6 the code will compile.

          We are separately having a discussion were we may drop support for 6 in 0.98 and allow introduction of use of a concurrency package class only available in the 7 runtime.

          Show
          Andrew Purtell added a comment - Ah. IIRC that's a bug that only appears in OpenJDK 6. If you use Oracle Java 6 the code will compile. We are separately having a discussion were we may drop support for 6 in 0.98 and allow introduction of use of a concurrency package class only available in the 7 runtime.
          Hide
          Roman Shaposhnik added a comment -

          Andrew Purtell The failure you're seeing is a result of me forcing a version of JDK7 that's only available on Ubuntu. This fixed things on Ubuntu (so we now know that JDK7 works with HBase) but I need to change the build to go back to the JDK6.

          I removed the hack from the build specification (which got us back to JDK6) and the builds is failing as before:
          http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-HBase/617/

          At this point, it seems to be pretty reproducible – boot up a clean Ubuntu 14.04 slave, install JDK6 on it and try building HBase the same way that our build does – failure.

          Show
          Roman Shaposhnik added a comment - Andrew Purtell The failure you're seeing is a result of me forcing a version of JDK7 that's only available on Ubuntu. This fixed things on Ubuntu (so we now know that JDK7 works with HBase) but I need to change the build to go back to the JDK6. I removed the hack from the build specification (which got us back to JDK6) and the builds is failing as before: http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-HBase/617/ At this point, it seems to be pretty reproducible – boot up a clean Ubuntu 14.04 slave, install JDK6 on it and try building HBase the same way that our build does – failure.
          Hide
          Andrew Purtell added a comment -

          I've looked more into this HBase compilation issue and this looks awfully close to HBASE-8479. It seems that we need to start using JDK7, at least for the compilation.

          We are going to run the RC vote for 0.98.4 starting next Monday. There's time to get a patch in to fix this beforehand.

          I went looking at the Bigtop-trunk-HBase http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-HBase/label=centos6/616/consoleFull and it looks like a build environment issue with Maven:

          05:08:18  + mvn -DskipTests -Dslf4j.version=1.6.1 -Dhadoop-two.version=2.3.0 -Dzookeeper.version=3.4.5 clean install
          05:08:18  Error: JAVA_HOME is not defined correctly.
          05:08:18    We cannot execute /usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/java
          

          I could have easily gone to the wrong place. Can someone post an observed compilation failure here or over on an HBase JIRA?

          Show
          Andrew Purtell added a comment - I've looked more into this HBase compilation issue and this looks awfully close to HBASE-8479 . It seems that we need to start using JDK7, at least for the compilation. We are going to run the RC vote for 0.98.4 starting next Monday. There's time to get a patch in to fix this beforehand. I went looking at the Bigtop-trunk-HBase http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-HBase/label=centos6/616/consoleFull and it looks like a build environment issue with Maven: 05:08:18 + mvn -DskipTests -Dslf4j.version=1.6.1 -Dhadoop-two.version=2.3.0 -Dzookeeper.version=3.4.5 clean install 05:08:18 Error: JAVA_HOME is not defined correctly. 05:08:18 We cannot execute /usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/java I could have easily gone to the wrong place. Can someone post an observed compilation failure here or over on an HBase JIRA?
          Hide
          Mark Grover added a comment -

          My personal opinion is that it may be a non-trivial amount for some components to build on JDK7 (see HIVE-4583, for example). I have had some experience looking into that for Hive and while things may have gotten better in later releases, my dated knowledge seems to suggest that it's not something we can do overnight.

          We should move to Java7, I just don't think Bigtop 0.8 is the right timeline for that, unless it's impossible to build HBase without it.

          Show
          Mark Grover added a comment - My personal opinion is that it may be a non-trivial amount for some components to build on JDK7 (see HIVE-4583 , for example). I have had some experience looking into that for Hive and while things may have gotten better in later releases, my dated knowledge seems to suggest that it's not something we can do overnight. We should move to Java7, I just don't think Bigtop 0.8 is the right timeline for that, unless it's impossible to build HBase without it.
          Hide
          Konstantin Boudnik added a comment -

          I think JDK7 has changed things for better (per an offline chat with Roman Shaposhnik). So, I think moving away from JKD6 might be a good idea for the 0.8.0. Thoughts?

          Show
          Konstantin Boudnik added a comment - I think JDK7 has changed things for better (per an offline chat with Roman Shaposhnik ). So, I think moving away from JKD6 might be a good idea for the 0.8.0. Thoughts?
          Hide
          Mark Grover added a comment - - edited

          Wow, that sounds bad. I just updated the patch on BIGTOP-1218 for the puppet code for installing JDK7. Perhaps, once that gets in, we can try compiling on Java 7 and seeing if that changes anything?

          Show
          Mark Grover added a comment - - edited Wow, that sounds bad. I just updated the patch on BIGTOP-1218 for the puppet code for installing JDK7. Perhaps, once that gets in, we can try compiling on Java 7 and seeing if that changes anything?
          Hide
          Konstantin Boudnik added a comment -

          Roman Shaposhnik, I've looked more into this HBase compilation issue and this looks awfully close to HBASE-8479
          It seems that we need to start using JDK7, at least for the compilation. What do you think?

          Show
          Konstantin Boudnik added a comment - Roman Shaposhnik , I've looked more into this HBase compilation issue and this looks awfully close to HBASE-8479 It seems that we need to start using JDK7, at least for the compilation. What do you think?
          Hide
          Mark Grover added a comment -

          I looked at the crunch failure and looks like Crunch 0.10* is the first release of crunch that supports HBase 0.98+ so we are going to have to update Crunch to a later release.

          See https://issues.apache.org/jira/browse/CRUNCH-386 for details.

          I will take care of that.

          Show
          Mark Grover added a comment - I looked at the crunch failure and looks like Crunch 0.10* is the first release of crunch that supports HBase 0.98+ so we are going to have to update Crunch to a later release. See https://issues.apache.org/jira/browse/CRUNCH-386 for details. I will take care of that.
          Hide
          Roman Shaposhnik added a comment -

          Konstantin Boudnik basically now that our build slaves are unscrewed the nightly trunk builds are a pretty reliable indicator of what's broken:
          http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/

          Show
          Roman Shaposhnik added a comment - Konstantin Boudnik basically now that our build slaves are unscrewed the nightly trunk builds are a pretty reliable indicator of what's broken: http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/
          Hide
          Konstantin Boudnik added a comment -

          Roman Shaposhnik, could you be more specific about the compilation issues? Perhaps a URL to the logs or something?
          HBase is constantly builds fine in my case. Pig has an issue with Forrest (logged as a subtask); Oozie doesn't compile without Pig.

          Show
          Konstantin Boudnik added a comment - Roman Shaposhnik , could you be more specific about the compilation issues? Perhaps a URL to the logs or something? HBase is constantly builds fine in my case. Pig has an issue with Forrest (logged as a subtask); Oozie doesn't compile without Pig.
          Hide
          Roman Shaposhnik added a comment -

          Konstantin Boudnik actually, now that I unscrewed our build slaves we've got more problems

          HBase, Pig, Oozie and a few other are now breaking (legitimately in compilation). We also need to look into those.

          Show
          Roman Shaposhnik added a comment - Konstantin Boudnik actually, now that I unscrewed our build slaves we've got more problems HBase, Pig, Oozie and a few other are now breaking (legitimately in compilation). We also need to look into those.
          Hide
          Konstantin Boudnik added a comment -

          I see 3 tickets left on this JIRA: Phoenix, Hue, and Giraph. Any timelines from the folks assigned to those?

          Show
          Konstantin Boudnik added a comment - I see 3 tickets left on this JIRA: Phoenix, Hue, and Giraph. Any timelines from the folks assigned to those?
          Hide
          Mark Grover added a comment -

          Indeed, same page. Thanks, Cos!

          Show
          Mark Grover added a comment - Indeed, same page. Thanks, Cos!
          Hide
          Konstantin Boudnik added a comment - - edited

          Good points Mark! I think we are all in agreement that adding major, potentially disruptive changes in a middle of a release is a bad idea. However, bumping say Hive 0.13 to 0.13.1 or Hbase 0.98.2 to 0.98.3 sound pretty harmless and - considering that 0.8.0 turns out to run longer that anyone wanted - it might be very sane thing to account for 'dot' bug-fix releases. Are we on the same page?

          WRT RC voting: +1 indeed. I think we'll be able to do much more as our CI infra is getting rebuilt. Unfortunately, not everyone in the community has handy CI to experiment with different versions of the stacks

          Show
          Konstantin Boudnik added a comment - - edited Good points Mark! I think we are all in agreement that adding major, potentially disruptive changes in a middle of a release is a bad idea. However, bumping say Hive 0.13 to 0.13.1 or Hbase 0.98.2 to 0.98.3 sound pretty harmless and - considering that 0.8.0 turns out to run longer that anyone wanted - it might be very sane thing to account for 'dot' bug-fix releases. Are we on the same page? WRT RC voting: +1 indeed. I think we'll be able to do much more as our CI infra is getting rebuilt. Unfortunately, not everyone in the community has handy CI to experiment with different versions of the stacks
          Hide
          Mark Grover added a comment -

          Fine by me, especially if someone is willing to take up the work!

          But, in general, I think we face this question all the time. For example, Spark 1.0.0 has been released and there was a discussion about whether we should bring in Spark 1.0.0 for Bigtop 0.8. So, I think it would be good to instantiate a policy around how close to the projected release date can we bump up versions. Likely, that policy would need to depend on how complicated the bump is. For example, Spark 0.9->Spark 1.0.0 is a non-trivial bump since it included a new history server and some changes to the distribution building step, along with non-compatible API changes (which don't really affect us from build/packaging perspective, but could involve some test code updates) so it doesn't make sense to bump it this late in the game. However, HBase 0.98.3 sounds benign from an integration perspective so it may be fine to bump it at this point but it would really depend on what we find once we kick off a new build.

          On that note, I would really like us to play a larger role in RC voting. For example, I did that for Crunch 0.10 recently and was able to find out a problem with the RC, thanks to Bigtop's customizability to newer RCs. I think it'd save us a lot of pain later in the game, make our work more spread out across the year instead of getting overly busy during release times Thoughts?

          Show
          Mark Grover added a comment - Fine by me, especially if someone is willing to take up the work! But, in general, I think we face this question all the time. For example, Spark 1.0.0 has been released and there was a discussion about whether we should bring in Spark 1.0.0 for Bigtop 0.8. So, I think it would be good to instantiate a policy around how close to the projected release date can we bump up versions. Likely, that policy would need to depend on how complicated the bump is. For example, Spark 0.9->Spark 1.0.0 is a non-trivial bump since it included a new history server and some changes to the distribution building step, along with non-compatible API changes (which don't really affect us from build/packaging perspective, but could involve some test code updates) so it doesn't make sense to bump it this late in the game. However, HBase 0.98.3 sounds benign from an integration perspective so it may be fine to bump it at this point but it would really depend on what we find once we kick off a new build. On that note, I would really like us to play a larger role in RC voting. For example, I did that for Crunch 0.10 recently and was able to find out a problem with the RC, thanks to Bigtop's customizability to newer RCs. I think it'd save us a lot of pain later in the game, make our work more spread out across the year instead of getting overly busy during release times Thoughts?
          Hide
          Konstantin Boudnik added a comment -

          Looks like HBase 0.98.3 is out (thanks to Andrew Purtell for RM'ing it). Here's the list of bug fixes, etc.
          https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326765
          Shall we bump the HBase version in the BOM? While delivering a number of improvements it is unlikely to give us any integration headache. Thoughts?

          Show
          Konstantin Boudnik added a comment - Looks like HBase 0.98.3 is out (thanks to Andrew Purtell for RM'ing it). Here's the list of bug fixes, etc. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326765 Shall we bump the HBase version in the BOM? While delivering a number of improvements it is unlikely to give us any integration headache. Thoughts?
          Hide
          Konstantin Boudnik added a comment -

          Good to know! I was contemplating to start a discussion about ZK upgrade, so now we don't need to

          Show
          Konstantin Boudnik added a comment - Good to know! I was contemplating to start a discussion about ZK upgrade, so now we don't need to
          Hide
          Andrew Purtell added a comment -

          Just want to document it here since it came up on another issue: We tried ZooKeeper 3.4.6 for HBase (in a 0.98 release) but this release has a change that breaks their CLI (ZOOKEEPER-1897), so recommend staying with 3.4.5 until 3.4.7 is released.

          Show
          Andrew Purtell added a comment - Just want to document it here since it came up on another issue: We tried ZooKeeper 3.4.6 for HBase (in a 0.98 release) but this release has a change that breaks their CLI ( ZOOKEEPER-1897 ), so recommend staying with 3.4.5 until 3.4.7 is released.
          Hide
          Konstantin Boudnik added a comment -

          Well, technically speaking this is the god damned Hive that is narrowing down to singularity, clearly ;(

          Mark, do you guys have a working packaging for Hive 0.13 yet? That'd be of a great help, I reckon!

          Show
          Konstantin Boudnik added a comment - Well, technically speaking this is the god damned Hive that is narrowing down to singularity, clearly ;( Mark, do you guys have a working packaging for Hive 0.13 yet? That'd be of a great help, I reckon!
          Hide
          Andrew Purtell added a comment -

          Yes you have encountered the HBase singularity and crossed the event horizon.

          Show
          Andrew Purtell added a comment - Yes you have encountered the HBase singularity and crossed the event horizon.
          Hide
          Mark Grover added a comment -

          I think you are right and I think Hive 0.13 is our only option.

          Show
          Mark Grover added a comment - I think you are right and I think Hive 0.13 is our only option.
          Hide
          Konstantin Boudnik added a comment -

          Here's a bad news gentlemen: Hive 0.12 hbase-handler is incompatible with HBase 0.98. No amount of build tweaking produces a successful build. What it bolts down to is this:

              [javac] /home/cos/work/bigtop/build/hive/rpm/BUILD/hive-0.12.0/src/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java:558: incompatible types
              [javac] found   : java.lang.Class<org.apache.hadoop.hbase.client.Put>
              [javac] required: java.lang.Class<? extends org.apache.hadoop.io.Writable>
              [javac]     return Put.class;
              [javac]               ^
              [javac] /home/cos/work/bigtop/build/hive/rpm/BUILD/hive-0.12.0/src/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java:608: incompatible types
              [javac] found   : org.apache.hadoop.hbase.client.Put
              [javac] required: org.apache.hadoop.io.Writable
              [javac]     return put;
              [javac]            ^
              [javac] /home/cos/work/bigtop/build/hive/rpm/BUILD/hive-0.12.0/src/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java:239: cannot find symbol
              [javac] symbol  : method copyWritable(org.apache.hadoop.hbase.client.Result,org.apache.hadoop.hbase.client.Result)
              [javac] location: class org.apache.hadoop.hbase.util.Writables
              [javac]             Writables.copyWritable(recordReader.getCurrentValue(), value);
              [javac]                      ^
          

          So, it might be a good point to consider Hive 0.13. Any input?

          Show
          Konstantin Boudnik added a comment - Here's a bad news gentlemen: Hive 0.12 hbase-handler is incompatible with HBase 0.98. No amount of build tweaking produces a successful build. What it bolts down to is this: [javac] /home/cos/work/bigtop/build/hive/rpm/BUILD/hive-0.12.0/src/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java:558: incompatible types [javac] found : java.lang. Class <org.apache.hadoop.hbase.client.Put> [javac] required: java.lang. Class <? extends org.apache.hadoop.io.Writable> [javac] return Put.class; [javac] ^ [javac] /home/cos/work/bigtop/build/hive/rpm/BUILD/hive-0.12.0/src/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java:608: incompatible types [javac] found : org.apache.hadoop.hbase.client.Put [javac] required: org.apache.hadoop.io.Writable [javac] return put; [javac] ^ [javac] /home/cos/work/bigtop/build/hive/rpm/BUILD/hive-0.12.0/src/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java:239: cannot find symbol [javac] symbol : method copyWritable(org.apache.hadoop.hbase.client.Result,org.apache.hadoop.hbase.client.Result) [javac] location: class org.apache.hadoop.hbase.util.Writables [javac] Writables.copyWritable(recordReader.getCurrentValue(), value); [javac] ^ So, it might be a good point to consider Hive 0.13. Any input?
          Hide
          Roman Shaposhnik added a comment -

          Konstantin Boudnik this coming week, I'll try to work on Giraph release and Hue. Stay tuned.

          Show
          Roman Shaposhnik added a comment - Konstantin Boudnik this coming week, I'll try to work on Giraph release and Hue. Stay tuned.
          Hide
          Konstantin Boudnik added a comment -

          Thanks for the reviews Roman Shaposhnik! We are close people!

          Show
          Konstantin Boudnik added a comment - Thanks for the reviews Roman Shaposhnik ! We are close people!
          Hide
          Konstantin Boudnik added a comment -

          Any takers to review tasks in Patch Available state?

          Show
          Konstantin Boudnik added a comment - Any takers to review tasks in Patch Available state?
          Hide
          Konstantin Boudnik added a comment -

          No worries dude. I'd appreciate reviews as usual. And if people have a bit of spare cycles - please contribute to the release. I know that quite a bit of people are looking for new Hadoop 2.3 based stack out there. And if you're one of them and reading this - please join effort!

          Show
          Konstantin Boudnik added a comment - No worries dude. I'd appreciate reviews as usual. And if people have a bit of spare cycles - please contribute to the release. I know that quite a bit of people are looking for new Hadoop 2.3 based stack out there. And if you're one of them and reading this - please join effort!
          Hide
          Roman Shaposhnik added a comment -

          Konstantin Boudnik btw, I'd be more than happy to help with the release as much as I can, but with extra load of the book that I need to finish there's not much chance I can be relied on for driving anything.

          I guess what I'm trying to say is this: thanks a million for picking up this thread!

          Show
          Roman Shaposhnik added a comment - Konstantin Boudnik btw, I'd be more than happy to help with the release as much as I can, but with extra load of the book that I need to finish there's not much chance I can be relied on for driving anything. I guess what I'm trying to say is this: thanks a million for picking up this thread!
          Hide
          Roman Shaposhnik added a comment -

          Konstantin Boudnik I'll double down on Giraph 1.1.0 shortly. I hope I can push it out in two weeks or so.

          Show
          Roman Shaposhnik added a comment - Konstantin Boudnik I'll double down on Giraph 1.1.0 shortly. I hope I can push it out in two weeks or so.
          Hide
          Andrew Purtell added a comment -

          After bumping HBase to 0.98.x, we should ship Phoenix 4.x, the released version meant to run on that branch of HBase.

          Show
          Andrew Purtell added a comment - After bumping HBase to 0.98.x, we should ship Phoenix 4.x, the released version meant to run on that branch of HBase.
          Hide
          Konstantin Boudnik added a comment -

          Crunch 0.9 build is broken on missing dependency org.apache.hbase:hbase-shell:jar:0.98.1

          Show
          Konstantin Boudnik added a comment - Crunch 0.9 build is broken on missing dependency org.apache.hbase:hbase-shell:jar:0.98.1
          Hide
          Konstantin Boudnik added a comment -

          we might consider bumping up ZK to 3.4.6 perhaps as Hbase has rebased on top it.
          However, Hadoop is still on 3.4.5

          Show
          Konstantin Boudnik added a comment - we might consider bumping up ZK to 3.4.6 perhaps as Hbase has rebased on top it. However, Hadoop is still on 3.4.5
          Hide
          Konstantin Boudnik added a comment - - edited

          Can anyone familiar for HUE take a look at adding 3.5 into the release? I guess Sean or Mark would know the best?

          Hue 3.5 seems to have a bunch of new system dependencies that aren't expressed in the bigtop_toolchain, so it might need to be updated as well.

          Show
          Konstantin Boudnik added a comment - - edited Can anyone familiar for HUE take a look at adding 3.5 into the release? I guess Sean or Mark would know the best? Hue 3.5 seems to have a bunch of new system dependencies that aren't expressed in the bigtop_toolchain, so it might need to be updated as well.
          Hide
          Konstantin Boudnik added a comment -

          Pinging Roman Shaposhnik: what's the deal with Giraph 1.1.0 release? Any updates on that front? Looks like the official release hadn't happened yet?

          Show
          Konstantin Boudnik added a comment - Pinging Roman Shaposhnik : what's the deal with Giraph 1.1.0 release? Any updates on that front? Looks like the official release hadn't happened yet?
          Hide
          Konstantin Boudnik added a comment -

          BTW, Oozie 4.1.0 isn't available as far as I can see. Does anyone know the story about the next release?

          Show
          Konstantin Boudnik added a comment - BTW, Oozie 4.1.0 isn't available as far as I can see. Does anyone know the story about the next release?
          Hide
          Konstantin Boudnik added a comment -

          Guys, I am moving forward with the release - any help is appreciated: contributions, reviews, etc.

          Show
          Konstantin Boudnik added a comment - Guys, I am moving forward with the release - any help is appreciated: contributions, reviews, etc.
          Hide
          Andrew Purtell added a comment -

          Cross compiling for 1.6 with 1.7 does seem to work by specifying the target switch alone but this can mask issues with the JRE. It should work based on my testing, but to be 100% sure of success when cross compiling, specify to the compiler a boot classpath pointing at the 1.6 JRE.

          Show
          Andrew Purtell added a comment - Cross compiling for 1.6 with 1.7 does seem to work by specifying the target switch alone but this can mask issues with the JRE. It should work based on my testing, but to be 100% sure of success when cross compiling, specify to the compiler a boot classpath pointing at the 1.6 JRE.
          Hide
          Konstantin Boudnik added a comment -
          1. Yes, whatever is the latest stable update
          2. Not necessarily in my opinion. Sticking to target=1.6 seems to be reasonable step: I don't want to make 0.8.0 to be non-runnable on JRE6
          Show
          Konstantin Boudnik added a comment - Yes, whatever is the latest stable update Not necessarily in my opinion. Sticking to target=1.6 seems to be reasonable step: I don't want to make 0.8.0 to be non-runnable on JRE6
          Hide
          Guo Ruijing added a comment -

          for JDK7/OpenJDK7, I am wondering:

          1) what version of JDK7? jdk1.7u45?
          2) does it mean bytecode is JDK7?

          Show
          Guo Ruijing added a comment - for JDK7/OpenJDK7, I am wondering: 1) what version of JDK7? jdk1.7u45? 2) does it mean bytecode is JDK7?
          Hide
          Peter Linnell added a comment -

          BIGTOP-1241 is now re-assigned to me and is progress.

          Show
          Peter Linnell added a comment - BIGTOP-1241 is now re-assigned to me and is progress.
          Hide
          Wenwu Peng added a comment -

          Hi Peter,

          Thanks for you effort.
          do you think BIGTOP-1241(BIGTOP should update protobuf to 2.5 from 2.4.x ) whether can closed except update the doc about protobuf version?

          Show
          Wenwu Peng added a comment - Hi Peter, Thanks for you effort. do you think BIGTOP-1241 (BIGTOP should update protobuf to 2.5 from 2.4.x ) whether can closed except update the doc about protobuf version?
          Hide
          Peter Linnell added a comment -

          http://download.opensuse.org/repositories/home:/mrdocs:/protobuf-rpm/ here is where the bits will land when compiled and packaged for rpm.

          Show
          Peter Linnell added a comment - http://download.opensuse.org/repositories/home:/mrdocs:/protobuf-rpm/ here is where the bits will land when compiled and packaged for rpm.
          Hide
          Peter Linnell added a comment -

          I've got 2.5.0 building in my repo for rpm for Fedora 19 and 20, RHEL/CentOS 5+ , openSUSE 12.3+ and SLES 11. Give me a couple of days to tackle the .debs.

          Show
          Peter Linnell added a comment - I've got 2.5.0 building in my repo for rpm for Fedora 19 and 20, RHEL/CentOS 5+ , openSUSE 12.3+ and SLES 11. Give me a couple of days to tackle the .debs.
          Hide
          Mark Grover added a comment -

          Sent.

          Show
          Mark Grover added a comment - Sent.
          Hide
          Konstantin Boudnik added a comment -

          Great! Thanks Mark!

          Show
          Konstantin Boudnik added a comment - Great! Thanks Mark!
          Hide
          Mark Grover added a comment -

          ok, let me send out an email to flume dev list then, so we can have this release of flume sooner than later.

          Show
          Mark Grover added a comment - ok, let me send out an email to flume dev list then, so we can have this release of flume sooner than later .
          Hide
          Konstantin Boudnik added a comment -

          Yup, that's the only way as I see it.

          Show
          Konstantin Boudnik added a comment - Yup, that's the only way as I see it.
          Hide
          Mark Grover added a comment -

          The last release of flume was 1.4.0 in July 2013 that relies on the old protobuf. Since we don't patch components in Bigtop, if I am getting this right, this translates to we will have to bundle a new (hopefully soon to be released) release of flume that's compatible with Hadoop 2.1.0-beta+ (i.e. protobuf 2.5.0).
          Anyone disagree?

          Show
          Mark Grover added a comment - The last release of flume was 1.4.0 in July 2013 that relies on the old protobuf. Since we don't patch components in Bigtop, if I am getting this right, this translates to we will have to bundle a new (hopefully soon to be released) release of flume that's compatible with Hadoop 2.1.0-beta+ (i.e. protobuf 2.5.0). Anyone disagree?
          Hide
          Andrew Purtell added a comment - - edited

          Are you saying we need to move to 2.5.x to support hadoop 2.2.x +

          Components which have a Hadoop dependency and also include generated protobuf code should regenerate with Hadoop's version of the compiler, and pull in the same version of the runtime support library as Hadoop. Otherwise there can be problems manifesting as Java linkage errors or runtime exceptions from the protobuf library. What you get depends on which version of protobuf comes first in classpath roulette.

          Show
          Andrew Purtell added a comment - - edited Are you saying we need to move to 2.5.x to support hadoop 2.2.x + Components which have a Hadoop dependency and also include generated protobuf code should regenerate with Hadoop's version of the compiler, and pull in the same version of the runtime support library as Hadoop. Otherwise there can be problems manifesting as Java linkage errors or runtime exceptions from the protobuf library. What you get depends on which version of protobuf comes first in classpath roulette.
          Hide
          Guo Ruijing added a comment -

          yes. we need to move 2.5.x to support hadoop 2.2.x +.

          if we move to 2.5.x, flume 1.4.0 will break. Flume-2172 is to fix the issue.

          Show
          Guo Ruijing added a comment - yes. we need to move 2.5.x to support hadoop 2.2.x +. if we move to 2.5.x, flume 1.4.0 will break. Flume-2172 is to fix the issue.
          Hide
          Peter Linnell added a comment -

          I do not understand the comment about protobuf. Are you saying we need to move to 2.5.x to support hadoop 2.2.x + ? If so, I'll be glad to update my repo, but I need some time to make sure it builds for all the supported distros.

          Show
          Peter Linnell added a comment - I do not understand the comment about protobuf. Are you saying we need to move to 2.5.x to support hadoop 2.2.x + ? If so, I'll be glad to update my repo, but I need some time to make sure it builds for all the supported distros.
          Hide
          Konstantin Boudnik added a comment -

          As commented on that JIRA I am not enthusiastic about it

          Show
          Konstantin Boudnik added a comment - As commented on that JIRA I am not enthusiastic about it
          Hide
          Guo Ruijing added a comment - - edited

          I propose to update BOM to reflect the inclusion of flume 1.4.0 + Flume-2172

          Show
          Guo Ruijing added a comment - - edited I propose to update BOM to reflect the inclusion of flume 1.4.0 + Flume-2172
          Hide
          Konstantin Boudnik added a comment -

          Sounds like a good idea to update jsvc

          Show
          Konstantin Boudnik added a comment - Sounds like a good idea to update jsvc
          Hide
          Wenwu Peng added a comment -

          Hadoop update protobuf to 2.5 from 2.4.X later version 2.2.0

          Show
          Wenwu Peng added a comment - Hadoop update protobuf to 2.5 from 2.4.X later version 2.2.0
          Hide
          Guo Ruijing added a comment - - edited

          I propose to update BOM to reflect the inclusion of jsvc 1.0.15 (as commented on https://issues.apache.org/jira/browse/BIGTOP-1242)

          Show
          Guo Ruijing added a comment - - edited I propose to update BOM to reflect the inclusion of jsvc 1.0.15 (as commented on https://issues.apache.org/jira/browse/BIGTOP-1242 )
          Hide
          Konstantin Boudnik added a comment -

          I have update BOM to reflect the inclusion of Hadoop 2.3.0 (as commented on BIGTOP-1184)

          Show
          Konstantin Boudnik added a comment - I have update BOM to reflect the inclusion of Hadoop 2.3.0 (as commented on BIGTOP-1184 )
          Hide
          Andrew Purtell added a comment -

          No, as I said above 2.3.0 is promising, when released. Mainly I wanted to start working on the 0.8 branch with an existing release to move things along. I can wait.

          Show
          Andrew Purtell added a comment - No, as I said above 2.3.0 is promising, when released. Mainly I wanted to start working on the 0.8 branch with an existing release to move things along. I can wait.
          Hide
          Roman Shaposhnik added a comment -

          Andrew Purtell any reason not to go with 2.3.0? The results of the RC testing I did look promising.

          Show
          Roman Shaposhnik added a comment - Andrew Purtell any reason not to go with 2.3.0? The results of the RC testing I did look promising.
          Hide
          Andrew Purtell added a comment -

          So to clarify the Hadoop version we are using, we have a patch already on BIGTOP-1184 that bumps it to 2.2.0. Accepting 2.2.x is congruent with the discussion above on this issue. We might want to ask Hadoop to release 2.2.1 from branch-2.2.1 since it contains several important bug fixes.

          Show
          Andrew Purtell added a comment - So to clarify the Hadoop version we are using, we have a patch already on BIGTOP-1184 that bumps it to 2.2.0. Accepting 2.2.x is congruent with the discussion above on this issue. We might want to ask Hadoop to release 2.2.1 from branch-2.2.1 since it contains several important bug fixes.
          Hide
          Bruno Mahé added a comment - - edited

          I am short on time, so forgive me for the briefness:
          tl;dr: I am with Mark on that one. I don't mind deprecating CentOS 5 and Ubuntu 12.04 if 14.04 is out. But I would rather keep JDK 6 for now (maybe deprecate it for 0.9.0, but not sure).
          "deprecating" means we keep it for 0.8.0 but drop them afterward.

          • Even if Ubuntu 12.04 gets out in time for Apache Bigtop 0.8.0, I doubt it will be ready for use before a few weeks have past. It has been my experience for most releases I have been through regardless of the GNU/Linux distributions and do not see a reason to be different in that case (.0 vs .1 releases )
          • If we are to drop support so abruptly, then users don't have much incentive to use us (too unstable, demanding) and we are just going to be upstream of other distributions. Which can be fine if this is what this community wants. But I believe this is not the case.
          • There is a whole continent, or two, between suddenly dropping a jdk version along with a few GNU/Linux distributions and supporting CentOS 5 until 2020. So let's do things in between. A proposal could be to send an email first to the user and dev mailing list and ask if anyone cares about X. Then if we still want to drop it, let's deprecate X for a release so it can be dropped in the one after. This should help us keep a balance between not supporting centos 5 forever and removing stuff too fast.
          • Source artefacts can have some if/else for different versions. Also convenience artefacts are really convenient in the case of Apache Bigtop
          • Redhat did lotta work around the what they call proper packaging of Hadoop which never been contributed here

            Where is the isssue? Also note that at least for Fedora, patches must be contributed upstream. Not sure about RHEL itself.
          Show
          Bruno Mahé added a comment - - edited I am short on time, so forgive me for the briefness: tl;dr: I am with Mark on that one. I don't mind deprecating CentOS 5 and Ubuntu 12.04 if 14.04 is out. But I would rather keep JDK 6 for now (maybe deprecate it for 0.9.0, but not sure). "deprecating" means we keep it for 0.8.0 but drop them afterward. Even if Ubuntu 12.04 gets out in time for Apache Bigtop 0.8.0, I doubt it will be ready for use before a few weeks have past. It has been my experience for most releases I have been through regardless of the GNU/Linux distributions and do not see a reason to be different in that case (.0 vs .1 releases ) If we are to drop support so abruptly, then users don't have much incentive to use us (too unstable, demanding) and we are just going to be upstream of other distributions. Which can be fine if this is what this community wants. But I believe this is not the case. There is a whole continent, or two, between suddenly dropping a jdk version along with a few GNU/Linux distributions and supporting CentOS 5 until 2020. So let's do things in between. A proposal could be to send an email first to the user and dev mailing list and ask if anyone cares about X. Then if we still want to drop it, let's deprecate X for a release so it can be dropped in the one after. This should help us keep a balance between not supporting centos 5 forever and removing stuff too fast. Source artefacts can have some if/else for different versions. Also convenience artefacts are really convenient in the case of Apache Bigtop Redhat did lotta work around the what they call proper packaging of Hadoop which never been contributed here Where is the isssue? Also note that at least for Fedora, patches must be contributed upstream. Not sure about RHEL itself.
          Hide
          Konstantin Boudnik added a comment -

          some time in 2017 and in extended support till 2020. I think we all concur on the fact that Bigtop is not just a integration platform for Hadoop ecosystem but is also a distribution of the same.

          Mark, we have very limited resources and please keep in mind that our official release is a bunch of source code - not the binary packages (which are called 'convenience artifacts' for a reason). Besides, Redhat did lotta work around the what they call proper packaging of Hadoop which never been contributed here. But that aside, RHEL5 is getting an extreme PITA to deal with because it is outdated, hard to find dev tools for, etc. So, I am all in favor of dropping the dead weight and makes the life of this community easier.

          Show
          Konstantin Boudnik added a comment - some time in 2017 and in extended support till 2020. I think we all concur on the fact that Bigtop is not just a integration platform for Hadoop ecosystem but is also a distribution of the same. Mark, we have very limited resources and please keep in mind that our official release is a bunch of source code - not the binary packages (which are called 'convenience artifacts' for a reason). Besides, Redhat did lotta work around the what they call proper packaging of Hadoop which never been contributed here. But that aside, RHEL5 is getting an extreme PITA to deal with because it is outdated, hard to find dev tools for, etc. So, I am all in favor of dropping the dead weight and makes the life of this community easier.
          Hide
          Mark Grover added a comment -

          Hi Roman, thanks for driving this!

          Crunch: Great! Let's go with Crunch 0.9.0 then.

          RHEL5: I totally understand where you are coming from. I don't really have any really strong use-cases for it but in all fairness, I don't have any really strong ones against it either. My rationale comes from the fact that RHEL5 is in full production support up until some time in 2017 and in extended support till 2020. I think we all concur on the fact that Bigtop is not just a integration platform for Hadoop ecosystem but is also a distribution of the same. And, if so, how can we do justice to our users by only supporting our distribution on OSs that just came out a month ago (using Ubuntu 14.04 as an example here) to which people haven't migrated yet instead of OSs that people are actually using on their servers due to their long term support. We have to maintain a fine balance here, no doubt. It's just the case, that my personal balance errs more on the side of Bigtop being more of stable distribution instead of just being an integration bed.

          Ubuntu 12.04: I am going to use the same rationale from the above point and voice that same concern that Bruno aptly pointed out earlier. Ubuntu 12.04 is supported at least till Q1 2017, so why get rid of it? In the past, we have had 2 LTS releases supported by a release of Bigtop. Should we consider supporting both Ubuntu 12.04 and 14.04 in Bigtop 0.8?

          Thanks again!

          Show
          Mark Grover added a comment - Hi Roman, thanks for driving this! Crunch : Great! Let's go with Crunch 0.9.0 then. RHEL5 : I totally understand where you are coming from. I don't really have any really strong use-cases for it but in all fairness, I don't have any really strong ones against it either. My rationale comes from the fact that RHEL5 is in full production support up until some time in 2017 and in extended support till 2020. I think we all concur on the fact that Bigtop is not just a integration platform for Hadoop ecosystem but is also a distribution of the same. And, if so, how can we do justice to our users by only supporting our distribution on OSs that just came out a month ago (using Ubuntu 14.04 as an example here) to which people haven't migrated yet instead of OSs that people are actually using on their servers due to their long term support. We have to maintain a fine balance here, no doubt. It's just the case, that my personal balance errs more on the side of Bigtop being more of stable distribution instead of just being an integration bed. Ubuntu 12.04 : I am going to use the same rationale from the above point and voice that same concern that Bruno aptly pointed out earlier. Ubuntu 12.04 is supported at least till Q1 2017, so why get rid of it? In the past, we have had 2 LTS releases supported by a release of Bigtop. Should we consider supporting both Ubuntu 12.04 and 14.04 in Bigtop 0.8? Thanks again!
          Hide
          Andrew Purtell added a comment -

          What do you guys think about Hadoop 2.3 then? Any feedback?

          My above concern was about a looming API discontinuity, but that has been resolved in favor of not breaking a downstream project midway in the 2.x release series.

          If 2.3.0 is released with sufficient lead time for 0.8, I think we should consider it. Or 2.4.0 for that matter. We generally take the most recent releases of the stack as informal policy, to try to increase usage and test coverage of these things? Other policies are reasonable also, if I am mistaken or if we want to do things differently.

          Looking at the Fix Version report for 2.3.0: https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20HDT%2C%20MAPREDUCE%2C%20YARN)%20AND%20fixVersion%20%3D%20%222.3.0%22%20ORDER%20BY%20key%20ASC , from a distribution perspective:

          HADOOP-9623 is a dependency version change but I can't seem to get the S3 or S3N filesystems working without it.

          HADOOP-10110 moves jetty-util to test scope, that's probably fine.

          HADOOP-10255 isn't resolved yet but is the cure for the above mentioned API discontinuity.

          Looks like many NFS gateway fixes and doc updates landed here.

          There are several issues with the UIs that will be fixed.

          Looks promising, caveat enough time for testing.

          Show
          Andrew Purtell added a comment - What do you guys think about Hadoop 2.3 then? Any feedback? My above concern was about a looming API discontinuity, but that has been resolved in favor of not breaking a downstream project midway in the 2.x release series. If 2.3.0 is released with sufficient lead time for 0.8, I think we should consider it. Or 2.4.0 for that matter. We generally take the most recent releases of the stack as informal policy, to try to increase usage and test coverage of these things? Other policies are reasonable also, if I am mistaken or if we want to do things differently. Looking at the Fix Version report for 2.3.0: https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20HDT%2C%20MAPREDUCE%2C%20YARN)%20AND%20fixVersion%20%3D%20%222.3.0%22%20ORDER%20BY%20key%20ASC , from a distribution perspective: HADOOP-9623 is a dependency version change but I can't seem to get the S3 or S3N filesystems working without it. HADOOP-10110 moves jetty-util to test scope, that's probably fine. HADOOP-10255 isn't resolved yet but is the cure for the above mentioned API discontinuity. Looks like many NFS gateway fixes and doc updates landed here. There are several issues with the UIs that will be fixed. Looks promising, caveat enough time for testing.
          Hide
          Konstantin Boudnik added a comment - - edited

          Konstantin Boudnik Andrew Purtell What do you guys think about Hadoop 2.3 then? Any feedback?

          2.3 might be an option, but I think even if released it will be still bleeding at the edges when we release 0.8

          JDK6 support

          As far as I understand, Java7 is backward compatible with Java6. The latter was EOLed by the maintainer. Hence, the criteria to drop it is seemingly there. However, everyone and his uncle is using it today. I'd say dropping it would be more a time-bound event. Say until the end of 2014?

          Show
          Konstantin Boudnik added a comment - - edited Konstantin Boudnik Andrew Purtell What do you guys think about Hadoop 2.3 then? Any feedback? 2.3 might be an option, but I think even if released it will be still bleeding at the edges when we release 0.8 JDK6 support As far as I understand, Java7 is backward compatible with Java6. The latter was EOLed by the maintainer. Hence, the criteria to drop it is seemingly there. However, everyone and his uncle is using it today. I'd say dropping it would be more a time-bound event. Say until the end of 2014?
          Hide
          Roman Shaposhnik added a comment -

          Mark Grover Crunch 0.9 makes a lot of sense – I'll update the list. As for RHEL5 support – if there's something I might fight for on that proposal it is dropping support for it. So unless there's anybody with a really strong use cases for it – I suggest we drop it.

          Konstantin Boudnik Andrew Purtell What do you guys think about Hadoop 2.3 then? Any feedback?

          Bruno Mahé Since Bigtop 0.8.0 will be late April/early May we can simply go with the latest LTS at that point (14.04) unless of course, there's a strong objection.

          Konstantin Boudnik Andrew Purtell Bruno Mahé As for the JDK6 support, let me ask you this guys: even if we keep it for the Bigtop 0.8.0 BOM, what would
          be your criteria for dropping it completely? Also, would it make sense if we commit to keeping it as a build environment in Bigtop 0.8.0 but fully migrate to [Open]JDK7 as far as deployment is concerned? Thoughts?

          Show
          Roman Shaposhnik added a comment - Mark Grover Crunch 0.9 makes a lot of sense – I'll update the list. As for RHEL5 support – if there's something I might fight for on that proposal it is dropping support for it. So unless there's anybody with a really strong use cases for it – I suggest we drop it. Konstantin Boudnik Andrew Purtell What do you guys think about Hadoop 2.3 then? Any feedback? Bruno Mahé Since Bigtop 0.8.0 will be late April/early May we can simply go with the latest LTS at that point (14.04) unless of course, there's a strong objection. Konstantin Boudnik Andrew Purtell Bruno Mahé As for the JDK6 support, let me ask you this guys: even if we keep it for the Bigtop 0.8.0 BOM, what would be your criteria for dropping it completely? Also, would it make sense if we commit to keeping it as a build environment in Bigtop 0.8.0 but fully migrate to [Open] JDK7 as far as deployment is concerned? Thoughts?
          Hide
          Andrew Purtell added a comment -

          I don't think Hadoop 2.4 is a good idea.

          From the HBase point of view, I concur with this, because I have heard we will have to refactor our use of HTTP servlets in order to operate (start up) successfully on the 2.4.0 release.

          There has been some additional activity in this area including some effort to make 2.4.0 backwards compatible.

          Show
          Andrew Purtell added a comment - I don't think Hadoop 2.4 is a good idea. From the HBase point of view, I concur with this, because I have heard we will have to refactor our use of HTTP servlets in order to operate (start up) successfully on the 2.4.0 release. There has been some additional activity in this area including some effort to make 2.4.0 backwards compatible.
          Hide
          Andrew Purtell added a comment -

          +1 also on keeping JDK6 as a target. I'll be testing with 6 and 7 when evaluating a release candidate anyhow.

          Show
          Andrew Purtell added a comment - +1 also on keeping JDK6 as a target. I'll be testing with 6 and 7 when evaluating a release candidate anyhow.
          Hide
          Konstantin Boudnik added a comment -

          +1 on Bruno's first two bullet points.

          Show
          Konstantin Boudnik added a comment - +1 on Bruno's first two bullet points.
          Hide
          Andrew Purtell added a comment - - edited

          I don't think Hadoop 2.4 is a good idea.

          From the HBase point of view, I concur with this, because I have heard we will have to refactor our use of HTTP servlets in order to operate (start up) successfully on the 2.4.0 release. See HBASE-10336. No current release version of HBase does this and I'm not sure it can go in on any minor bump.

          Show
          Andrew Purtell added a comment - - edited I don't think Hadoop 2.4 is a good idea. From the HBase point of view, I concur with this, because I have heard we will have to refactor our use of HTTP servlets in order to operate (start up) successfully on the 2.4.0 release. See HBASE-10336 . No current release version of HBase does this and I'm not sure it can go in on any minor bump.
          Hide
          Bruno Mahé added a comment -
          • Why are we dropping the current ubuntu LTS? I would very much keep it until the next LTS is out
          • JDK6 is still used almost everywhere. So we should also support it
          • If Apache Flume 1.5 gets out before the coming release of Apache Bigtop, I would like to get Kite completely reviewed and in Apache Bigtop as well. Given the patch is done and as far as I can tell working. It is just about me finishing testing since Sean has done all the work already.
          Show
          Bruno Mahé added a comment - Why are we dropping the current ubuntu LTS? I would very much keep it until the next LTS is out JDK6 is still used almost everywhere. So we should also support it If Apache Flume 1.5 gets out before the coming release of Apache Bigtop, I would like to get Kite completely reviewed and in Apache Bigtop as well. Given the patch is done and as far as I can tell working. It is just about me finishing testing since Sean has done all the work already.
          Hide
          Peter Linnell added a comment -

          WRT to RHEL5, it is getting a bit long in the tooth, but that said, I think it is best to make #3 option and deprecate it in this release, meaning no RHEL5 support in 0.9. Besides RHEL7 is coming down the pike, like SLES 12.

          The challenge is python 2.x vs 3.x support when these land in the public. Not sure if this will have any impact on us in the future.

          This has a doc impact and if this is the direction people want to go, a big fat warning should go out with the release notes.

          Show
          Peter Linnell added a comment - WRT to RHEL5, it is getting a bit long in the tooth, but that said, I think it is best to make #3 option and deprecate it in this release, meaning no RHEL5 support in 0.9. Besides RHEL7 is coming down the pike, like SLES 12. The challenge is python 2.x vs 3.x support when these land in the public. Not sure if this will have any impact on us in the future. This has a doc impact and if this is the direction people want to go, a big fat warning should go out with the release notes.
          Hide
          Konstantin Boudnik added a comment -

          I don't think Hadoop 2.4 is a good idea. I think we'll have handful with 2.2 and 2.4 is sorta to bleeding to my taste.

          Show
          Konstantin Boudnik added a comment - I don't think Hadoop 2.4 is a good idea. I think we'll have handful with 2.2 and 2.4 is sorta to bleeding to my taste.
          Hide
          Mark Grover added a comment -

          Hi Roman, thanks for starting this thread!

          Here are my thoughts:
          1. Apache Crunch 0.9 has been released. As far as I can tell, Apache Crunch 0.8* does not work with HBase 0.96+ so we should definitely incorporate the latest release of Crunch, 0.9.0
          2. I am not convinced it's in the best interest of the bigtop community to get rid of RHEL5 at this point. From the small subset of population of users that I meet at conferences (for example, I presented on Bigtop at ApacheCon last year in Portland along with Roman), the impression I get is that RHEL5 is still in very prevalent use. Having said that, I am very well aware of the increased burden of RHEL5 support in some cases - the only two that come to mind right now are python related issues with Hue and Spark, both of whose proposed Bigtop 0.8 versions require Python 2.6.

          Long story short, for me, to vote for a decision to get rid of RHEL/Centos 5 support in Bigtop is not an easy one. While I think the aforementioned increased burden is relatively not a lot, I can understand while others may think it is. So, I think we have 3 alternatives:
          1. Keep the status quo. Support RHEL5.
          2. Don't support RHEL5 at all.
          3. Support RHEL5 as a deprecated platform.

          The alternatives 1 and 2 are fairly clear on their own but #3 is about a happy medium in my opinion.

          In any case, my vote here would be to keep the status quo at #1 but if the community is inclined to not do that, I'd be ok with #3 as well. I am just not convinced if now is the right time to go with #2. I would love to hear what you all have to say though! Thanks!

          Show
          Mark Grover added a comment - Hi Roman, thanks for starting this thread! Here are my thoughts: 1. Apache Crunch 0.9 has been released. As far as I can tell, Apache Crunch 0.8* does not work with HBase 0.96+ so we should definitely incorporate the latest release of Crunch, 0.9.0 2. I am not convinced it's in the best interest of the bigtop community to get rid of RHEL5 at this point. From the small subset of population of users that I meet at conferences (for example, I presented on Bigtop at ApacheCon last year in Portland along with Roman), the impression I get is that RHEL5 is still in very prevalent use. Having said that, I am very well aware of the increased burden of RHEL5 support in some cases - the only two that come to mind right now are python related issues with Hue and Spark, both of whose proposed Bigtop 0.8 versions require Python 2.6. Long story short, for me, to vote for a decision to get rid of RHEL/Centos 5 support in Bigtop is not an easy one. While I think the aforementioned increased burden is relatively not a lot, I can understand while others may think it is. So, I think we have 3 alternatives: 1. Keep the status quo. Support RHEL5. 2. Don't support RHEL5 at all. 3. Support RHEL5 as a deprecated platform. The alternatives 1 and 2 are fairly clear on their own but #3 is about a happy medium in my opinion. In any case, my vote here would be to keep the status quo at #1 but if the community is inclined to not do that, I'd be ok with #3 as well. I am just not convinced if now is the right time to go with #2. I would love to hear what you all have to say though! Thanks!
          Hide
          Roman Shaposhnik added a comment -

          Phoenix 3.0 makes a lot of sense. Andrew, would you be willing to drive that in Bigtop? Let me know and I can edit my 'BOM comment' to reflect it.

          Show
          Roman Shaposhnik added a comment - Phoenix 3.0 makes a lot of sense. Andrew, would you be willing to drive that in Bigtop? Let me know and I can edit my 'BOM comment' to reflect it.
          Hide
          Andrew Purtell added a comment -
          Show
          Andrew Purtell added a comment - See PHOENIX-8
          Hide
          Roman Shaposhnik added a comment - - edited

          A bit of time has passed since we last discussed BOM for 0.8.0. Quite a few projects moved on with releases and I'd like to propose my updated understanding of what may make sense for us. We can still tweak it somewhat, but I think it gives a good flavor for what 0.8.0 is (at least in my mind):

          OS:

          • CentOS6, CentOS7, Fedora 20
          • SLES11sp3, OpenSUSE 13.1
          • Ubuntu 14.04 LTS

          Java:

          • JDK7/OpenJDK7

          Projects:

          • Zookeeper 3.4.5
          • Hadoop 2.4.1
          • HBase 0.98.5
          • Pig 0.12.1
          • Hive 0.13
          • Sqoop 1.99.2
          • Oozie 4..0.1
          • Whirr 0.8.2
          • Mahout 0.9
          • Flume 1.5.0.1
          • Giraph 1.1.0
          • Hue 3.6.0
          • DataFU 1.0.0
          • Solr 4.6.0
          • Crunch 0.10.0
          • Spark 0.9.1
          • Phoenix 4.1.0
          • Tomcat 6.0.36
          • jsvc 1.0.15
          Show
          Roman Shaposhnik added a comment - - edited A bit of time has passed since we last discussed BOM for 0.8.0. Quite a few projects moved on with releases and I'd like to propose my updated understanding of what may make sense for us. We can still tweak it somewhat, but I think it gives a good flavor for what 0.8.0 is (at least in my mind): OS: CentOS6, CentOS7, Fedora 20 SLES11sp3, OpenSUSE 13.1 Ubuntu 14.04 LTS Java: JDK7/OpenJDK7 Projects: Zookeeper 3.4.5 Hadoop 2.4.1 HBase 0.98.5 Pig 0.12.1 Hive 0.13 Sqoop 1.99.2 Oozie 4..0.1 Whirr 0.8.2 Mahout 0.9 Flume 1.5.0.1 Giraph 1.1.0 Hue 3.6.0 DataFU 1.0.0 Solr 4.6.0 Crunch 0.10.0 Spark 0.9.1 Phoenix 4.1.0 Tomcat 6.0.36 jsvc 1.0.15
          Hide
          Roman Shaposhnik added a comment -

          Indeed too many releases to juggle.

          Show
          Roman Shaposhnik added a comment - Indeed too many releases to juggle.
          Hide
          Raymie Stata added a comment -

          Shouldn't the title be 0.8.0?

          Show
          Raymie Stata added a comment - Shouldn't the title be 0.8.0?

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development