Details

    • Type: New Feature
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: general
    • Labels:
      None

      Description

      It is this time again to start thinking about what we'd like to put into the next release, once 1.1.0 is out of the doors.

      I know we have discussed this in the past, but I wonder if trying to make a quick consequent release with just some components updates would be make sense? Obvious candidates will be Hadoop 2.7.2, Hive 1.2.2 (might be ready soon), Spark 1.6.1

      Or the community would rather spend more time and deliver upgraded components along with the improvements to our own software as well?

      Here's the proposed list

      alluxio 1.0.1 ==> 1.3.0
      ambari 2.5.0
      apex 3.4.0
      bigtop-groovy 2.4.4  ==> 2.4.10
      bigtop-jsvc 1.0.15
      bigtop-tomcat 6.0.45
      bigtop-utils 1.2.0-SNAPSHOT
      crunch 0.13.0 ==> 0.14.0
      datafu 1.0.0 ==> 1.3.0
      flink 1.0.3 ==> 1.1.3
      flume 1.7.0
      giraph 1.1.0
      gpdb 4.3.99.0  ==> 5.0.0-alpha.0
      hadoop 2.7.2 ==> we should move to 2.7.3
      hama 0.7.0
      hbase 1.1.3 ==> 1.2.4
      hive 1.2.1 ==> should we add Hive 2.1.x, as well? needs Tez 0.9.x??
      hue 3.9.0 ==> 3.11.0
      ignite-hadoop 1.5.0.final  ==> 1.9.0
      kafka 0.9.0.1 ==> 0.10.1.1
      kite 1.1.0
      mahout 0.11.1 ==> 0.12.2
      oozie 4.2.0 ==> 4.3.0
      phoenix 4.9.0-HBase-1.1
      pig 0.15.0 ==> 0.16.0
      qfs 1.1.4
      solr 4.10.4 ==> 5.2.x
      spark 2.1.0
      spark1 1.6.2
      sqoop 1.4.6
      sqoop2 1.99.4
      tajo 0.11.1
      tez 0.6.2 ==> 0.7.x
      zeppelin 0.7.0
      

        Issue Links

          Activity

          Hide
          oflebbe Olaf Flebbe added a comment - - edited

          For 1.2 I see the following goals:

          Support for Java 8 (maybe removing JDK7)
          Fedora 22
          Ubuntu 15.04 (or newer)
          opensuse leap
          Focus on testing components, rather simple compile

          Removing of centos-6, opensuse 13.2 and Fedora 20

          Show
          oflebbe Olaf Flebbe added a comment - - edited For 1.2 I see the following goals: Support for Java 8 (maybe removing JDK7) Fedora 22 Ubuntu 15.04 (or newer) opensuse leap Focus on testing components, rather simple compile Removing of centos-6, opensuse 13.2 and Fedora 20
          Hide
          cos Konstantin Boudnik added a comment -

          Focus on testing components, rather simple compile

          It has been an ongoing effort and with more development on the smoke-tests side it will be getting better.

          Show
          cos Konstantin Boudnik added a comment - Focus on testing components, rather simple compile It has been an ongoing effort and with more development on the smoke-tests side it will be getting better.
          Hide
          asanjar Amir Sanjar added a comment -

          Olaf,
          I have submitted patches to add support for Ubuntu 15.04 on x86l. Will contribute support for ubuntu 16.04 when it becomes available .
          https://issues.apache.org/jira/browse/BIGTOP-2261
          https://issues.apache.org/jira/browse/BIGTOP-2262
          https://issues.apache.org/jira/browse/BIGTOP-2263
          Regarding Fedora 22, I could own that as I have done the same for ppc.
          +1 on jdk8 support, but then we need move up to Hadoop 2.8 or backport patch https://issues.apache.org/jira/browse/HADOOP-12457

          Show
          asanjar Amir Sanjar added a comment - Olaf, I have submitted patches to add support for Ubuntu 15.04 on x86l. Will contribute support for ubuntu 16.04 when it becomes available . https://issues.apache.org/jira/browse/BIGTOP-2261 https://issues.apache.org/jira/browse/BIGTOP-2262 https://issues.apache.org/jira/browse/BIGTOP-2263 Regarding Fedora 22, I could own that as I have done the same for ppc. +1 on jdk8 support, but then we need move up to Hadoop 2.8 or backport patch https://issues.apache.org/jira/browse/HADOOP-12457
          Hide
          plinnell Peter Linnell added a comment -

          We definitely want to be removing openSUSE 13.2 and getting openSUSE Leap 42.1 as the default openSUSE platform.

          The nice part is Leap is and will be much closer in codebase to SLES 12.

          Show
          plinnell Peter Linnell added a comment - We definitely want to be removing openSUSE 13.2 and getting openSUSE Leap 42.1 as the default openSUSE platform. The nice part is Leap is and will be much closer in codebase to SLES 12.
          Hide
          cos Konstantin Boudnik added a comment -

          Should it say Leap 14.1 ?

          Show
          cos Konstantin Boudnik added a comment - Should it say Leap 14.1 ?
          Hide
          oflebbe Olaf Flebbe added a comment -

          42.1 tries to mimik SLES 12 SP1 .

          https://en.opensuse.org/Portal:42.1

          Show
          oflebbe Olaf Flebbe added a comment - 42.1 tries to mimik SLES 12 SP1 . https://en.opensuse.org/Portal:42.1
          Hide
          cos Konstantin Boudnik added a comment -

          Oh damn - that teaches me a lesson of not being lazy Thanks!

          Show
          cos Konstantin Boudnik added a comment - Oh damn - that teaches me a lesson of not being lazy Thanks!
          Hide
          asanjar Amir Sanjar added a comment -

          Below is the list of the latest versions.

          << add Ambari support - major work, might be too late >>*
          alluxio 1.0.1
          apex 3.4.0
          bigtop-groovy 2.4.4
          bigtop-jsvc 1.0.15
          bigtop-tomcat 6.0.45
          bigtop-utils 1.2.0-SNAPSHOT
          crunch 0.13.0
          datafu 1.0.0
          flink 1.0.3
          flume 1.7.0
          giraph 1.1.0
          gpdb 4.3.99.0
          hadoop 2.7.2 ==> we should move to 2.7.3
          hama 0.7.0
          hbase 1.1.3
          hive 1.2.1 ==> should we add Hive 2.1.x, as well? needs Tez 0.9.x??
          hue 3.9.0
          ignite-hadoop 1.5.0.final
          kafka 0.9.0.1 ==> 0.10.0
          kite 1.1.0
          mahout 0.11.1
          oozie 4.2.0
          phoenix 4.8.1-HBase-1.1
          pig 0.15.0 ==> 0.16.0
          qfs 1.1.4
          solr 4.10.4 ==> 5.2.x
          spark 2.0.2
          spark1 1.6.2
          sqoop 1.4.6
          sqoop2 1.99.4
          tajo 0.11.1
          tez 0.6.2 ==> 0.7.x

          Show
          asanjar Amir Sanjar added a comment - Below is the list of the latest versions. << add Ambari support - major work, might be too late >>* alluxio 1.0.1 apex 3.4.0 bigtop-groovy 2.4.4 bigtop-jsvc 1.0.15 bigtop-tomcat 6.0.45 bigtop-utils 1.2.0-SNAPSHOT crunch 0.13.0 datafu 1.0.0 flink 1.0.3 flume 1.7.0 giraph 1.1.0 gpdb 4.3.99.0 hadoop 2.7.2 ==> we should move to 2.7.3 hama 0.7.0 hbase 1.1.3 hive 1.2.1 ==> should we add Hive 2.1.x, as well? needs Tez 0.9.x ?? hue 3.9.0 ignite-hadoop 1.5.0.final kafka 0.9.0.1 ==> 0.10.0 kite 1.1.0 mahout 0.11.1 oozie 4.2.0 phoenix 4.8.1-HBase-1.1 pig 0.15.0 ==> 0.16.0 qfs 1.1.4 solr 4.10.4 ==> 5.2.x spark 2.0.2 spark1 1.6.2 sqoop 1.4.6 sqoop2 1.99.4 tajo 0.11.1 tez 0.6.2 ==> 0.7.x
          Hide
          cos Konstantin Boudnik added a comment -

          Ignite is at 1.8.0 release cycle right now. Let's bump it up as a lot of new good stuff has been contributed there, including Cassandra as KV store support. Ideally, we need to revisit the content of this package and, perhaps, start supporting the whole data fabric instead of the narrow hadoop- and spark- focused acceleration.

          Show
          cos Konstantin Boudnik added a comment - Ignite is at 1.8.0 release cycle right now. Let's bump it up as a lot of new good stuff has been contributed there, including Cassandra as KV store support. Ideally, we need to revisit the content of this package and, perhaps, start supporting the whole data fabric instead of the narrow hadoop- and spark- focused acceleration.
          Hide
          cos Konstantin Boudnik added a comment -

          A minor bump of Groovy to 2.4.7 would make sense as there was a decent amount of fixes done since 2.4.4. I will open a ticket for it.

          Show
          cos Konstantin Boudnik added a comment - A minor bump of Groovy to 2.4.7 would make sense as there was a decent amount of fixes done since 2.4.4. I will open a ticket for it.
          Hide
          jonathak Jonathan Kelly added a comment -

          Here are a few more I'd suggest:

          alluxio => 1.3.0
          flink => 1.1.3
          hbase => 1.2.4
          hue => 3.11.0
          mahout => 0.12.2

          Show
          jonathak Jonathan Kelly added a comment - Here are a few more I'd suggest: alluxio => 1.3.0 flink => 1.1.3 hbase => 1.2.4 hue => 3.11.0 mahout => 0.12.2
          Hide
          cos Konstantin Boudnik added a comment -

          I have put the list to the description so we don't need to chase the proposed changes across all the comments. Could you update it with your list as well? Thanks!

          Show
          cos Konstantin Boudnik added a comment - I have put the list to the description so we don't need to chase the proposed changes across all the comments. Could you update it with your list as well? Thanks!
          Hide
          jonathak Jonathan Kelly added a comment -

          Thanks, done.

          Show
          jonathak Jonathan Kelly added a comment - Thanks, done.
          Hide
          rvs Roman Shaposhnik added a comment -

          Oozie 4.3.0 got released and it allows us to get rid of a custom patch. I say we upgrade.

          Show
          rvs Roman Shaposhnik added a comment - Oozie 4.3.0 got released and it allows us to get rid of a custom patch. I say we upgrade.
          Hide
          asanjar Amir Sanjar added a comment -

          excellent, that fixed Oozie issue with JAVA 1.8 thx

          Show
          asanjar Amir Sanjar added a comment - excellent, that fixed Oozie issue with JAVA 1.8 thx
          Hide
          warwithin YoungWoo Kim added a comment -

          I've filed BIGTOP-2624 and submitted a patch for that.

          Show
          warwithin YoungWoo Kim added a comment - I've filed BIGTOP-2624 and submitted a patch for that.
          Hide
          rvs Roman Shaposhnik added a comment -

          What's the consensus on Hive 2.x? I think it should be stable enough for a Bigtop release. Thoughts?

          Show
          rvs Roman Shaposhnik added a comment - What's the consensus on Hive 2.x? I think it should be stable enough for a Bigtop release. Thoughts?
          Hide
          jonathak Jonathan Kelly added a comment -

          I agree that Hive 2.x is stable enough to upgrade to it, and we on EMR should be able to contribute back what we have for this. Shall we do the same thing we did with Spark, where "hive" becomes 2.x, and if there's demand for Hive 1.x, we add "hive1", similarly to how we added "spark1"? I realize that this was a bit controversial, but it would be good to be consistent, and I also think that it would be best for the newer version to have no version suffix while the older version, if we keep it, has a suffix.

          Show
          jonathak Jonathan Kelly added a comment - I agree that Hive 2.x is stable enough to upgrade to it, and we on EMR should be able to contribute back what we have for this. Shall we do the same thing we did with Spark, where "hive" becomes 2.x, and if there's demand for Hive 1.x, we add "hive1", similarly to how we added "spark1"? I realize that this was a bit controversial, but it would be good to be consistent, and I also think that it would be best for the newer version to have no version suffix while the older version, if we keep it, has a suffix.
          Hide
          cos Konstantin Boudnik added a comment -

          Shall we do the same thing we did with Spark, where "hive" becomes 2.x, and if there's demand for Hive 1.x, we add "hive1", similarly to how we added "spark1"? I realize that this was a bit controversial, but it would be good to be consistent

          It wasn't controversial - it was outright bad idea that has induced a lot of needless work and a lot of instability into the stack. Consistency isn't a good reason to keep making the same bad decisions consistently. So, let's keep hive component at 1.x; while new hive 2.x will become hive2 or whatever. Once we are certain that new Hive doesn't break everything it touches - we can always discuss the transition plan.

          Show
          cos Konstantin Boudnik added a comment - Shall we do the same thing we did with Spark, where "hive" becomes 2.x, and if there's demand for Hive 1.x, we add "hive1", similarly to how we added "spark1"? I realize that this was a bit controversial, but it would be good to be consistent It wasn't controversial - it was outright bad idea that has induced a lot of needless work and a lot of instability into the stack. Consistency isn't a good reason to keep making the same bad decisions consistently. So, let's keep hive component at 1.x; while new hive 2.x will become hive2 or whatever. Once we are certain that new Hive doesn't break everything it touches - we can always discuss the transition plan.
          Hide
          asanjar Amir Sanjar added a comment -

          +1 , however I would like to recommand to introduce Hive 2.x as a tech-preview "hive2" component at first. Then, we gradually test and migrate other components to "hive2". When we all reach consensus on stability of stack we will do the switch ( "hive"=hive 2.x and "hive1"=hive 1.x ) .

          Show
          asanjar Amir Sanjar added a comment - +1 , however I would like to recommand to introduce Hive 2.x as a tech-preview "hive2" component at first. Then, we gradually test and migrate other components to "hive2". When we all reach consensus on stability of stack we will do the switch ( "hive"=hive 2.x and "hive1"=hive 1.x ) .
          Hide
          rvs Roman Shaposhnik added a comment -

          Konstantin Boudnik I don't see spark 1 vs 2 and hive 1 vs 2 being the same. Also, I think you're raising an important issue but from your phrasing it sounds like it is a solve problem (or at least something where there's a consensus). I don't think that's the case either – so let me take that part to the mailing list. Olaf Flebbe Amir Sanjar & Jonathan Kelly please consider following up there.

          Show
          rvs Roman Shaposhnik added a comment - Konstantin Boudnik I don't see spark 1 vs 2 and hive 1 vs 2 being the same. Also, I think you're raising an important issue but from your phrasing it sounds like it is a solve problem (or at least something where there's a consensus). I don't think that's the case either – so let me take that part to the mailing list. Olaf Flebbe Amir Sanjar & Jonathan Kelly please consider following up there.
          Hide
          cos Konstantin Boudnik added a comment -

          Well, if we going to start renameing components around sure it will be the same. Because we have dependencies in the stack. At least these are dependent {{hive:['oozie', 'pig','zeppelin'] }} on hive and would be affected. Who knows what else.

          Hence, in the spirit of small reversal changes we'd be better off by introducing hive2 component and then making sure that the rest of the stack works with it as expected. Once we have the reassurance, we can make hive2 to be the default and vu'a la - we done without much of a disruption. Does it make sense?

          Show
          cos Konstantin Boudnik added a comment - Well, if we going to start renameing components around sure it will be the same. Because we have dependencies in the stack. At least these are dependent {{hive: ['oozie', 'pig','zeppelin'] }} on hive and would be affected. Who knows what else. Hence, in the spirit of small reversal changes we'd be better off by introducing hive2 component and then making sure that the rest of the stack works with it as expected. Once we have the reassurance, we can make hive2 to be the default and vu'a la - we done without much of a disruption. Does it make sense?
          Hide
          jonathak Jonathan Kelly added a comment -

          Some of us thought it was a good idea, and some of us didn't. That is the definition of controversial.

          Show
          jonathak Jonathan Kelly added a comment - Some of us thought it was a good idea, and some of us didn't. That is the definition of controversial.
          Hide
          cos Konstantin Boudnik added a comment -

          That's true

          Show
          cos Konstantin Boudnik added a comment - That's true
          Hide
          sekikn Kengo Seki added a comment -

          Just updated the list.

          datafu 1.0.0 ==> 1.3.0 (BIGTOP-2370)
          kafka 0.10.1.0 ==> 0.10.1.1 (BIGTOP-2680)

          Show
          sekikn Kengo Seki added a comment - Just updated the list. datafu 1.0.0 ==> 1.3.0 ( BIGTOP-2370 ) kafka 0.10.1.0 ==> 0.10.1.1 ( BIGTOP-2680 )
          Hide
          warwithin YoungWoo Kim added a comment - - edited

          Added 'zeppelin 0.7.0' (BIGTOP-2689)

          Show
          warwithin YoungWoo Kim added a comment - - edited Added 'zeppelin 0.7.0' ( BIGTOP-2689 )
          Hide
          rvs Roman Shaposhnik added a comment -

          The BOM has been finalized!

          Show
          rvs Roman Shaposhnik added a comment - The BOM has been finalized!

            People

            • Assignee:
              rvs Roman Shaposhnik
              Reporter:
              cos Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development