Details

    • Type: Task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: spark
    • Labels:
      None

      Description

      Spark 1.3.0 was released today (see http://spark.apache.org/news/spark-1-3-0-released.html), so I figured that I might as well upgrade it in Bigtop as soon as possible. I have already tested that Spark 1.3.0 works fine with just the trivial changes to the Spark version number in bigtop.mk.

      1. BIGTOP-1755.patch
        2 kB
        Jonathan Kelly

        Issue Links

          Activity

          Hide
          jayunit100 jay vyas added a comment -

          Thanks, jonathan. Do we know if we need to upgrade anything in the packages? I guess not.

          I'm assuming you've tested this and it works?

          Show
          jayunit100 jay vyas added a comment - Thanks, jonathan. Do we know if we need to upgrade anything in the packages? I guess not. I'm assuming you've tested this and it works?
          Hide
          jonathak Jonathan Kelly added a comment -

          If you're referring to anything like the do-component-build script, the RPM/DEB specs, or the install_spark.sh script, no, I don't think any of them need to be updated.

          I tested it by building an RPM with Spark 1.3.0 and installing it on EMR. spark-shell runs fine, as do some simple example apps. That's about the extent of the testing I did.

          Show
          jonathak Jonathan Kelly added a comment - If you're referring to anything like the do-component-build script, the RPM/DEB specs, or the install_spark.sh script, no, I don't think any of them need to be updated. I tested it by building an RPM with Spark 1.3.0 and installing it on EMR. spark-shell runs fine, as do some simple example apps. That's about the extent of the testing I did.
          Hide
          jayunit100 jay vyas added a comment -

          testing this now. im definetely okay with updating, will check back tomnorrow, so far the build looks good, taking a long time though of course...

          Show
          jayunit100 jay vyas added a comment - testing this now. im definetely okay with updating, will check back tomnorrow, so far the build looks good, taking a long time though of course...
          Hide
          kaiyzen Nate DAmico added a comment -

          1.3+, should be api compatible across the 1.x line of spark releases, that is what databricks stands by, so it "should" work with existing setup.

          Sounds like 1.3.1, or equiv, might be out pretty soon, seen a couple issues come across dev list

          Show
          kaiyzen Nate DAmico added a comment - 1.3+, should be api compatible across the 1.x line of spark releases, that is what databricks stands by, so it "should" work with existing setup. Sounds like 1.3.1, or equiv, might be out pretty soon, seen a couple issues come across dev list
          Hide
          warwithin YoungWoo Kim added a comment -

          +1

          I've done tests like below:

          1. Build packages
          2. running examples with fresh installation 1.3.0
          3. Upgrading from 1.2.1 to 1.3.0 and running examples
          4. Running a terasort application written in scala on Spark 1.3.0
          Show
          warwithin YoungWoo Kim added a comment - +1 I've done tests like below: Build packages running examples with fresh installation 1.3.0 Upgrading from 1.2.1 to 1.3.0 and running examples Running a terasort application written in scala on Spark 1.3.0
          Hide
          jonathak Jonathan Kelly added a comment -

          Bigtop doesn't really care about the fact that Spark ensures API compatibility across 1.x releases. It's more just that simply upgrading the Spark version in Bigtop could potentially cause the RPM/DEB builds to fail (due to, say, build output files being in different places, or requiring different Maven build options, etc.) or could cause the installed packages not to work properly (e.g., maybe the new major version adds some feature or file that do-component-build or install_spark.sh doesn't take into account and thus doesn't make its way into the RPM/DEB). Both of these situations could certainly occur across 1.x releases, but I did verify that this is not the case for 1.2.1->1.3.0, as it does still work without any changes other than upgrading the version in bigtop.mk.

          Show
          jonathak Jonathan Kelly added a comment - Bigtop doesn't really care about the fact that Spark ensures API compatibility across 1.x releases. It's more just that simply upgrading the Spark version in Bigtop could potentially cause the RPM/DEB builds to fail (due to, say, build output files being in different places, or requiring different Maven build options, etc.) or could cause the installed packages not to work properly (e.g., maybe the new major version adds some feature or file that do-component-build or install_spark.sh doesn't take into account and thus doesn't make its way into the RPM/DEB). Both of these situations could certainly occur across 1.x releases, but I did verify that this is not the case for 1.2.1->1.3.0, as it does still work without any changes other than upgrading the version in bigtop.mk.
          Hide
          cos Konstantin Boudnik added a comment -

          YoungWoo Kim, shall it be committed then?

          Show
          cos Konstantin Boudnik added a comment - YoungWoo Kim , shall it be committed then?
          Hide
          warwithin YoungWoo Kim added a comment - - edited

          I think so but I'm waiting for jay vyas's comment. Seems that Jay is looking into this.

          Show
          warwithin YoungWoo Kim added a comment - - edited I think so but I'm waiting for jay vyas 's comment. Seems that Jay is looking into this.
          Hide
          jayunit100 jay vyas added a comment -

          go for it ! No need to wait for me .... After all you're our newest commiter

          Show
          jayunit100 jay vyas added a comment - go for it ! No need to wait for me .... After all you're our newest commiter
          Hide
          jayunit100 jay vyas added a comment -

          iirc, when i tested, i had no problems - but forgot to update the jura.

          Show
          jayunit100 jay vyas added a comment - iirc, when i tested, i had no problems - but forgot to update the jura.
          Hide
          warwithin YoungWoo Kim added a comment -

          jay vyas, Thanks for your comment! Will commit it.

          Show
          warwithin YoungWoo Kim added a comment - jay vyas , Thanks for your comment! Will commit it.
          Hide
          warwithin YoungWoo Kim added a comment -

          Committed. Thanks Jonathan Kelly!

          Show
          warwithin YoungWoo Kim added a comment - Committed. Thanks Jonathan Kelly !

            People

            • Assignee:
              jonathak Jonathan Kelly
              Reporter:
              jonathak Jonathan Kelly
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development