Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2080

Investigate removing Scala from the toolchain

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: toolchain
    • Labels:
      None

      Description

      Older versions of Spark needed Scala to be installed separately but not anymore. For a while, Spark and Kafka have pulled down a copy of the Scala compiler and libraries through Maven – BigPetStore does the same through the Groovy Scala plugin.

      We may be able to drop the Scala installation in the Bigtop toolchain. We should investigate this.

      1. BIGTOP-2080.2.patch
        3 kB
        RJ Nowling
      2. BIGTOP-2080.1.patch
        3 kB
        RJ Nowling

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        moving to dev@

        Show
        cos Konstantin Boudnik added a comment - moving to dev@
        Hide
        oflebbe Olaf Flebbe added a comment -

        JDK-8 on ubuntu (and debian) is about 300MB in size ... seems to be a lot.

        AFAIK we are not allowed to distribute Oracle JDK.

        Can we use a new JIRA or dev@ to discuss?

        Show
        oflebbe Olaf Flebbe added a comment - JDK-8 on ubuntu (and debian) is about 300MB in size ... seems to be a lot. AFAIK we are not allowed to distribute Oracle JDK. Can we use a new JIRA or dev@ to discuss?
        Hide
        cos Konstantin Boudnik added a comment - - edited

        Interestingly the new image went in size significantly: from 2.1GB to 2.5GB
        bigtop/slaves trunk-ubuntu-14.04 e9223253a71e 2 days ago 2.5 GB
        bigtop/slaves ubuntu-14.04 f4dafee135a5 3 months ago 2.175 GB

        Yet, centos-7 went up in size just about 60MB.

        Have we really added almost 0.5GB of the new stuff in the latest images?

        Show
        cos Konstantin Boudnik added a comment - - edited Interestingly the new image went in size significantly: from 2.1GB to 2.5GB bigtop/slaves trunk-ubuntu-14.04 e9223253a71e 2 days ago 2.5 GB bigtop/slaves ubuntu-14.04 f4dafee135a5 3 months ago 2.175 GB Yet, centos-7 went up in size just about 60MB. Have we really added almost 0.5GB of the new stuff in the latest images?
        Hide
        rnowling RJ Nowling added a comment -

        Thanks Evans Ye!

        Show
        rnowling RJ Nowling added a comment - Thanks Evans Ye !
        Hide
        evans_ye Evans Ye added a comment -

        I've built and pushed the new set of bigtop/slaves:trunk-* images on the docker hub. You can start using them now.

        Show
        evans_ye Evans Ye added a comment - I've built and pushed the new set of bigtop/slaves:trunk-* images on the docker hub. You can start using them now.
        Hide
        jonathak Jonathan Kelly added a comment -

        Good to know, thanks. Just wanted to confirm.

        Show
        jonathak Jonathan Kelly added a comment - Good to know, thanks. Just wanted to confirm.
        Hide
        rnowling RJ Nowling added a comment -

        Jonathan Kelly The Scala version is set in the bigtop.bom and passed to the Maven, etc. builds for Kafka and Spark. The builds for those projects handle downloading the Scala compilers and libraries. It was in early releases of Spark that you needed a compiler installed separately.

        Show
        rnowling RJ Nowling added a comment - Jonathan Kelly The Scala version is set in the bigtop.bom and passed to the Maven, etc. builds for Kafka and Spark. The builds for those projects handle downloading the Scala compilers and libraries. It was in early releases of Spark that you needed a compiler installed separately.
        Hide
        jonathak Jonathan Kelly added a comment -

        Isn't removing Scala from the toolchain in conflict with upgrading it to 2.10.4 (BIGTOP-1943), which is needed to upgrade to Phoenix 4.5.0 (BIGTOP-1942) (which is, btw, needed to upgrade to Spark 1.5 (BIGTOP-1944))?

        Show
        jonathak Jonathan Kelly added a comment - Isn't removing Scala from the toolchain in conflict with upgrading it to 2.10.4 ( BIGTOP-1943 ), which is needed to upgrade to Phoenix 4.5.0 ( BIGTOP-1942 ) (which is, btw, needed to upgrade to Spark 1.5 ( BIGTOP-1944 ))?
        Show
        oflebbe Olaf Flebbe added a comment - Unfortunatly it failed miserably. http://ci.bigtop.apache.org:8080/view/Docker/job/Docker-Toolchain-Trunk/10/BUILD_ENVIRONMENTS=centos-6,label=docker-slave-05/console
        Hide
        rnowling RJ Nowling added a comment -

        Committed. Olaf Flebbe, Evans Ye do we need to respin the CI docker containers since we've updated the toolchain?

        Show
        rnowling RJ Nowling added a comment - Committed. Olaf Flebbe , Evans Ye do we need to respin the CI docker containers since we've updated the toolchain?
        Hide
        oflebbe Olaf Flebbe added a comment -

        LGTM: +1

        Show
        oflebbe Olaf Flebbe added a comment - LGTM: +1
        Hide
        rnowling RJ Nowling added a comment -

        Updated patch to remove Scala version info from toolchain README. Thanks Olaf Flebbe for the catching that!

        Show
        rnowling RJ Nowling added a comment - Updated patch to remove Scala version info from toolchain README. Thanks Olaf Flebbe for the catching that!
        Hide
        oflebbe Olaf Flebbe added a comment -

        RJ Nowling: Could you please remove the scala line from bigtop_toolchain/README.md , too ?

        Show
        oflebbe Olaf Flebbe added a comment - RJ Nowling : Could you please remove the scala line from bigtop_toolchain/README.md , too ?
        Hide
        rnowling RJ Nowling added a comment -

        services.gradle.org is back up so I tested the Kafka RPM build. It worked without issues. I'll give the patch 24 hours for review before committing.

        Show
        rnowling RJ Nowling added a comment - services.gradle.org is back up so I tested the Kafka RPM build. It worked without issues. I'll give the patch 24 hours for review before committing.
        Hide
        cos Konstantin Boudnik added a comment -

        Dynamic compilers pulling is a security hole asking to be exploited, but I guess it is not on us
        I'd go with it - makes our environment lighter, which is good!

        Show
        cos Konstantin Boudnik added a comment - Dynamic compilers pulling is a security hole asking to be exploited, but I guess it is not on us I'd go with it - makes our environment lighter, which is good!
        Hide
        rnowling RJ Nowling added a comment -

        Removes the Scala toolchain module. Tested by building the Spark RPM – still need to test Kafka but services.gradle.org seems down at the moment, receiving timeouts. I'll test Kafka when services.gradle.org is back up.

        Show
        rnowling RJ Nowling added a comment - Removes the Scala toolchain module. Tested by building the Spark RPM – still need to test Kafka but services.gradle.org seems down at the moment, receiving timeouts. I'll test Kafka when services.gradle.org is back up.

          People

          • Assignee:
            rnowling RJ Nowling
            Reporter:
            rnowling RJ Nowling
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development