Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1415

Remove redundant jars from spark package.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: spark
    • Labels:
      None

      Description

      Looking at the current package for Spark - it packages a lot of things that are not necessary, and is missing some others that might be useful.

      bagel/lib/spark-bagel_2.10-1.0.0-*.jar
      core/lib/spark-core_2.10-1.0.0-*.jar
      lib/*
      mllib/lib/spark-mllib_2.10-1.0.0-*.jar
      repl/lib/spark-repl_2.10-1.0.0-*.jar
      streaming/lib/spark-streaming_2.10-1.0.0-*.jar

      None of the above are necessary, since they are already packaged in assembly/lib/spark-assembly-1.0.0-*.jar, which is what everybody should be using.
      examples/lib/spark-examples_2.10-1.0.0-*-javadoc.jar
      examples/lib/spark-examples_2.10-1.0.0-*-sources.jar
      Again not necessary, just because, well, anyone who needs that can use maven.

      1. BIGTOP-1415.patch
        3 kB
        Manikandan Narayanaswamy

        Activity

        Hide
        jayunit100 jay vyas added a comment -

        thanks for noticing this. i noticed cdh* versions on the end of your jars... i assume we're not naming bigtop jars with that suffix?

        Show
        jayunit100 jay vyas added a comment - thanks for noticing this. i noticed cdh* versions on the end of your jars... i assume we're not naming bigtop jars with that suffix?
        Hide
        mnarayan Manikandan Narayanaswamy added a comment -

        We fixed this internally and wanted to push a patch upstream - hence the CDH versions. Updated the description now.

        Show
        mnarayan Manikandan Narayanaswamy added a comment - We fixed this internally and wanted to push a patch upstream - hence the CDH versions. Updated the description now.
        Hide
        rvs Roman Shaposhnik added a comment -

        Would be awesome to get this patched. On a related note, it would be very nice if somebody had cycles to setup a jenkins job that would audit for duplicate jars in our packges and alert us about such duplication.

        Show
        rvs Roman Shaposhnik added a comment - Would be awesome to get this patched. On a related note, it would be very nice if somebody had cycles to setup a jenkins job that would audit for duplicate jars in our packges and alert us about such duplication.
        Hide
        mnarayan Manikandan Narayanaswamy added a comment -

        jay vyas, Roman Shaposhnik: Could you guys take a look at the patch to see if it looks good. Thanks!

        Show
        mnarayan Manikandan Narayanaswamy added a comment - jay vyas , Roman Shaposhnik : Could you guys take a look at the patch to see if it looks good. Thanks!
        Hide
        rvs Roman Shaposhnik added a comment -

        +1 and committed! Thanks a bunch for the patch!

        Show
        rvs Roman Shaposhnik added a comment - +1 and committed! Thanks a bunch for the patch!

          People

          • Assignee:
            Unassigned
            Reporter:
            mnarayan Manikandan Narayanaswamy
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development