Bigtop
  1. Bigtop
  2. BIGTOP-1193

Update Spark packaging for various conventions

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This is a bit of a mixed bag of things I noticed:

      • We're using the old style of checking multiple locations for bigtop-utils. If we depend on bigtop-utils 0.7 we can safely assume where it is.
      • We're invoking install_spark.sh with `sh` instead of `bash` in one place - someone's going to get a cryptic syntax error one day
      • We're invoking do-component-build in deb/.../rules with an unnecessary environment variable (accidentally introduced in BIGTOP-1150 - it's a downstream modification specific to a Certain Distribution of Hadoop based on Bigtop).

        Activity

        Hide
        Mark Grover added a comment -

        I am not sure if we need "-Divy.home=$

        {HOME}

        /.ivy2" in the line invoking do-component-build. Do we?

        Show
        Mark Grover added a comment - I am not sure if we need "-Divy.home=$ {HOME} /.ivy2" in the line invoking do-component-build. Do we?
        Hide
        Sean Mackrory added a comment -

        I wouldn't have thought so, now that you mention it, but I don't know why that would ever be needed, so I can't say with confidence it isn't needed. That was also added by BIGTOP-1150. Any insight Roman Shaposhnik?

        Show
        Sean Mackrory added a comment - I wouldn't have thought so, now that you mention it, but I don't know why that would ever be needed, so I can't say with confidence it isn't needed. That was also added by BIGTOP-1150 . Any insight Roman Shaposhnik ?
        Hide
        Konstantin Boudnik added a comment -

        I think ivy thingy is a left over from an initial copy-paste implementation. We are using maven, not sbt, for the build, so ivy is irrelevant.

        Show
        Konstantin Boudnik added a comment - I think ivy thingy is a left over from an initial copy-paste implementation. We are using maven, not sbt, for the build, so ivy is irrelevant.
        Hide
        Mark Grover added a comment -

        Thanks Cos for the confirmation.

        So, yeah, I am a +1 if you take out the ivy.home thing.

        Show
        Mark Grover added a comment - Thanks Cos for the confirmation. So, yeah, I am a +1 if you take out the ivy.home thing.
        Hide
        Sean Mackrory added a comment - - edited

        Sounds good to me - patch updated. While I was at it, the common/spark/run file does not appear to be used, so I yanked that too. Packages still build fine and I can't find that file in previously built packages. Is that cool with you, Konstantin Boudnik?

        Show
        Sean Mackrory added a comment - - edited Sounds good to me - patch updated. While I was at it, the common/spark/run file does not appear to be used, so I yanked that too. Packages still build fine and I can't find that file in previously built packages. Is that cool with you, Konstantin Boudnik ?
        Hide
        Mark Grover added a comment -

        +1, removing common/spark/run sounds good to me but let's wait for Cos' response just in case.

        Thanks Sean!

        Show
        Mark Grover added a comment - +1, removing common/spark/run sounds good to me but let's wait for Cos' response just in case. Thanks Sean!
        Hide
        Konstantin Boudnik added a comment -

        Sean, the file might a left over from 0.7.0 version work. But please make sure that it isn't used in runtime.

        Show
        Konstantin Boudnik added a comment - Sean, the file might a left over from 0.7.0 version work. But please make sure that it isn't used in runtime.
        Hide
        Roman Shaposhnik added a comment -

        I'd say – go ahead and get rid of ivy

        Show
        Roman Shaposhnik added a comment - I'd say – go ahead and get rid of ivy

          People

          • Assignee:
            Sean Mackrory
            Reporter:
            Sean Mackrory
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development