Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: debian, rpm
    • Labels:
      None

      Description

      Bumping up the version of the Spark component and, potentially, fixing the component build around it.

      1. BIGTOP-1284.1.patch
        31 kB
        Mark Grover
      2. BIGTOP-1284.patch
        3 kB
        Konstantin Boudnik
      3. BIGTOP-1284.patch
        3 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Any progress here?

          Show
          Konstantin Boudnik added a comment - Any progress here?
          Hide
          Mark Grover added a comment -

          Sorry, Cos I've been swamped. Will try to work on this later this week.

          Show
          Mark Grover added a comment - Sorry, Cos I've been swamped. Will try to work on this later this week.
          Hide
          Konstantin Boudnik added a comment -

          Thanks Mark - lovely!

          Show
          Konstantin Boudnik added a comment - Thanks Mark - lovely!
          Hide
          Konstantin Boudnik added a comment -

          The attached patch produces all spark packages with Scala 2.10 (you might need to have BIGTOP-1304 in place)

          Show
          Konstantin Boudnik added a comment - The attached patch produces all spark packages with Scala 2.10 (you might need to have BIGTOP-1304 in place)
          Hide
          Konstantin Boudnik added a comment -

          Missed one.

          Show
          Konstantin Boudnik added a comment - Missed one.
          Hide
          Mark Grover added a comment -

          Quick update. I have most of the code ready, need to test it and then post. Will keep you all posted.

          I also noticed, Giri is the assignee now. Should I stop working on this?

          Show
          Mark Grover added a comment - Quick update. I have most of the code ready, need to test it and then post. Will keep you all posted. I also noticed, Giri is the assignee now. Should I stop working on this?
          Hide
          Konstantin Boudnik added a comment -

          Giri was expressing the interest to get it covered as the JIRA went silent for a bit. If you have the patch then please post it here by all means! Giri, please update here if you are moving forward on your side. Thanks guys and sorry about the mid-air collide, that I might have cause by publishing the earlier version of the patch - just trying to wrap up the release

          Show
          Konstantin Boudnik added a comment - Giri was expressing the interest to get it covered as the JIRA went silent for a bit. If you have the patch then please post it here by all means! Giri, please update here if you are moving forward on your side. Thanks guys and sorry about the mid-air collide, that I might have cause by publishing the earlier version of the patch - just trying to wrap up the release
          Hide
          Giridharan Kesavan added a comment -

          Mark Grover Since you mentioned that you have the patch almost ready, I will back off. Let me know if you need any help.

          Show
          Giridharan Kesavan added a comment - Mark Grover Since you mentioned that you have the patch almost ready, I will back off. Let me know if you need any help.
          Hide
          Mark Grover added a comment -

          Thanks all. Posting an update, testing my patch right now, running into some issues but nothing unexpected.

          FWIW, Spark 1.0.0 (which is not guaranteed to be compatible with 0.9*) is being voted right now. Of course, a) I have no idea, when the release would be made, b) I have no idea how much packaging re-jig we will have to do for that, so I am planning to continue working on 0.9.1 but we should consider 1.0.0 if things line up right.

          And, a few other things (mostly for me so I don't forget):
          Need to ensure our toolchain has Maven 3.0.4+
          Need to ensure we have Scala 2.10 installed

          Show
          Mark Grover added a comment - Thanks all. Posting an update, testing my patch right now, running into some issues but nothing unexpected. FWIW, Spark 1.0.0 (which is not guaranteed to be compatible with 0.9*) is being voted right now. Of course, a) I have no idea, when the release would be made, b) I have no idea how much packaging re-jig we will have to do for that, so I am planning to continue working on 0.9.1 but we should consider 1.0.0 if things line up right. And, a few other things (mostly for me so I don't forget): Need to ensure our toolchain has Maven 3.0.4+ Need to ensure we have Scala 2.10 installed
          Hide
          Konstantin Boudnik added a comment -

          IMO, if we keep chasing new releases in the eco-system we never will be able to release our stack. Spark, Flume, and Giraph are the last bits we are missing at this point - I'd say let's get it done first. If the community feels like adding Spark 1.0 soon - we always can do a follow up release soon after.

          Show
          Konstantin Boudnik added a comment - IMO, if we keep chasing new releases in the eco-system we never will be able to release our stack. Spark, Flume, and Giraph are the last bits we are missing at this point - I'd say let's get it done first. If the community feels like adding Spark 1.0 soon - we always can do a follow up release soon after.
          Hide
          Roman Shaposhnik added a comment -

          +1 to what Cos said – lets have the pre-1.0 first and will go from there.

          Show
          Roman Shaposhnik added a comment - +1 to what Cos said – lets have the pre-1.0 first and will go from there.
          Hide
          Mark Grover added a comment -

          Konstantin Boudnik would appreciate a review!

          Show
          Mark Grover added a comment - Konstantin Boudnik would appreciate a review!
          Hide
          Konstantin Boudnik added a comment -

          Thanks Mark! Looks pretty good overall. A couple of comments:

          • do we still need to keep a custom compute-classpath.sh ? I did it back in a day as a quick hack, but I am not sure if the script got any better since - hence the quesion.
          • in the compute-classpath.sh: why Spark needs MAPRED_HOME?
            {{export HADOOP_MAPRED_HOME=$
            Unknown macro: {HADOOP_MAPRED_HOME}

            }}

          Other than that +1. Please commit. Thank you!

          Show
          Konstantin Boudnik added a comment - Thanks Mark! Looks pretty good overall. A couple of comments: do we still need to keep a custom compute-classpath.sh ? I did it back in a day as a quick hack, but I am not sure if the script got any better since - hence the quesion. in the compute-classpath.sh : why Spark needs MAPRED_HOME? {{export HADOOP_MAPRED_HOME=$ Unknown macro: {HADOOP_MAPRED_HOME} }} Other than that +1. Please commit. Thank you!
          Hide
          Sean Mackrory added a comment -

          As I recall, HADOOP_MAPRED_HOME was used to pick up various I/O format classes. I don't know if that's still the case, but it was as of a release or two ago.

          Show
          Sean Mackrory added a comment - As I recall, HADOOP_MAPRED_HOME was used to pick up various I/O format classes. I don't know if that's still the case, but it was as of a release or two ago.
          Hide
          Mark Grover added a comment -

          Yeah, sorry emails have been very problematic recently so I am just polling on this JIRA

          Indeed, Spark relies on mapred home to pick up Input/Output format classes. Things may have changed more recently but I think that's still the case, which is why we need it.

          As far as compute-classpath.sh goes, my original testing revealed that we needed it. So, it's there. We can look more into if it's possible to remove it as an follow-on JIRA (which may or may not get resolved by Bigtop 0.8). Sounds like you are ok with that since you mentioned "Other than that +1. Please commit."

          So, in spirit of marching towards the release, I will commit this but please yell at me if there is anything else!

          Show
          Mark Grover added a comment - Yeah, sorry emails have been very problematic recently so I am just polling on this JIRA Indeed, Spark relies on mapred home to pick up Input/Output format classes. Things may have changed more recently but I think that's still the case, which is why we need it. As far as compute-classpath.sh goes, my original testing revealed that we needed it. So, it's there. We can look more into if it's possible to remove it as an follow-on JIRA (which may or may not get resolved by Bigtop 0.8). Sounds like you are ok with that since you mentioned "Other than that +1. Please commit." So, in spirit of marching towards the release, I will commit this but please yell at me if there is anything else!
          Hide
          Konstantin Boudnik added a comment -

          Well done, thanks mate!

          Show
          Konstantin Boudnik added a comment - Well done, thanks mate!
          Hide
          Mark Grover added a comment -

          FYI, created BIGTOP-1318 to stop forking compute-classpath.sh in Bigtop and leverage the upstream compute-classpath.sh instead.

          Show
          Mark Grover added a comment - FYI, created BIGTOP-1318 to stop forking compute-classpath.sh in Bigtop and leverage the upstream compute-classpath.sh instead.

            People

            • Assignee:
              Mark Grover
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development