Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: debian, rpm
    • Labels:
      None

      Description

      Bumping up the version of the Spark component and, potentially, fixing the component build around it.

      1. BIGTOP-1284.patch
        3 kB
        Konstantin Boudnik
      2. BIGTOP-1284.patch
        3 kB
        Konstantin Boudnik
      3. BIGTOP-1284.1.patch
        31 kB
        Mark Grover

        Issue Links

          Activity

          Konstantin Boudnik created issue -
          Konstantin Boudnik made changes -
          Field Original Value New Value
          Fix Version/s 0.8.0 [ 12324841 ]
          Konstantin Boudnik made changes -
          Component/s General [ 12315313 ]
          Mark Grover made changes -
          Due Date 02/May/14
          Hide
          Konstantin Boudnik added a comment -

          Any progress here?

          Show
          Konstantin Boudnik added a comment - Any progress here?
          Hide
          Mark Grover added a comment -

          Sorry, Cos I've been swamped. Will try to work on this later this week.

          Show
          Mark Grover added a comment - Sorry, Cos I've been swamped. Will try to work on this later this week.
          Hide
          Konstantin Boudnik added a comment -

          Thanks Mark - lovely!

          Show
          Konstantin Boudnik added a comment - Thanks Mark - lovely!
          Hide
          Konstantin Boudnik added a comment -

          The attached patch produces all spark packages with Scala 2.10 (you might need to have BIGTOP-1304 in place)

          Show
          Konstantin Boudnik added a comment - The attached patch produces all spark packages with Scala 2.10 (you might need to have BIGTOP-1304 in place)
          Konstantin Boudnik made changes -
          Attachment BIGTOP-1284.patch [ 12644153 ]
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Konstantin Boudnik made changes -
          Assignee Mark Grover [ mgrover ] Konstantin Boudnik [ cos ]
          Konstantin Boudnik made changes -
          Link This issue is blocked by BIGTOP-1304 [ BIGTOP-1304 ]
          Hide
          Konstantin Boudnik added a comment -

          Missed one.

          Show
          Konstantin Boudnik added a comment - Missed one.
          Konstantin Boudnik made changes -
          Attachment BIGTOP-1284.patch [ 12644154 ]
          Konstantin Boudnik made changes -
          Summary bump Spark version to 0.9 bump Spark version to 0.9.1
          Konstantin Boudnik made changes -
          Component/s Debian [ 12315312 ]
          Component/s RPM [ 12315311 ]
          Component/s General [ 12315313 ]
          Konstantin Boudnik made changes -
          Assignee Konstantin Boudnik [ cos ] Giridharan Kesavan [ gkesavan ]
          Hide
          Mark Grover added a comment -

          Quick update. I have most of the code ready, need to test it and then post. Will keep you all posted.

          I also noticed, Giri is the assignee now. Should I stop working on this?

          Show
          Mark Grover added a comment - Quick update. I have most of the code ready, need to test it and then post. Will keep you all posted. I also noticed, Giri is the assignee now. Should I stop working on this?
          Hide
          Konstantin Boudnik added a comment -

          Giri was expressing the interest to get it covered as the JIRA went silent for a bit. If you have the patch then please post it here by all means! Giri, please update here if you are moving forward on your side. Thanks guys and sorry about the mid-air collide, that I might have cause by publishing the earlier version of the patch - just trying to wrap up the release

          Show
          Konstantin Boudnik added a comment - Giri was expressing the interest to get it covered as the JIRA went silent for a bit. If you have the patch then please post it here by all means! Giri, please update here if you are moving forward on your side. Thanks guys and sorry about the mid-air collide, that I might have cause by publishing the earlier version of the patch - just trying to wrap up the release
          Hide
          Giridharan Kesavan added a comment -

          Mark Grover Since you mentioned that you have the patch almost ready, I will back off. Let me know if you need any help.

          Show
          Giridharan Kesavan added a comment - Mark Grover Since you mentioned that you have the patch almost ready, I will back off. Let me know if you need any help.
          Hide
          Mark Grover added a comment -

          Thanks all. Posting an update, testing my patch right now, running into some issues but nothing unexpected.

          FWIW, Spark 1.0.0 (which is not guaranteed to be compatible with 0.9*) is being voted right now. Of course, a) I have no idea, when the release would be made, b) I have no idea how much packaging re-jig we will have to do for that, so I am planning to continue working on 0.9.1 but we should consider 1.0.0 if things line up right.

          And, a few other things (mostly for me so I don't forget):
          Need to ensure our toolchain has Maven 3.0.4+
          Need to ensure we have Scala 2.10 installed

          Show
          Mark Grover added a comment - Thanks all. Posting an update, testing my patch right now, running into some issues but nothing unexpected. FWIW, Spark 1.0.0 (which is not guaranteed to be compatible with 0.9*) is being voted right now. Of course, a) I have no idea, when the release would be made, b) I have no idea how much packaging re-jig we will have to do for that, so I am planning to continue working on 0.9.1 but we should consider 1.0.0 if things line up right. And, a few other things (mostly for me so I don't forget): Need to ensure our toolchain has Maven 3.0.4+ Need to ensure we have Scala 2.10 installed
          Hide
          Konstantin Boudnik added a comment -

          IMO, if we keep chasing new releases in the eco-system we never will be able to release our stack. Spark, Flume, and Giraph are the last bits we are missing at this point - I'd say let's get it done first. If the community feels like adding Spark 1.0 soon - we always can do a follow up release soon after.

          Show
          Konstantin Boudnik added a comment - IMO, if we keep chasing new releases in the eco-system we never will be able to release our stack. Spark, Flume, and Giraph are the last bits we are missing at this point - I'd say let's get it done first. If the community feels like adding Spark 1.0 soon - we always can do a follow up release soon after.
          Hide
          Roman Shaposhnik added a comment -

          +1 to what Cos said – lets have the pre-1.0 first and will go from there.

          Show
          Roman Shaposhnik added a comment - +1 to what Cos said – lets have the pre-1.0 first and will go from there.
          Mark Grover made changes -
          Assignee Giridharan Kesavan [ gkesavan ] Mark Grover [ mgrover ]
          Mark Grover made changes -
          Attachment BIGTOP-1284.1.patch [ 12645068 ]
          Hide
          Mark Grover added a comment -

          Konstantin Boudnik would appreciate a review!

          Show
          Mark Grover added a comment - Konstantin Boudnik would appreciate a review!
          Hide
          Konstantin Boudnik added a comment -

          Thanks Mark! Looks pretty good overall. A couple of comments:

          • do we still need to keep a custom compute-classpath.sh ? I did it back in a day as a quick hack, but I am not sure if the script got any better since - hence the quesion.
          • in the compute-classpath.sh: why Spark needs MAPRED_HOME?
            {{export HADOOP_MAPRED_HOME=$
            Unknown macro: {HADOOP_MAPRED_HOME}

            }}

          Other than that +1. Please commit. Thank you!

          Show
          Konstantin Boudnik added a comment - Thanks Mark! Looks pretty good overall. A couple of comments: do we still need to keep a custom compute-classpath.sh ? I did it back in a day as a quick hack, but I am not sure if the script got any better since - hence the quesion. in the compute-classpath.sh : why Spark needs MAPRED_HOME? {{export HADOOP_MAPRED_HOME=$ Unknown macro: {HADOOP_MAPRED_HOME} }} Other than that +1. Please commit. Thank you!
          Hide
          Sean Mackrory added a comment -

          As I recall, HADOOP_MAPRED_HOME was used to pick up various I/O format classes. I don't know if that's still the case, but it was as of a release or two ago.

          Show
          Sean Mackrory added a comment - As I recall, HADOOP_MAPRED_HOME was used to pick up various I/O format classes. I don't know if that's still the case, but it was as of a release or two ago.
          Hide
          Mark Grover added a comment -

          Yeah, sorry emails have been very problematic recently so I am just polling on this JIRA

          Indeed, Spark relies on mapred home to pick up Input/Output format classes. Things may have changed more recently but I think that's still the case, which is why we need it.

          As far as compute-classpath.sh goes, my original testing revealed that we needed it. So, it's there. We can look more into if it's possible to remove it as an follow-on JIRA (which may or may not get resolved by Bigtop 0.8). Sounds like you are ok with that since you mentioned "Other than that +1. Please commit."

          So, in spirit of marching towards the release, I will commit this but please yell at me if there is anything else!

          Show
          Mark Grover added a comment - Yeah, sorry emails have been very problematic recently so I am just polling on this JIRA Indeed, Spark relies on mapred home to pick up Input/Output format classes. Things may have changed more recently but I think that's still the case, which is why we need it. As far as compute-classpath.sh goes, my original testing revealed that we needed it. So, it's there. We can look more into if it's possible to remove it as an follow-on JIRA (which may or may not get resolved by Bigtop 0.8). Sounds like you are ok with that since you mentioned "Other than that +1. Please commit." So, in spirit of marching towards the release, I will commit this but please yell at me if there is anything else!
          Mark Grover made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Konstantin Boudnik added a comment -

          Well done, thanks mate!

          Show
          Konstantin Boudnik added a comment - Well done, thanks mate!
          Hide
          Mark Grover added a comment -

          FYI, created BIGTOP-1318 to stop forking compute-classpath.sh in Bigtop and leverage the upstream compute-classpath.sh instead.

          Show
          Mark Grover added a comment - FYI, created BIGTOP-1318 to stop forking compute-classpath.sh in Bigtop and leverage the upstream compute-classpath.sh instead.
          Konstantin Boudnik made changes -
          Link This issue relates to BIGTOP-1318 [ BIGTOP-1318 ]
          Allen Wittenauer made changes -
          Link This issue is related to BIGTOP-1345 [ BIGTOP-1345 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          20d 21h 18m 1 Konstantin Boudnik 09/May/14 18:41
          Patch Available Patch Available Resolved Resolved
          7d 32m 1 Mark Grover 16/May/14 19:13

            People

            • Assignee:
              Mark Grover
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development