Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None

      Description

      We are basing our next internal Hadoop distribution on Bigtop and want to make Apache Giraph available to our users as part of the platform. Apache Giraph is a promising platform for running large-scale graph algorithms on top of existing Hadoop infrastructure, currently in incubation. For more information on the project, please see: http://incubator.apache.org/giraph/ If this is of interest to anyone, I've ported the RPM specfile and common build scripts to both Bigtop and Giraph trunks. Unfortunately I don't know how to package for Debian.

      1. BIGTOP-570-debian.patch.txt
        9 kB
        Roman Shaposhnik
      2. BIGTOP-570.patch.gz
        6 kB
        Andrew Purtell
      3. 0001-Giraph-package-WIP.patch.gz
        7 kB
        Andrew Purtell

        Issue Links

          Activity

          Hide
          Roman Shaposhnik added a comment -

          +1 and committed.

          Once again – thanks a million for the patch!

          One last request – I really would like to have Giraph 0.2.0-incubating in Bigtop 0.4.0 distribution. In order to do that we have to have it be released. Any chance you can help
          influencing Giraph community to have that release within next couple of weeks? Obviously it shouldn't be rushed, but if the schedule allows it can come very handy

          Show
          Roman Shaposhnik added a comment - +1 and committed. Once again – thanks a million for the patch! One last request – I really would like to have Giraph 0.2.0-incubating in Bigtop 0.4.0 distribution. In order to do that we have to have it be released. Any chance you can help influencing Giraph community to have that release within next couple of weeks? Obviously it shouldn't be rushed, but if the schedule allows it can come very handy
          Hide
          Andrew Purtell added a comment -

          Attached updated package WIP.

          Show
          Andrew Purtell added a comment - Attached updated package WIP.
          Hide
          Andrew Purtell added a comment -

          I see on GIRAPH-198 we are missing a property definition to use the new Netty RPC. Will update the patch momentarily.

          Show
          Andrew Purtell added a comment - I see on GIRAPH-198 we are missing a property definition to use the new Netty RPC. Will update the patch momentarily.
          Hide
          Roman Shaposhnik added a comment -

          Andrew, we're already at the tip of trunk with your patch https://github.com/apache/giraph/commit/79962a3c8c5b9a29aa346e66959ad6b5ddd79022 and according to CHANGELOG it does include GIRAPH-37

          Any other suggestions?

          Show
          Roman Shaposhnik added a comment - Andrew, we're already at the tip of trunk with your patch https://github.com/apache/giraph/commit/79962a3c8c5b9a29aa346e66959ad6b5ddd79022 and according to CHANGELOG it does include GIRAPH-37 Any other suggestions?
          Hide
          Andrew Purtell added a comment -

          GIRAPH-37 may make GIRAPH-198 moot. Suggest updating the SHA of the source tarball pull for latest Giraph trunk.

          Show
          Andrew Purtell added a comment - GIRAPH-37 may make GIRAPH-198 moot. Suggest updating the SHA of the source tarball pull for latest Giraph trunk.
          Hide
          Roman Shaposhnik added a comment -

          Andrew, the latest patches seem to be ok, but we're now blocked on the following 2 Giraph issues: GIRAPH-198 and GIRAPH-199

          I can personally provide patches for GIRAPH-199 but I would appreciate if you could help with rounding up folks who can take a look at GIRAPH-198

          Show
          Roman Shaposhnik added a comment - Andrew, the latest patches seem to be ok, but we're now blocked on the following 2 Giraph issues: GIRAPH-198 and GIRAPH-199 I can personally provide patches for GIRAPH-199 but I would appreciate if you could help with rounding up folks who can take a look at GIRAPH-198
          Hide
          Andrew Purtell added a comment -

          Updated patch.

          • Giraph do-component-build should -Dhadoop.version=2.0.0-alpha
          • Fix file not found issue with config dir in RPM specfile.
          Show
          Andrew Purtell added a comment - Updated patch. Giraph do-component-build should -Dhadoop.version=2.0.0-alpha Fix file not found issue with config dir in RPM specfile.
          Hide
          Andrew Purtell added a comment -

          Thanks Roman, attached is the latest that incorporates your patch plus changes the smoke test to a very simple Groovy script that assumes a live test cluster, much smaller/simpler, thanks for the guidance there.

          Show
          Andrew Purtell added a comment - Thanks Roman, attached is the latest that incorporates your patch plus changes the smoke test to a very simple Groovy script that assumes a live test cluster, much smaller/simpler, thanks for the guidance there.
          Hide
          Roman Shaposhnik added a comment -

          Andrew, I'm attaching a debian side of things plus some trivial modifications to your RPM code.

          Please update your test code to run against a real cluster and I think I can then submit this to our testing machinery to see how it fares.

          Once again – thanks a million for helping us add this useful project to Bigtop!

          Show
          Roman Shaposhnik added a comment - Andrew, I'm attaching a debian side of things plus some trivial modifications to your RPM code. Please update your test code to run against a real cluster and I think I can then submit this to our testing machinery to see how it fares. Once again – thanks a million for helping us add this useful project to Bigtop!
          Hide
          Roman Shaposhnik added a comment -

          Looking at it. Apologies for the complexity of POM logic – we're actually working on a Groovy solution to simplify it.

          Show
          Roman Shaposhnik added a comment - Looking at it. Apologies for the complexity of POM logic – we're actually working on a Groovy solution to simplify it.
          Hide
          Andrew Purtell added a comment -

          I had to touch a bunch of POMs, if there might be a better way, please advise.

          Show
          Andrew Purtell added a comment - I had to touch a bunch of POMs, if there might be a better way, please advise.
          Hide
          Andrew Purtell added a comment -

          Got it.

          Show
          Andrew Purtell added a comment - Got it.
          Hide
          Andrew Purtell added a comment -

          Making progress.

          Show
          Andrew Purtell added a comment - Making progress.
          Hide
          Andrew Purtell added a comment - - edited

          Attached is an updated patch that includes a Giraph smoke test, cribbed from a unit test. I wish I could report more progress. It took too much time just to figure out Maven was pulling in dependencies to some defunct java.net repo from ZooKeeper. Now I can't seem to get Maven to select the correct Hadoop artifact ("java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected"). Perhaps this is a familiar problem?

          Unfortunately just executing 'hadoop jar ...' using iTest Shell is not an option because Giraph requires execution of more than one task in parallel, something not supported by LocalTaskRunner.

          Edit: Fixed formatting.

          Show
          Andrew Purtell added a comment - - edited Attached is an updated patch that includes a Giraph smoke test, cribbed from a unit test. I wish I could report more progress. It took too much time just to figure out Maven was pulling in dependencies to some defunct java.net repo from ZooKeeper. Now I can't seem to get Maven to select the correct Hadoop artifact ("java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected"). Perhaps this is a familiar problem? Unfortunately just executing 'hadoop jar ...' using iTest Shell is not an option because Giraph requires execution of more than one task in parallel, something not supported by LocalTaskRunner. Edit: Fixed formatting.
          Hide
          Andrew Purtell added a comment -

          @Roman, ok I'll commit to getting that to you early next week.

          Show
          Andrew Purtell added a comment - @Roman, ok I'll commit to getting that to you early next week.
          Hide
          Roman Shaposhnik added a comment -

          Andrew, I have a first cut at Debian packaging. If you can help us with creating a smoke test for Giraph – it'll be all we need.

          Show
          Roman Shaposhnik added a comment - Andrew, I have a first cut at Debian packaging. If you can help us with creating a smoke test for Giraph – it'll be all we need.
          Hide
          Patrick Hunt added a comment -

          Giraph is about to graduate to TLP status, would be great to see this available in the next release!

          Show
          Patrick Hunt added a comment - Giraph is about to graduate to TLP status, would be great to see this available in the next release!
          Hide
          Peter Linnell added a comment -

          I had a look at the rpm side and it looks ready to commit. Thanks for the patch!

          Show
          Peter Linnell added a comment - I had a look at the rpm side and it looks ready to commit. Thanks for the patch!
          Hide
          Roman Shaposhnik added a comment -

          Excellent news! Thanks a million for the patch, Andrew! Giraph is a keeper. Now, given that I'm a Debian geek, I'll take a look at what it takes to add deb part to your patch. I'm sure James Page would also be interested

          Show
          Roman Shaposhnik added a comment - Excellent news! Thanks a million for the patch, Andrew! Giraph is a keeper. Now, given that I'm a Debian geek, I'll take a look at what it takes to add deb part to your patch. I'm sure James Page would also be interested

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development