Giraph
  1. Giraph
  2. GIRAPH-64

Create VertexRunner to make it easier to run users' computations

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, if a user wants to implement a Giraph algorithm by extending Vertex they must also write all the boilerplate around the Tool interface and bundle it with the Giraph jar (or get Giraph on the classpath and playing nice with the implementation). For example, what is included in the PageRankBenchmark and what Kohei has done: https://github.com/smly/java-Giraph-LabelPropagation It would be better if we had perhaps a Vertex implementation to be subclassed that already had all the standard Tooling included such that all one had to run would be (assuming the Giraph jar was already on the classpath):

      hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble -of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble

      This wouldn't work with every algorithm, but would be useful in a large number of cases.

      1. GIRAPH-64-2.patch
        14 kB
        Jakob Homan
      2. GIRAPH-64.patch
        15 kB
        Jakob Homan

        Activity

        Hide
        Owen O'Malley added a comment -

        I think that we should have a giraph script that works like:

        % giraph -cp my-awesome.jar -i jazz_input -v my.vertex -o jazz_output
        
        Show
        Owen O'Malley added a comment - I think that we should have a giraph script that works like: % giraph -cp my-awesome.jar -i jazz_input -v my.vertex -o jazz_output
        Hide
        Owen O'Malley added a comment -

        I guess I should expand on that a bit. So the standard giraph main should build up the job and submit it and just provide a cli option to override the vertex and automatically put both the giraph.jar and my-awesome.jar into the dist cache.

        Show
        Owen O'Malley added a comment - I guess I should expand on that a bit. So the standard giraph main should build up the job and submit it and just provide a cli option to override the vertex and automatically put both the giraph.jar and my-awesome.jar into the dist cache.
        Hide
        Jakob Homan added a comment -

        Here's a patch that introduces that old bin folder we all know and lo

        {ve|athe}

        . This also gives us the start of the package we'll need to think about making releases. Users no longer have to merge their code into the Giraph source to get it to run.
        With the new bin/giraph, assuming an implementation of Vertex such as (taken from the pagerankbenchmark, obviously):

        import java.util.Iterator;
        
        public class FirstVertex extends
            Vertex<LongWritable, DoubleWritable, DoubleWritable, DoubleWritable> {
            /** Configuration from Configurable */
            private Configuration conf;
        
            /** How many supersteps to run */
            public static String SUPERSTEP_COUNT = "PageRankBenchmark.superstepCount";
        
            @Override
            public void preApplication()
                throws InstantiationException, IllegalAccessException {
            }
        
            @Override
            public void postApplication() {
            }
        
            @Override
            public void preSuperstep() {
            }
        
            @Override
            public void compute(Iterator<DoubleWritable> msgIterator) {
                if (getSuperstep() >= 1) {
                    double sum = 0;
                    while (msgIterator.hasNext()) {
                        sum += msgIterator.next().get();
                    }
                    DoubleWritable vertexValue =
                        new DoubleWritable((0.15f / getNumVertices()) + 0.85f * sum);
                    setVertexValue(vertexValue);
                }
        
                if (getSuperstep() < getConf().getInt(SUPERSTEP_COUNT, -1)) {
                    long edges = getNumOutEdges();
                    sendMsgToAllEdges(new DoubleWritable(getVertexValue().get() / edges));
                } else {
                    voteToHalt();
                }
            }
        
          @Override
          public Configuration getConf() {
              return conf;
          }
        
          @Override
          public void setConf(Configuration conf) {
              this.conf = conf;
          }
        
        }

        one can run it via:

        bin/giraph \
        -DPageRankBenchmark.superstepCount=30 \
        -DpseduoRandomVertexReader.aggregateVertices=220 \
        -DpseduoRandomVertexReader.edgesPerVertex=37 \
        ~/kick-ass-vertex-1.0.jar giraph1.FirstVertex \
        -w 10 \
        -if org.apache.giraph.benchmark.PseudoRandomVertexInputFormat \
        -of org.apache.giraph.lib.JsonBase64VertexOutputFormat \
        -op output_path

        bin/giraph is heavily cribbed from mahout and pig, btw.
        Is there any reason the fatjar approach was taken other than expediency? This patch uses the fatjar approach for testing, but uses a standard lib folder approach for the actual package. I'd like to remove the fatjar entirely, eventually.

        This is a rough script and will need lots of enhancements as we go, but I think it's a good start.

        Show
        Jakob Homan added a comment - Here's a patch that introduces that old bin folder we all know and lo {ve|athe} . This also gives us the start of the package we'll need to think about making releases. Users no longer have to merge their code into the Giraph source to get it to run. With the new bin/giraph, assuming an implementation of Vertex such as (taken from the pagerankbenchmark, obviously): import java.util.Iterator; public class FirstVertex extends Vertex<LongWritable, DoubleWritable, DoubleWritable, DoubleWritable> { /** Configuration from Configurable */ private Configuration conf; /** How many supersteps to run */ public static String SUPERSTEP_COUNT = "PageRankBenchmark.superstepCount" ; @Override public void preApplication() throws InstantiationException, IllegalAccessException { } @Override public void postApplication() { } @Override public void preSuperstep() { } @Override public void compute(Iterator<DoubleWritable> msgIterator) { if (getSuperstep() >= 1) { double sum = 0; while (msgIterator.hasNext()) { sum += msgIterator.next().get(); } DoubleWritable vertexValue = new DoubleWritable((0.15f / getNumVertices()) + 0.85f * sum); setVertexValue(vertexValue); } if (getSuperstep() < getConf().getInt(SUPERSTEP_COUNT, -1)) { long edges = getNumOutEdges(); sendMsgToAllEdges( new DoubleWritable(getVertexValue().get() / edges)); } else { voteToHalt(); } } @Override public Configuration getConf() { return conf; } @Override public void setConf(Configuration conf) { this .conf = conf; } } one can run it via: bin/giraph \ -DPageRankBenchmark.superstepCount=30 \ -DpseduoRandomVertexReader.aggregateVertices=220 \ -DpseduoRandomVertexReader.edgesPerVertex=37 \ ~/kick-ass-vertex-1.0.jar giraph1.FirstVertex \ -w 10 \ -if org.apache.giraph.benchmark.PseudoRandomVertexInputFormat \ -of org.apache.giraph.lib.JsonBase64VertexOutputFormat \ -op output_path bin/giraph is heavily cribbed from mahout and pig, btw. Is there any reason the fatjar approach was taken other than expediency? This patch uses the fatjar approach for testing, but uses a standard lib folder approach for the actual package. I'd like to remove the fatjar entirely, eventually. This is a rough script and will need lots of enhancements as we go, but I think it's a good start.
        Hide
        Jakob Homan added a comment -

        oh, I left the -w in as a required argument (and its min/max value assigned to both values in the GiraphJob), but it should be moved out as an optional parameter in this or another patch. I imagine most jobs will want to set this themselves.

        Show
        Jakob Homan added a comment - oh, I left the -w in as a required argument (and its min/max value assigned to both values in the GiraphJob), but it should be moved out as an optional parameter in this or another patch. I imagine most jobs will want to set this themselves.
        Hide
        Avery Ching added a comment -

        Anyone want to take a crack at reviewing this? If not, I'll get to it tomorrow night.

        Show
        Avery Ching added a comment - Anyone want to take a crack at reviewing this? If not, I'll get to it tomorrow night.
        Hide
        Jakob Homan added a comment -

        Anyone want to take a crack at reviewing this? If not, I'll get to it tomorrow night.

        I'll be gone for the next few days, so I'm hoping it can get reviewed and committed. I think it's an important step forward. Once this is in, we generalize the provided examples (pagerank, shortest path, etc.) to work with any inputformat/outputformat and make Giraph immediately useful to anyone who downloads it and wants to play with a reasonably stored graph.

        I've verified it still merges cleanly after GIRAPH-36.

        Show
        Jakob Homan added a comment - Anyone want to take a crack at reviewing this? If not, I'll get to it tomorrow night. I'll be gone for the next few days, so I'm hoping it can get reviewed and committed. I think it's an important step forward. Once this is in, we generalize the provided examples (pagerank, shortest path, etc.) to work with any inputformat/outputformat and make Giraph immediately useful to anyone who downloads it and wants to play with a reasonably stored graph. I've verified it still merges cleanly after GIRAPH-36 .
        Hide
        Jake Mannix added a comment -

        I'd be happy to review this too (I wrote the original form of bin/mahout's MahoutDriver program, although it's gone and evolved over time from there) is there a way to automagically have ReviewBoard create a review for this patch, so we can see it in a GUI? My brain has atrophied greatly as IDE's have advanced. ;P

        Show
        Jake Mannix added a comment - I'd be happy to review this too (I wrote the original form of bin/mahout's MahoutDriver program, although it's gone and evolved over time from there) is there a way to automagically have ReviewBoard create a review for this patch, so we can see it in a GUI? My brain has atrophied greatly as IDE's have advanced. ;P
        Hide
        Jakob Homan added a comment -

        Not that I know of. I'm allergic to rb, but I've never seen one. In Hive land we have to manually create an instance for each issue. I just tried to create one for this and, while Giraph exists as a target, I get a 500 exception ("something broke!" - gee, that's helpful).

        Show
        Jakob Homan added a comment - Not that I know of. I'm allergic to rb, but I've never seen one. In Hive land we have to manually create an instance for each issue. I just tried to create one for this and, while Giraph exists as a target, I get a 500 exception ("something broke!" - gee, that's helpful).
        Hide
        Avery Ching added a comment -

        I don't know how to make reviewboard do it automatically, but it's fairly easy. I've done it a couple times. I can do it for this one as an example. Btw, Jake I'll let you review this one. I think your expertise will be useful.

        Show
        Avery Ching added a comment - I don't know how to make reviewboard do it automatically, but it's fairly easy. I've done it a couple times. I can do it for this one as an example. Btw, Jake I'll let you review this one. I think your expertise will be useful.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2691/
        -----------------------------------------------------------

        Review request for giraph and Jake Mannix.

        Summary
        -------

        This is filed on behalf of Jakob Homan.

        This addresses bug GIRAPH-64.
        https://issues.apache.org/jira/browse/GIRAPH-64

        Diffs


        http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1196921
        http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1196921

        Diff: https://reviews.apache.org/r/2691/diff

        Testing
        -------

        Don't know.

        Thanks,

        Avery

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2691/ ----------------------------------------------------------- Review request for giraph and Jake Mannix. Summary ------- This is filed on behalf of Jakob Homan. This addresses bug GIRAPH-64 . https://issues.apache.org/jira/browse/GIRAPH-64 Diffs http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1196921 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1196921 Diff: https://reviews.apache.org/r/2691/diff Testing ------- Don't know. Thanks, Avery
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2691/#review3021
        -----------------------------------------------------------

        Does this have the whole diff? Seems like it's missing stuff. Only has GraphMapper.java changes and pom.xml...

        • Jake

        On 2011-11-03 02:29:54, Avery Ching wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2691/

        -----------------------------------------------------------

        (Updated 2011-11-03 02:29:54)

        Review request for giraph and Jake Mannix.

        Summary

        -------

        This is filed on behalf of Jakob Homan.

        This addresses bug GIRAPH-64.

        https://issues.apache.org/jira/browse/GIRAPH-64

        Diffs

        -----

        http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1196921

        http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1196921

        Diff: https://reviews.apache.org/r/2691/diff

        Testing

        -------

        Don't know.

        Thanks,

        Avery

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2691/#review3021 ----------------------------------------------------------- Does this have the whole diff? Seems like it's missing stuff. Only has GraphMapper.java changes and pom.xml... Jake On 2011-11-03 02:29:54, Avery Ching wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2691/ ----------------------------------------------------------- (Updated 2011-11-03 02:29:54) Review request for giraph and Jake Mannix. Summary ------- This is filed on behalf of Jakob Homan. This addresses bug GIRAPH-64 . https://issues.apache.org/jira/browse/GIRAPH-64 Diffs ----- http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1196921 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1196921 Diff: https://reviews.apache.org/r/2691/diff Testing ------- Don't know. Thanks, Avery
        Hide
        Jakob Homan added a comment -

        Yeah, rb is missing the files. My allergy to it grows.

        Show
        Jakob Homan added a comment - Yeah, rb is missing the files. My allergy to it grows.
        Hide
        Jakob Homan added a comment -

        ok, new rb request: https://reviews.apache.org/r/2700/diff/#index_header Apparently rb isn't smart enough to be able to use a patch generated from git. The mind boggles.

        Show
        Jakob Homan added a comment - ok, new rb request: https://reviews.apache.org/r/2700/diff/#index_header Apparently rb isn't smart enough to be able to use a patch generated from git. The mind boggles.
        Hide
        Avery Ching added a comment -

        Sorry, that was my bad. Forgot to do a 'svn st -u' before the 'svn diff'. Thanks for the updated rb Jakob. Btw, I'm a huge rb fan =). If you specify the GIRAPH-64 in the 'Bugs:' section, it will automatically post to this issue.

        Show
        Avery Ching added a comment - Sorry, that was my bad. Forgot to do a 'svn st -u' before the 'svn diff'. Thanks for the updated rb Jakob. Btw, I'm a huge rb fan =). If you specify the GIRAPH-64 in the 'Bugs:' section, it will automatically post to this issue.
        Hide
        Jake Mannix added a comment -

        Hey Jakob, I put some comments in rb, but looks like they didn't crosspost over to here (due to the missing Bugs: section not having been pulled out).

        There was one bug/typo which should be fixed, but other than that, the rest of my comments were mostly "wishlist", so let's zoom and get this in!

        +1

        Show
        Jake Mannix added a comment - Hey Jakob, I put some comments in rb, but looks like they didn't crosspost over to here (due to the missing Bugs: section not having been pulled out). There was one bug/typo which should be fixed, but other than that, the rest of my comments were mostly "wishlist", so let's zoom and get this in! +1
        Hide
        Hyunsik Choi added a comment -

        The patch looks nice!
        I have really wanted this feature!

        However, when I applied the patch to trunk and executed 'mvn assembly:assembly', it caused some error as follows:

        [INFO] Compiling 14 source files to /home/hyunsik/Code/giraph/giraph-review/target/test-classes
        [INFO] 
        [INFO] --- maven-surefire-plugin:2.6:test (default-test) @ giraph ---
        [INFO] Tests are skipped.
        [INFO] 
        [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ giraph ---
        [INFO] Building jar: /home/hyunsik/Code/giraph/giraph-review/target/giraph-0.70.jar
        [INFO] 
        [INFO] --- maven-assembly-plugin:2.2:single (make-assembly) @ giraph ---
        [INFO] Reading assembly descriptor: /home/hyunsik/Code/giraph/giraph-review/src/main/assembly/assembly.xml
        [WARNING] NOTE: Currently, inclusion of module dependencies may produce unpredictable results if a version conflict occurs.
        [INFO] Building tar : /home/hyunsik/Code/giraph/giraph-review/target/giraph-0.70-bin.tar.gz
        [WARNING] Entry: giraph-0.70/src/test/java/org/apache/giraph/lib/TestLongDoubleDoubleAdjacencyListVertexInputFormat.java longer than 100 characters.
        [WARNING] Resulting tar file can only be processed successfully by GNU compatible tar commands
        [WARNING] Entry: giraph-0.70/src/test/java/org/apache/giraph/lib/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java longer than 100 characters.
        [INFO] 
        [INFO] <<< maven-assembly-plugin:2.2:assembly (default-cli) @ giraph <<<
        [INFO] 
        [INFO] --- maven-assembly-plugin:2.2:assembly (default-cli) @ giraph ---
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 11.818s
        [INFO] Finished at: Fri Nov 04 00:36:35 KST 2011
        [INFO] Final Memory: 21M/413M
        [INFO] ------------------------------------------------------------------------
        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2:assembly (default-cli) on project giraph: Error reading assemblies: No assembly descriptors found. -> [Help 1]
        [ERROR] 
        [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
        [ERROR] Re-run Maven using the -X switch to enable full debug logging.
        [ERROR] 
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
        

        I'm using mvn 3.0.3 and sun-jdk 1.6.0_26.
        I have surveyed this problem. I still don't know whether this is some bug or my fault.

        Any one knows what is problem?

        Show
        Hyunsik Choi added a comment - The patch looks nice! I have really wanted this feature! However, when I applied the patch to trunk and executed 'mvn assembly:assembly', it caused some error as follows: [INFO] Compiling 14 source files to /home/hyunsik/Code/giraph/giraph-review/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.6:test (default-test) @ giraph --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ giraph --- [INFO] Building jar: /home/hyunsik/Code/giraph/giraph-review/target/giraph-0.70.jar [INFO] [INFO] --- maven-assembly-plugin:2.2:single (make-assembly) @ giraph --- [INFO] Reading assembly descriptor: /home/hyunsik/Code/giraph/giraph-review/src/main/assembly/assembly.xml [WARNING] NOTE: Currently, inclusion of module dependencies may produce unpredictable results if a version conflict occurs. [INFO] Building tar : /home/hyunsik/Code/giraph/giraph-review/target/giraph-0.70-bin.tar.gz [WARNING] Entry: giraph-0.70/src/test/java/org/apache/giraph/lib/TestLongDoubleDoubleAdjacencyListVertexInputFormat.java longer than 100 characters. [WARNING] Resulting tar file can only be processed successfully by GNU compatible tar commands [WARNING] Entry: giraph-0.70/src/test/java/org/apache/giraph/lib/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java longer than 100 characters. [INFO] [INFO] <<< maven-assembly-plugin:2.2:assembly (default-cli) @ giraph <<< [INFO] [INFO] --- maven-assembly-plugin:2.2:assembly (default-cli) @ giraph --- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 11.818s [INFO] Finished at: Fri Nov 04 00:36:35 KST 2011 [INFO] Final Memory: 21M/413M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2:assembly (default-cli) on project giraph: Error reading assemblies: No assembly descriptors found. -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException I'm using mvn 3.0.3 and sun-jdk 1.6.0_26. I have surveyed this problem. I still don't know whether this is some bug or my fault. Any one knows what is problem?
        Hide
        Avery Ching added a comment -

        Jakob, pulled the diff from your last rb (updating is better than using a new reviewboard to see changes fyi). I got the following issues on 'mvn package'

        mvn package
        [INFO] Scanning for projects...
        [INFO]
        [INFO] ------------------------------------------------------------------------
        [INFO] Building Apache Incubator Giraph 0.70
        [INFO] ------------------------------------------------------------------------
        [INFO]
        [INFO] — maven-enforcer-plugin:1.0.1:enforce (enforce-maven) @ giraph —
        [INFO]
        [INFO] — maven-resources-plugin:2.4.3:resources (default-resources) @ giraph —
        [INFO] Using 'UTF-8' encoding to copy filtered resources.
        [INFO] skip non existing resourceDirectory /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/resources
        [INFO]
        [INFO] — maven-compiler-plugin:2.3.2:compile (default-compile) @ giraph —
        [INFO] Compiling 87 source files to /Users/aching/Documents/workspace/Trunk Apache Giraph/target/classes
        [INFO] -------------------------------------------------------------
        [ERROR] COMPILATION ERROR :
        [INFO] -------------------------------------------------------------
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[136,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[138,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[139,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[140,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[141,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[142,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[143,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[144,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[145,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[146,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[147,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[148,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[149,0] class, interface, or enum expected
        [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java:[150,0] class, interface, or enum expected
        [INFO] 14 errors
        [INFO] -------------------------------------------------------------
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE

        Show
        Avery Ching added a comment - Jakob, pulled the diff from your last rb (updating is better than using a new reviewboard to see changes fyi). I got the following issues on 'mvn package' mvn package [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Apache Incubator Giraph 0.70 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] — maven-enforcer-plugin:1.0.1:enforce (enforce-maven) @ giraph — [INFO] [INFO] — maven-resources-plugin:2.4.3:resources (default-resources) @ giraph — [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/resources [INFO] [INFO] — maven-compiler-plugin:2.3.2:compile (default-compile) @ giraph — [INFO] Compiling 87 source files to /Users/aching/Documents/workspace/Trunk Apache Giraph/target/classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [136,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [138,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [139,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [140,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [141,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [142,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [143,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [144,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [145,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [146,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [147,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [148,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [149,0] class, interface, or enum expected [ERROR] /Users/aching/Documents/workspace/Trunk Apache Giraph/src/main/java/org/apache/giraph/GiraphRunner.java: [150,0] class, interface, or enum expected [INFO] 14 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE
        Hide
        Jakob Homan added a comment -

        Hey guys, sorry for the delay. Have been traveling (and will be this week too). Will update per jake's comments tomorrow and upload new patch. Not sure about the compile errors though. What environments are you running? Maybe platform problem?

        Show
        Jakob Homan added a comment - Hey guys, sorry for the delay. Have been traveling (and will be this week too). Will update per jake's comments tomorrow and upload new patch. Not sure about the compile errors though. What environments are you running? Maybe platform problem?
        Hide
        Avery Ching added a comment -

        $ mvn --version
        Apache Maven 3.0.3 (r1075438; 2011-02-28 09:31:09-0800)
        Maven home: /usr/share/maven
        Java version: 1.6.0_26, vendor: Apple Inc.
        Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
        Default locale: en_US, platform encoding: MacRoman
        OS name: "mac os x", version: "10.6.8", arch: "x86_64", family: "mac"

        Show
        Avery Ching added a comment - $ mvn --version Apache Maven 3.0.3 (r1075438; 2011-02-28 09:31:09-0800) Maven home: /usr/share/maven Java version: 1.6.0_26, vendor: Apple Inc. Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home Default locale: en_US, platform encoding: MacRoman OS name: "mac os x", version: "10.6.8", arch: "x86_64", family: "mac"
        Hide
        Hyunsik Choi added a comment -

        In my case, 'mvn package' is ok, but 'mvn assembly:assembly' incurs the error as I mentioned above.

        hyunsik@code:~$ mvn --version
        Apache Maven 3.0.3 (r1075438; 2011-03-01 02:31:09+0900)
        Maven home: /home/hyunsik/Local/maven-3
        Java version: 1.6.0_26, vendor: Sun Microsystems Inc.
        Java home: /usr/lib/jvm/java-6-sun-1.6.0.26/jre
        Default locale: en_US, platform encoding: UTF-8
        OS name: "linux", version: "3.0.0-12-generic", arch: "amd64", family: "unix"
        
        Show
        Hyunsik Choi added a comment - In my case, 'mvn package' is ok, but 'mvn assembly:assembly' incurs the error as I mentioned above. hyunsik@code:~$ mvn --version Apache Maven 3.0.3 (r1075438; 2011-03-01 02:31:09+0900) Maven home: /home/hyunsik/Local/maven-3 Java version: 1.6.0_26, vendor: Sun Microsystems Inc. Java home: /usr/lib/jvm/java-6-sun-1.6.0.26/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux" , version: "3.0.0-12- generic " , arch: "amd64" , family: "unix"
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2700/
        -----------------------------------------------------------

        (Updated 2011-11-12 02:03:20.188294)

        Review request for giraph.

        Summary
        -------

        create giraphrunner

        This addresses bug GIRAPH-64.
        https://issues.apache.org/jira/browse/GIRAPH-64

        Diffs


        /trunk/bin/giraph PRE-CREATION
        /trunk/pom.xml 1196926
        /trunk/src/main/assembly/assembly.xml PRE-CREATION
        /trunk/src/main/java/org/apache/giraph/GiraphRunner.java PRE-CREATION
        /trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1196926

        Diff: https://reviews.apache.org/r/2700/diff

        Testing
        -------

        Thanks,

        Jakob

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2700/ ----------------------------------------------------------- (Updated 2011-11-12 02:03:20.188294) Review request for giraph. Summary ------- create giraphrunner This addresses bug GIRAPH-64 . https://issues.apache.org/jira/browse/GIRAPH-64 Diffs /trunk/bin/giraph PRE-CREATION /trunk/pom.xml 1196926 /trunk/src/main/assembly/assembly.xml PRE-CREATION /trunk/src/main/java/org/apache/giraph/GiraphRunner.java PRE-CREATION /trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1196926 Diff: https://reviews.apache.org/r/2700/diff Testing ------- Thanks, Jakob
        Hide
        Jakob Homan added a comment -

        Updated patch for Jake's comments. Outputformat is no longer a required param (though will open a new JIRA to make it so; defaulting to /dev/null is bad) and inputpath is correctly used.

        Hyunsik: mvn assembly:single (or assembly, but that's been deprecated for single) won't work since we're defining separate assemblies (either the regular or fat jar), and no default is provided. I tried adding one, but it generated basically an empty .tar.gz, which is pretty useless. It might be better to error out than create an empty file. This will go away once we get rid of the fatjar.

        Avery: I can't reproduce what you saw and I'm a bit confused because the error is referencing lines 136-150 in GiraphRunner, but that file is only 117 lines long. Can you try it again with this patch on a fresh checkout?

        Show
        Jakob Homan added a comment - Updated patch for Jake's comments. Outputformat is no longer a required param (though will open a new JIRA to make it so; defaulting to /dev/null is bad) and inputpath is correctly used. Hyunsik: mvn assembly:single (or assembly, but that's been deprecated for single) won't work since we're defining separate assemblies (either the regular or fat jar), and no default is provided. I tried adding one, but it generated basically an empty .tar.gz, which is pretty useless. It might be better to error out than create an empty file. This will go away once we get rid of the fatjar. Avery: I can't reproduce what you saw and I'm a bit confused because the error is referencing lines 136-150 in GiraphRunner, but that file is only 117 lines long. Can you try it again with this patch on a fresh checkout?
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-11-03 05:32:49, Jake Mannix wrote:

        > It would be super cool if, like in Mahout and Hadoop, we could print out the set of Vertex impls / algorithms / actions available. In Mahout, we have a properties file which we keep updating. Even nicer (maybe?) would be to just search the classpath for all classes which implement BasicVertex, and list them. Even cooler: have a new abstract method on BasicVertex "String getHelp()" which GiraphRunner could run via reflection to get the help output for the task.

        Jakob Homan wrote:

        Absolutely. Another JIRA?

        abstract method or annotation?

        Nice thing about annotations is that you can hook into them during compilation and produce docs...

        • Dmitriy

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2700/#review3023
        -----------------------------------------------------------

        On 2011-11-12 02:03:20, Jakob Homan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2700/

        -----------------------------------------------------------

        (Updated 2011-11-12 02:03:20)

        Review request for giraph.

        Summary

        -------

        create giraphrunner

        This addresses bug GIRAPH-64.

        https://issues.apache.org/jira/browse/GIRAPH-64

        Diffs

        -----

        /trunk/bin/giraph PRE-CREATION

        /trunk/pom.xml 1196926

        /trunk/src/main/assembly/assembly.xml PRE-CREATION

        /trunk/src/main/java/org/apache/giraph/GiraphRunner.java PRE-CREATION

        /trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1196926

        Diff: https://reviews.apache.org/r/2700/diff

        Testing

        -------

        Thanks,

        Jakob

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-11-03 05:32:49, Jake Mannix wrote: > It would be super cool if, like in Mahout and Hadoop, we could print out the set of Vertex impls / algorithms / actions available. In Mahout, we have a properties file which we keep updating. Even nicer (maybe?) would be to just search the classpath for all classes which implement BasicVertex, and list them. Even cooler: have a new abstract method on BasicVertex "String getHelp()" which GiraphRunner could run via reflection to get the help output for the task. Jakob Homan wrote: Absolutely. Another JIRA? abstract method or annotation? Nice thing about annotations is that you can hook into them during compilation and produce docs... Dmitriy ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2700/#review3023 ----------------------------------------------------------- On 2011-11-12 02:03:20, Jakob Homan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2700/ ----------------------------------------------------------- (Updated 2011-11-12 02:03:20) Review request for giraph. Summary ------- create giraphrunner This addresses bug GIRAPH-64 . https://issues.apache.org/jira/browse/GIRAPH-64 Diffs ----- /trunk/bin/giraph PRE-CREATION /trunk/pom.xml 1196926 /trunk/src/main/assembly/assembly.xml PRE-CREATION /trunk/src/main/java/org/apache/giraph/GiraphRunner.java PRE-CREATION /trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1196926 Diff: https://reviews.apache.org/r/2700/diff Testing ------- Thanks, Jakob
        Hide
        Avery Ching added a comment -

        Jakob, it now builds fine for me. Not sure what happened. Local unittest passed to, didn't try MR unittests.

        But I did find this warning at the end of 'mvn package'

        [WARNING] Entry: giraph-0.70/src/test/java/org/apache/giraph/lib/TestLongDoubleDoubleAdjacencyListVertexInputFormat.java longer than 100 characters.
        [WARNING] Resulting tar file can only be processed successfully by GNU compatible tar commands
        [WARNING] Entry: giraph-0.70/src/test/java/org/apache/giraph/lib/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java longer than 100 characters.

        Although, it is unrelated to this patch.

        Show
        Avery Ching added a comment - Jakob, it now builds fine for me. Not sure what happened. Local unittest passed to, didn't try MR unittests. But I did find this warning at the end of 'mvn package' [WARNING] Entry: giraph-0.70/src/test/java/org/apache/giraph/lib/TestLongDoubleDoubleAdjacencyListVertexInputFormat.java longer than 100 characters. [WARNING] Resulting tar file can only be processed successfully by GNU compatible tar commands [WARNING] Entry: giraph-0.70/src/test/java/org/apache/giraph/lib/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java longer than 100 characters. Although, it is unrelated to this patch.
        Hide
        Jakob Homan added a comment -

        The warning is an artifact of our silly names; at the moment there's no way around it but it shouldn't cause a problem. I've verified the runner works on our clusters here. Since it's just adding a runner (with small tweaks to how zk is dealt with), there shouldn't be the possibility of screwing up the tests on a real cluster.

        Is this ready to be committed?

        Show
        Jakob Homan added a comment - The warning is an artifact of our silly names; at the moment there's no way around it but it shouldn't cause a problem. I've verified the runner works on our clusters here. Since it's just adding a runner (with small tweaks to how zk is dealt with), there shouldn't be the possibility of screwing up the tests on a real cluster. Is this ready to be committed?
        Hide
        Avery Ching added a comment -

        Given that Jake is doing the review, I'll let him decide on this one.

        Show
        Avery Ching added a comment - Given that Jake is doing the review, I'll let him decide on this one.
        Hide
        Jakob Homan added a comment -

        OK, Jake had +1ed it earlier and except for addressing his comments, it's unchanged, so I'll go ahead and commit.

        Show
        Jakob Homan added a comment - OK, Jake had +1ed it earlier and except for addressing his comments, it's unchanged, so I'll go ahead and commit.
        Hide
        Jakob Homan added a comment -

        I've committed this. Resolving as fixed. Thanks for the reviews!

        Show
        Jakob Homan added a comment - I've committed this. Resolving as fixed. Thanks for the reviews!
        Hide
        Jake Mannix added a comment -

        woohoo!

        Show
        Jake Mannix added a comment - woohoo!
        Hide
        Hudson added a comment -

        Integrated in Giraph-trunk-Commit #30 (See https://builds.apache.org/job/Giraph-trunk-Commit/30/)
        GIRAPH-64. Create VertexRunner to make it easier to run users' computations.

        jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1201899
        Files :

        • /incubator/giraph/trunk/CHANGELOG
        • /incubator/giraph/trunk/bin
        • /incubator/giraph/trunk/bin/giraph
        • /incubator/giraph/trunk/pom.xml
        • /incubator/giraph/trunk/src/main/assembly
        • /incubator/giraph/trunk/src/main/assembly/assembly.xml
        • /incubator/giraph/trunk/src/main/java/org/apache/giraph/GiraphRunner.java
        • /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
        Show
        Hudson added a comment - Integrated in Giraph-trunk-Commit #30 (See https://builds.apache.org/job/Giraph-trunk-Commit/30/ ) GIRAPH-64 . Create VertexRunner to make it easier to run users' computations. jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1201899 Files : /incubator/giraph/trunk/CHANGELOG /incubator/giraph/trunk/bin /incubator/giraph/trunk/bin/giraph /incubator/giraph/trunk/pom.xml /incubator/giraph/trunk/src/main/assembly /incubator/giraph/trunk/src/main/assembly/assembly.xml /incubator/giraph/trunk/src/main/java/org/apache/giraph/GiraphRunner.java /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java

          People

          • Assignee:
            Jakob Homan
            Reporter:
            Jakob Homan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development