Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.1
    • Component/s: gora-core
    • Labels:

      Description

      Now Gora supports Map Reduce Framework. With this umbrella issue we try to develop Apache Spark Backend.

        Issue Links

          Activity

          Hide
          kamaci Furkan KAMACI added a comment -

          Talat UYARER I would like to apply GSoC for this issue. I'll share my application proposal with you.

          Show
          kamaci Furkan KAMACI added a comment - Talat UYARER I would like to apply GSoC for this issue. I'll share my application proposal with you.
          Hide
          kamaci Furkan KAMACI added a comment -

          Here is my repository for this issue: https://github.com/kamaci/gora

          Show
          kamaci Furkan KAMACI added a comment - Here is my repository for this issue: https://github.com/kamaci/gora
          Hide
          lewismc Lewis John McGibbney added a comment -

          Hi Furkan KAMACI can you please send a pull request so we can undertake further code review, test, etc?
          Thank you

          Show
          lewismc Lewis John McGibbney added a comment - Hi Furkan KAMACI can you please send a pull request so we can undertake further code review, test, etc? Thank you
          Hide
          kamaci Furkan KAMACI added a comment -

          I've ignored word count test at here: https://github.com/kamaci/gora/blob/master/gora-hbase/src/test/java/org/apache/gora/hbase/mapreduce/TestHBaseStoreWordCount.java#L65 due to the connection error it throws. I've attached full stack trace as connection_refused.txt

          Show
          kamaci Furkan KAMACI added a comment - I've ignored word count test at here: https://github.com/kamaci/gora/blob/master/gora-hbase/src/test/java/org/apache/gora/hbase/mapreduce/TestHBaseStoreWordCount.java#L65 due to the connection error it throws. I've attached full stack trace as connection_refused.txt
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kamaci opened a pull request:

          https://github.com/apache/gora/pull/29

          Spark Backend Support for Gora (GORA-386)

          Pull request of the work for GSoC project (Spark Backend Support for Gora) for Apache Gora.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/kamaci/gora master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/gora/pull/29.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #29


          commit 9c2d225d04cfa746244373fa661a1aa6f03250bb
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-06-27T22:57:51Z

          • JavaPairRDD support for GoraInputFormat.

          commit ef68cead273324797cf292dbe6da18ee3fd819cb
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-06-28T14:17:52Z

          • GoraSpark.java initialize method renamed to initializeInput.
          • Architectural change is made.

          commit 8fbdef7de963defb1387f0fbbe9e56506b627ba3
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-06-28T15:49:13Z

          • Serialization support via Kyro.

          commit 5644a21c5c5678611c7cef4e2c922643088951b8
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-06-28T23:10:26Z

          • map function is implemented as like in LogAnalytics.java

          commit 445edb12c35f867e7b81919ddfb509dd8c179ba3
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-06-28T23:26:41Z

          • map was not collecting the counts and it is fixed.

          commit 81af4d3afcba4633d0c5d06ead9b4256ea60862f
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-06-29T15:33:38Z

          • GoraSpark.java initializeInput method renamed to initialize
          • reduce part is added to example.

          commit 80c0c26d8fb6a9a84ea39f5aa96cc343b4546266
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-06-29T16:26:28Z

          Organizing codes
          Documentation

          commit c111e6290fa16d2cc560eb29dbc07eb2f8b7734b
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-07-15T17:23:51Z

          • Writing to Hbase via Spark is implemented.

          commit cf6e76585b18ff926f8e7e4928f2d7cd28ac7c6f
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-07-16T16:38:37Z

          • Added dependency to write output to Solr.

          commit 219557002f0e33ef6d0e2bb49471d24fb867b0ac
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-08-17T18:21:43Z

          Code is organized at LogAnalyticsSpark.java.

          commit 62be0c312927e3ea4962eb966689b49dc1fcebce
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-08-17T19:25:12Z

          • GoraSparkEngine.java architecture is improved.

          commit 8584911176712f63e316707ace23a12060588cdc
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-08-17T20:25:15Z

          GoraSparkEngine.java setOutput method renamed to generateOutputConf.
          Added a new method to GoraSparkEngine.java which creates a job and returns necessary conf for Spark to use.

          commit 92b71a6d2c95492621a56e48470e951e982cc34f
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-08-17T20:34:40Z

          Unnecessary reuseObjects is removed.

          commit 8893fd5c6c53b19925c99668d24df23c52e70b1e
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-08-21T12:58:21Z

          Unnecessary imports are removed from LogAnalyticsSpark.java

          commit 9cff33598231e9cf54060cecf1ccacdf62e64c5b
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-08-22T13:46:55Z

          Minor improvements for LogAnalyticsSpark.java

          commit e817a7f0eeb2efb98937efb92701705ec80891b9
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-08-22T14:31:57Z

          Spark engine word count tests are implemented.
          Due to a connection problem, spark engine word count tests are ignored.

          commit 8f1acc6d4ef6c192e8fc06287558b7bc7c39b040
          Author: Furkan KAMACI <furkankamaci@gmail.com>
          Date: 2015-08-22T19:06:42Z

          Minor change at SparkWordCount.java


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kamaci opened a pull request: https://github.com/apache/gora/pull/29 Spark Backend Support for Gora ( GORA-386 ) Pull request of the work for GSoC project (Spark Backend Support for Gora) for Apache Gora. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kamaci/gora master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/gora/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit 9c2d225d04cfa746244373fa661a1aa6f03250bb Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-06-27T22:57:51Z JavaPairRDD support for GoraInputFormat. commit ef68cead273324797cf292dbe6da18ee3fd819cb Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-06-28T14:17:52Z GoraSpark.java initialize method renamed to initializeInput. Architectural change is made. commit 8fbdef7de963defb1387f0fbbe9e56506b627ba3 Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-06-28T15:49:13Z Serialization support via Kyro. commit 5644a21c5c5678611c7cef4e2c922643088951b8 Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-06-28T23:10:26Z map function is implemented as like in LogAnalytics.java commit 445edb12c35f867e7b81919ddfb509dd8c179ba3 Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-06-28T23:26:41Z map was not collecting the counts and it is fixed. commit 81af4d3afcba4633d0c5d06ead9b4256ea60862f Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-06-29T15:33:38Z GoraSpark.java initializeInput method renamed to initialize reduce part is added to example. commit 80c0c26d8fb6a9a84ea39f5aa96cc343b4546266 Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-06-29T16:26:28Z Organizing codes Documentation commit c111e6290fa16d2cc560eb29dbc07eb2f8b7734b Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-07-15T17:23:51Z Writing to Hbase via Spark is implemented. commit cf6e76585b18ff926f8e7e4928f2d7cd28ac7c6f Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-07-16T16:38:37Z Added dependency to write output to Solr. commit 219557002f0e33ef6d0e2bb49471d24fb867b0ac Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-08-17T18:21:43Z Code is organized at LogAnalyticsSpark.java. commit 62be0c312927e3ea4962eb966689b49dc1fcebce Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-08-17T19:25:12Z GoraSparkEngine.java architecture is improved. commit 8584911176712f63e316707ace23a12060588cdc Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-08-17T20:25:15Z GoraSparkEngine.java setOutput method renamed to generateOutputConf. Added a new method to GoraSparkEngine.java which creates a job and returns necessary conf for Spark to use. commit 92b71a6d2c95492621a56e48470e951e982cc34f Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-08-17T20:34:40Z Unnecessary reuseObjects is removed. commit 8893fd5c6c53b19925c99668d24df23c52e70b1e Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-08-21T12:58:21Z Unnecessary imports are removed from LogAnalyticsSpark.java commit 9cff33598231e9cf54060cecf1ccacdf62e64c5b Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-08-22T13:46:55Z Minor improvements for LogAnalyticsSpark.java commit e817a7f0eeb2efb98937efb92701705ec80891b9 Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-08-22T14:31:57Z Spark engine word count tests are implemented. Due to a connection problem, spark engine word count tests are ignored. commit 8f1acc6d4ef6c192e8fc06287558b7bc7c39b040 Author: Furkan KAMACI <furkankamaci@gmail.com> Date: 2015-08-22T19:06:42Z Minor change at SparkWordCount.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38060140

          — Diff: gora-core/pom.xml —
          @@ -141,6 +141,13 @@
          <artifactId>guava</artifactId>
          </dependency>

          + <!-- Spark dependency -->
          — End diff –

          Dependency definitions usually happen within parent pom.xml. Modules then inherit the dependency without versions. This also enables us best opportunity to synchronize dependencies across modules.

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38060140 — Diff: gora-core/pom.xml — @@ -141,6 +141,13 @@ <artifactId>guava</artifactId> </dependency> + <!-- Spark dependency --> — End diff – Dependency definitions usually happen within parent pom.xml. Modules then inherit the dependency without versions. This also enables us best opportunity to synchronize dependencies across modules.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38060312

          — Diff: pom.xml —
          @@ -893,6 +893,10 @@
          <groupId>ant</groupId>
          <artifactId>ant</artifactId>
          </exclusion>
          + <exclusion>
          — End diff –

          Why is this required?

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38060312 — Diff: pom.xml — @@ -893,6 +893,10 @@ <groupId>ant</groupId> <artifactId>ant</artifactId> </exclusion> + <exclusion> — End diff – Why is this required?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38060355

          — Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java —
          @@ -0,0 +1,200 @@
          +/**
          — End diff –

          Can you please add an entry to https://github.com/apache/gora/blob/master/bin/gora which enables us to run this very easily from the command line? It will help with testing and validation of your code.

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38060355 — Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java — @@ -0,0 +1,200 @@ +/** — End diff – Can you please add an entry to https://github.com/apache/gora/blob/master/bin/gora which enables us to run this very easily from the command line? It will help with testing and validation of your code.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38060396

          — Diff: gora-tutorial/pom.xml —
          @@ -108,6 +108,11 @@
          <artifactId>gora-cassandra</artifactId>
          </dependency>

          + <dependency>
          — End diff –

          Thanks for adding this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38060396 — Diff: gora-tutorial/pom.xml — @@ -108,6 +108,11 @@ <artifactId>gora-cassandra</artifactId> </dependency> + <dependency> — End diff – Thanks for adding this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38060417

          — Diff: gora-hbase/src/test/java/org/apache/gora/hbase/mapreduce/TestHBaseStoreWordCount.java —
          @@ -58,4 +59,11 @@ public void testWordCount() throws Exception

          { MapReduceTestUtils.testWordCount(cluster.getConf(), webPageStore, tokenStore); }

          + //todo fix config
          + @Ignore
          — End diff –

          I am OK for us ignoring the tests at this point in time. It is high desired though that we ensure the issue you've described at http://www.mail-archive.com/dev%40gora.apache.org/msg06020.html is addressed in a separate issue.

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38060417 — Diff: gora-hbase/src/test/java/org/apache/gora/hbase/mapreduce/TestHBaseStoreWordCount.java — @@ -58,4 +59,11 @@ public void testWordCount() throws Exception { MapReduceTestUtils.testWordCount(cluster.getConf(), webPageStore, tokenStore); } + //todo fix config + @Ignore — End diff – I am OK for us ignoring the tests at this point in time. It is high desired though that we ensure the issue you've described at http://www.mail-archive.com/dev%40gora.apache.org/msg06020.html is addressed in a separate issue.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38060425

          — Diff: gora-mongodb/src/test/java/org/apache/gora/mongodb/mapreduce/TestMongoStoreWordCount.java —
          @@ -55,4 +56,12 @@ public void testWordCount() throws Exception

          { webPageStore, tokenStore); }

          + //todo fix config
          + @Ignore
          — End diff –

          I am OK for us ignoring the tests at this point in time. It is high desired though that we ensure the issue you've described at http://www.mail-archive.com/dev%40gora.apache.org/msg06020.html is addressed in a separate issue.

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38060425 — Diff: gora-mongodb/src/test/java/org/apache/gora/mongodb/mapreduce/TestMongoStoreWordCount.java — @@ -55,4 +56,12 @@ public void testWordCount() throws Exception { webPageStore, tokenStore); } + //todo fix config + @Ignore — End diff – I am OK for us ignoring the tests at this point in time. It is high desired though that we ensure the issue you've described at http://www.mail-archive.com/dev%40gora.apache.org/msg06020.html is addressed in a separate issue.
          Hide
          lewismc Lewis John McGibbney added a comment -

          Furkan KAMACI please address the above comments when you have the time. Thank you.

          Show
          lewismc Lewis John McGibbney added a comment - Furkan KAMACI please address the above comments when you have the time. Thank you.
          Hide
          lewismc Lewis John McGibbney added a comment -

          Can you also add a page to the current documentation detailing your additions e.g. GoraSparkEngine?
          This would go in http://svn.apache.org/repos/asf/gora/site/trunk/content/current/

          Show
          lewismc Lewis John McGibbney added a comment - Can you also add a page to the current documentation detailing your additions e.g. GoraSparkEngine? This would go in http://svn.apache.org/repos/asf/gora/site/trunk/content/current/
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kamaci commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38144036

          — Diff: gora-core/pom.xml —
          @@ -141,6 +141,13 @@
          <artifactId>guava</artifactId>
          </dependency>

          + <!-- Spark dependency -->
          — End diff –

          Spark version should be placed at parent pom and it's done.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kamaci commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38144036 — Diff: gora-core/pom.xml — @@ -141,6 +141,13 @@ <artifactId>guava</artifactId> </dependency> + <!-- Spark dependency --> — End diff – Spark version should be placed at parent pom and it's done.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38146393

          — Diff: pom.xml —
          @@ -644,6 +644,7 @@
          <commons-io.version>1.3.2</commons-io.version>
          <restlet.version>2.3.1</restlet.version>

          + <spark.version>1.3.1</spark.version>
          — End diff –

          Any reason to not use Spark version 1.4.1?

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38146393 — Diff: pom.xml — @@ -644,6 +644,7 @@ <commons-io.version>1.3.2</commons-io.version> <restlet.version>2.3.1</restlet.version> + <spark.version>1.3.1</spark.version> — End diff – Any reason to not use Spark version 1.4.1?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kamaci commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38147131

          — Diff: pom.xml —
          @@ -893,6 +893,10 @@
          <groupId>ant</groupId>
          <artifactId>ant</artifactId>
          </exclusion>
          + <exclusion>
          — End diff –

          servlet dependency is excluded from spark, not ant. Without excluding it throws a java security exception. Motivation for it: https://issues.apache.org/jira/browse/SPARK-1693

          Show
          githubbot ASF GitHub Bot added a comment - Github user kamaci commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38147131 — Diff: pom.xml — @@ -893,6 +893,10 @@ <groupId>ant</groupId> <artifactId>ant</artifactId> </exclusion> + <exclusion> — End diff – servlet dependency is excluded from spark, not ant. Without excluding it throws a java security exception. Motivation for it: https://issues.apache.org/jira/browse/SPARK-1693
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kamaci commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38147685

          — Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java —
          @@ -0,0 +1,200 @@
          +/**
          — End diff –

          LogAnalyticsSpark is let to run from command line by passing "loganalyticsspark" argument.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kamaci commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38147685 — Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java — @@ -0,0 +1,200 @@ +/** — End diff – LogAnalyticsSpark is let to run from command line by passing "loganalyticsspark" argument.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38148077

          — Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java —
          @@ -0,0 +1,200 @@
          +/**
          — End diff –

          Acknowledged. We do already however add examples to the Gora script which I mentioned. I have this on my path and use it to test out some jobs based on new patches coming in. It is not a big deal but just makes your work easier to use. Up to you, if you do not wish to implement then don't. I will.

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38148077 — Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java — @@ -0,0 +1,200 @@ +/** — End diff – Acknowledged. We do already however add examples to the Gora script which I mentioned. I have this on my path and use it to test out some jobs based on new patches coming in. It is not a big deal but just makes your work easier to use. Up to you, if you do not wish to implement then don't. I will.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kamaci commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38151353

          — Diff: pom.xml —
          @@ -644,6 +644,7 @@
          <commons-io.version>1.3.2</commons-io.version>
          <restlet.version>2.3.1</restlet.version>

          + <spark.version>1.3.1</spark.version>
          — End diff –

          Spark version is changed to 1.4.1 from 1.3.1. I've tested both built-in tests of Gora and some specific tests for Spark.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kamaci commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38151353 — Diff: pom.xml — @@ -644,6 +644,7 @@ <commons-io.version>1.3.2</commons-io.version> <restlet.version>2.3.1</restlet.version> + <spark.version>1.3.1</spark.version> — End diff – Spark version is changed to 1.4.1 from 1.3.1. I've tested both built-in tests of Gora and some specific tests for Spark.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kamaci commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38151501

          — Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java —
          @@ -0,0 +1,200 @@
          +/**
          — End diff –

          I can implement them with pleasure if you have some more examples and want me add GoraSparkEngine versions of them?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kamaci commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38151501 — Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java — @@ -0,0 +1,200 @@ +/** — End diff – I can implement them with pleasure if you have some more examples and want me add GoraSparkEngine versions of them?
          Hide
          lewismc Lewis John McGibbney added a comment - - edited

          OK folks. I have reviewed the PR and I am quite happy to commit this unless there are objections. Nice work Furkan KAMACI and Talat UYARER.

          Show
          lewismc Lewis John McGibbney added a comment - - edited OK folks. I have reviewed the PR and I am quite happy to commit this unless there are objections. Nice work Furkan KAMACI and Talat UYARER .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user lewismc commented on a diff in the pull request:

          https://github.com/apache/gora/pull/29#discussion_r38289205

          — Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java —
          @@ -0,0 +1,200 @@
          +/**
          — End diff –

          We will work on a more comprehensive test suite within the 0.7 development drive.

          Show
          githubbot ASF GitHub Bot added a comment - Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/29#discussion_r38289205 — Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java — @@ -0,0 +1,200 @@ +/** — End diff – We will work on a more comprehensive test suite within the 0.7 development drive.
          Hide
          lewismc Lewis John McGibbney added a comment -

          Would like to commit EoB today unless there are objections folks. Thanks.

          Show
          lewismc Lewis John McGibbney added a comment - Would like to commit EoB today unless there are objections folks. Thanks.
          Hide
          lewismc Lewis John McGibbney added a comment -

          Hi Furkan KAMACI I am getting a EOFException thrown when I test this patch locally against the existing test suite

          lmcgibbn@LMC-032857 /usr/local/gora-dev(master) $ mvn -version
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1024m; support was removed in 8.0
          Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T04:57:37-07:00)
          Maven home: /usr/local/apache-maven-3.3.3
          Java version: 1.8.0_05, vendor: Oracle Corporation
          Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre
          Default locale: en_US, platform encoding: UTF-8
          OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"

          -------------------------------------------------------------------------------
          Test set: org.apache.gora.query.impl.TestQueryBase
          -------------------------------------------------------------------------------
          Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.611 sec <<< FAILURE!
          testReadWrite(org.apache.gora.query.impl.TestQueryBase)  Time elapsed: 0.26 sec  <<< ERROR!
          java.io.EOFException
          	at org.apache.avro.util.ByteBufferInputStream.getBuffer(ByteBufferInputStream.java:86)
          	at org.apache.avro.util.ByteBufferInputStream.read(ByteBufferInputStream.java:48)
          	at java.io.DataInputStream.readFully(DataInputStream.java:195)
          	at org.apache.hadoop.io.Text.readString(Text.java:466)
          	at org.apache.hadoop.io.Text.readString(Text.java:457)
          	at org.apache.gora.mapreduce.StringSerialization$1.deserialize(StringSerialization.java:55)
          	at org.apache.gora.mapreduce.StringSerialization$1.deserialize(StringSerialization.java:40)
          	at org.apache.gora.util.IOUtils.deserialize(IOUtils.java:228)
          	at org.apache.gora.query.impl.QueryBase.readFields(QueryBase.java:227)
          	at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
          	at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
          	at org.apache.gora.util.IOUtils.deserialize(IOUtils.java:228)
          	at org.apache.gora.util.TestIOUtils.testSerializeDeserialize(TestIOUtils.java:121)
          	at org.apache.gora.query.impl.TestQueryBase.testReadWrite(TestQueryBase.java:50)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:483)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:483)
          	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
          	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
          	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
          
          Show
          lewismc Lewis John McGibbney added a comment - Hi Furkan KAMACI I am getting a EOFException thrown when I test this patch locally against the existing test suite lmcgibbn@LMC-032857 /usr/local/gora-dev(master) $ mvn -version Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1024m; support was removed in 8.0 Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T04:57:37-07:00) Maven home: /usr/local/apache-maven-3.3.3 Java version: 1.8.0_05, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac" ------------------------------------------------------------------------------- Test set: org.apache.gora.query.impl.TestQueryBase ------------------------------------------------------------------------------- Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.611 sec <<< FAILURE! testReadWrite(org.apache.gora.query.impl.TestQueryBase) Time elapsed: 0.26 sec <<< ERROR! java.io.EOFException at org.apache.avro.util.ByteBufferInputStream.getBuffer(ByteBufferInputStream.java:86) at org.apache.avro.util.ByteBufferInputStream.read(ByteBufferInputStream.java:48) at java.io.DataInputStream.readFully(DataInputStream.java:195) at org.apache.hadoop.io.Text.readString(Text.java:466) at org.apache.hadoop.io.Text.readString(Text.java:457) at org.apache.gora.mapreduce.StringSerialization$1.deserialize(StringSerialization.java:55) at org.apache.gora.mapreduce.StringSerialization$1.deserialize(StringSerialization.java:40) at org.apache.gora.util.IOUtils.deserialize(IOUtils.java:228) at org.apache.gora.query.impl.QueryBase.readFields(QueryBase.java:227) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42) at org.apache.gora.util.IOUtils.deserialize(IOUtils.java:228) at org.apache.gora.util.TestIOUtils.testSerializeDeserialize(TestIOUtils.java:121) at org.apache.gora.query.impl.TestQueryBase.testReadWrite(TestQueryBase.java:50) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
          Hide
          kamaci Furkan KAMACI added a comment -

          Lewis John McGibbney did you make any modification before you run the tests? How did you test it?

          Show
          kamaci Furkan KAMACI added a comment - Lewis John McGibbney did you make any modification before you run the tests? How did you test it?
          Hide
          lewismc Lewis John McGibbney added a comment -

          I'll attach a patch in about an hour or two

          On Wednesday, September 2, 2015, Furkan KAMACI (JIRA) <jira@apache.org>


          Lewis

          Show
          lewismc Lewis John McGibbney added a comment - I'll attach a patch in about an hour or two On Wednesday, September 2, 2015, Furkan KAMACI (JIRA) <jira@apache.org> – Lewis
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/gora/pull/29

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/gora/pull/29
          Hide
          lewismc Lewis John McGibbney added a comment -

          Excellent work Furkan KAMACI and Talat UYARER. It was a pretty quick summer and I am VERY glad that this project has come to fruition. Fantastic. This code was committed to master branch as it tested successfully locally.

          Show
          lewismc Lewis John McGibbney added a comment - Excellent work Furkan KAMACI and Talat UYARER . It was a pretty quick summer and I am VERY glad that this project has come to fruition. Fantastic. This code was committed to master branch as it tested successfully locally.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in gora-trunk #1610 (See https://builds.apache.org/job/gora-trunk/1610/)
          GORA-386 Gora Spark Backend Support addition to make forbidden api compliant (lewis.j.mcgibbney: rev 2ce2fda5b7f688f1c5fa9e10f7c8ff64ce99be12)

          • gora-core/src/examples/java/org/apache/gora/examples/spark/SparkWordCount.java
            GORA-386 Gora Spark Backend Support - Final update of LogAnalyticsSpark to comply with forbidden-api plugin (lewis.j.mcgibbney: rev ea44388f9883218f7314ed4bec51e3f2115cf33f)
          • gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java
          • CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in gora-trunk #1610 (See https://builds.apache.org/job/gora-trunk/1610/ ) GORA-386 Gora Spark Backend Support addition to make forbidden api compliant (lewis.j.mcgibbney: rev 2ce2fda5b7f688f1c5fa9e10f7c8ff64ce99be12) gora-core/src/examples/java/org/apache/gora/examples/spark/SparkWordCount.java GORA-386 Gora Spark Backend Support - Final update of LogAnalyticsSpark to comply with forbidden-api plugin (lewis.j.mcgibbney: rev ea44388f9883218f7314ed4bec51e3f2115cf33f) gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java CHANGES.txt

            People

            • Assignee:
              kamaci Furkan KAMACI
              Reporter:
              talat Talat UYARER
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development