Talat UYARER I would like to apply GSoC for this issue. I'll share my application proposal with you.

Here is my repository for this issue: https://github.com/kamaci/gora

Hi Furkan KAMACI can you please send a pull request so we can undertake further code review, test, etc?
Thank you

I've ignored word count test at here: https://github.com/kamaci/gora/blob/master/gora-hbase/src/test/java/org/apache/gora/hbase/mapreduce/TestHBaseStoreWordCount.java#L65 due to the connection error it throws. I've attached full stack trace as connection_refused.txt

GitHub user kamaci opened a pull request:

https://github.com/apache/gora/pull/29

Spark Backend Support for Gora (GORA-386)

Pull request of the work for GSoC project (Spark Backend Support for Gora) for Apache Gora.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kamaci/gora master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/gora/pull/29.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #29

commit 9c2d225d04cfa746244373fa661a1aa6f03250bb
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-06-27T22:57:51Z

• JavaPairRDD support for GoraInputFormat.

commit ef68cead273324797cf292dbe6da18ee3fd819cb
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-06-28T14:17:52Z

• GoraSpark.java initialize method renamed to initializeInput.
• Architectural change is made.

commit 8fbdef7de963defb1387f0fbbe9e56506b627ba3
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-06-28T15:49:13Z

• Serialization support via Kyro.

commit 5644a21c5c5678611c7cef4e2c922643088951b8
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-06-28T23:10:26Z

• map function is implemented as like in LogAnalytics.java

commit 445edb12c35f867e7b81919ddfb509dd8c179ba3
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-06-28T23:26:41Z

• map was not collecting the counts and it is fixed.

commit 81af4d3afcba4633d0c5d06ead9b4256ea60862f
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-06-29T15:33:38Z

• GoraSpark.java initializeInput method renamed to initialize
• reduce part is added to example.

commit 80c0c26d8fb6a9a84ea39f5aa96cc343b4546266
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-06-29T16:26:28Z

Organizing codes
Documentation

commit c111e6290fa16d2cc560eb29dbc07eb2f8b7734b
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-07-15T17:23:51Z

• Writing to Hbase via Spark is implemented.

commit cf6e76585b18ff926f8e7e4928f2d7cd28ac7c6f
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-07-16T16:38:37Z

• Added dependency to write output to Solr.

commit 219557002f0e33ef6d0e2bb49471d24fb867b0ac
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-08-17T18:21:43Z

Code is organized at LogAnalyticsSpark.java.

commit 62be0c312927e3ea4962eb966689b49dc1fcebce
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-08-17T19:25:12Z

• GoraSparkEngine.java architecture is improved.

commit 8584911176712f63e316707ace23a12060588cdc
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-08-17T20:25:15Z

GoraSparkEngine.java setOutput method renamed to generateOutputConf.
Added a new method to GoraSparkEngine.java which creates a job and returns necessary conf for Spark to use.

commit 92b71a6d2c95492621a56e48470e951e982cc34f
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-08-17T20:34:40Z

Unnecessary reuseObjects is removed.

commit 8893fd5c6c53b19925c99668d24df23c52e70b1e
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-08-21T12:58:21Z

Unnecessary imports are removed from LogAnalyticsSpark.java

commit 9cff33598231e9cf54060cecf1ccacdf62e64c5b
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-08-22T13:46:55Z

Minor improvements for LogAnalyticsSpark.java

commit e817a7f0eeb2efb98937efb92701705ec80891b9
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-08-22T14:31:57Z

Spark engine word count tests are implemented.
Due to a connection problem, spark engine word count tests are ignored.

commit 8f1acc6d4ef6c192e8fc06287558b7bc7c39b040
Author: Furkan KAMACI <furkankamaci@gmail.com>
Date: 2015-08-22T19:06:42Z

Minor change at SparkWordCount.java

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38060140

— Diff: gora-core/pom.xml —
@@ -141,6 +141,13 @@
<artifactId>guava</artifactId>
</dependency>

+ <!-- Spark dependency -->
— End diff –

Dependency definitions usually happen within parent pom.xml. Modules then inherit the dependency without versions. This also enables us best opportunity to synchronize dependencies across modules.

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38060312

— Diff: pom.xml —
@@ -893,6 +893,10 @@
<groupId>ant</groupId>
<artifactId>ant</artifactId>
</exclusion>
+ <exclusion>
— End diff –

Why is this required?

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38060355

— Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java —
@@ -0,0 +1,200 @@
+/**
— End diff –

Can you please add an entry to https://github.com/apache/gora/blob/master/bin/gora which enables us to run this very easily from the command line? It will help with testing and validation of your code.

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38060396

— Diff: gora-tutorial/pom.xml —
@@ -108,6 +108,11 @@
<artifactId>gora-cassandra</artifactId>
</dependency>

+ <dependency>
— End diff –

Thanks for adding this.

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38060417

— Diff: gora-hbase/src/test/java/org/apache/gora/hbase/mapreduce/TestHBaseStoreWordCount.java —
@@ -58,4 +59,11 @@ public void testWordCount() throws Exception

+ //todo fix config
+ @Ignore
— End diff –

I am OK for us ignoring the tests at this point in time. It is high desired though that we ensure the issue you've described at http://www.mail-archive.com/dev%40gora.apache.org/msg06020.html is addressed in a separate issue.

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38060425

— Diff: gora-mongodb/src/test/java/org/apache/gora/mongodb/mapreduce/TestMongoStoreWordCount.java —
@@ -55,4 +56,12 @@ public void testWordCount() throws Exception

+ //todo fix config
+ @Ignore
— End diff –

I am OK for us ignoring the tests at this point in time. It is high desired though that we ensure the issue you've described at http://www.mail-archive.com/dev%40gora.apache.org/msg06020.html is addressed in a separate issue.

Furkan KAMACI please address the above comments when you have the time. Thank you.

Can you also add a page to the current documentation detailing your additions e.g. GoraSparkEngine?
This would go in http://svn.apache.org/repos/asf/gora/site/trunk/content/current/

Github user kamaci commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38144036

— Diff: gora-core/pom.xml —
@@ -141,6 +141,13 @@
<artifactId>guava</artifactId>
</dependency>

+ <!-- Spark dependency -->
— End diff –

Spark version should be placed at parent pom and it's done.

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38146393

— Diff: pom.xml —
@@ -644,6 +644,7 @@
<commons-io.version>1.3.2</commons-io.version>
<restlet.version>2.3.1</restlet.version>

+ <spark.version>1.3.1</spark.version>
— End diff –

Any reason to not use Spark version 1.4.1?

Github user kamaci commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38147131

— Diff: pom.xml —
@@ -893,6 +893,10 @@
<groupId>ant</groupId>
<artifactId>ant</artifactId>
</exclusion>
+ <exclusion>
— End diff –

servlet dependency is excluded from spark, not ant. Without excluding it throws a java security exception. Motivation for it: https://issues.apache.org/jira/browse/SPARK-1693

Github user kamaci commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38147685

— Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java —
@@ -0,0 +1,200 @@
+/**
— End diff –

LogAnalyticsSpark is let to run from command line by passing "loganalyticsspark" argument.

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38148077

— Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java —
@@ -0,0 +1,200 @@
+/**
— End diff –

Acknowledged. We do already however add examples to the Gora script which I mentioned. I have this on my path and use it to test out some jobs based on new patches coming in. It is not a big deal but just makes your work easier to use. Up to you, if you do not wish to implement then don't. I will.

Github user kamaci commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38151353

— Diff: pom.xml —
@@ -644,6 +644,7 @@
<commons-io.version>1.3.2</commons-io.version>
<restlet.version>2.3.1</restlet.version>

+ <spark.version>1.3.1</spark.version>
— End diff –

Spark version is changed to 1.4.1 from 1.3.1. I've tested both built-in tests of Gora and some specific tests for Spark.

Github user kamaci commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38151501

— Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java —
@@ -0,0 +1,200 @@
+/**
— End diff –

I can implement them with pleasure if you have some more examples and want me add GoraSparkEngine versions of them?

OK folks. I have reviewed the PR and I am quite happy to commit this unless there are objections. Nice work Furkan KAMACI and Talat UYARER.

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/29#discussion_r38289205

— Diff: gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java —
@@ -0,0 +1,200 @@
+/**
— End diff –

We will work on a more comprehensive test suite within the 0.7 development drive.

Would like to commit EoB today unless there are objections folks. Thanks.

Hi Furkan KAMACI I am getting a EOFException thrown when I test this patch locally against the existing test suite

lmcgibbn@LMC-032857 /usr/local/gora-dev(master) $ mvn -version
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1024m; support was removed in 8.0
Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T04:57:37-07:00)
Maven home: /usr/local/apache-maven-3.3.3
Java version: 1.8.0_05, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"

-------------------------------------------------------------------------------
Test set: org.apache.gora.query.impl.TestQueryBase
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.611 sec <<< FAILURE!
testReadWrite(org.apache.gora.query.impl.TestQueryBase)  Time elapsed: 0.26 sec  <<< ERROR!
java.io.EOFException
	at org.apache.avro.util.ByteBufferInputStream.getBuffer(ByteBufferInputStream.java:86)
	at org.apache.avro.util.ByteBufferInputStream.read(ByteBufferInputStream.java:48)
	at java.io.DataInputStream.readFully(DataInputStream.java:195)
	at org.apache.hadoop.io.Text.readString(Text.java:466)
	at org.apache.hadoop.io.Text.readString(Text.java:457)
	at org.apache.gora.mapreduce.StringSerialization$1.deserialize(StringSerialization.java:55)
	at org.apache.gora.mapreduce.StringSerialization$1.deserialize(StringSerialization.java:40)
	at org.apache.gora.util.IOUtils.deserialize(IOUtils.java:228)
	at org.apache.gora.query.impl.QueryBase.readFields(QueryBase.java:227)
	at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
	at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
	at org.apache.gora.util.IOUtils.deserialize(IOUtils.java:228)
	at org.apache.gora.util.TestIOUtils.testSerializeDeserialize(TestIOUtils.java:121)
	at org.apache.gora.query.impl.TestQueryBase.testReadWrite(TestQueryBase.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

Lewis John McGibbney did you make any modification before you run the tests? How did you test it?

I'll attach a patch in about an hour or two

On Wednesday, September 2, 2015, Furkan KAMACI (JIRA) <jira@apache.org>

–
Lewis

Github user asfgit closed the pull request at:

https://github.com/apache/gora/pull/29

Excellent work Furkan KAMACI and Talat UYARER. It was a pretty quick summer and I am VERY glad that this project has come to fruition. Fantastic. This code was committed to master branch as it tested successfully locally.

SUCCESS: Integrated in gora-trunk #1610 (See https://builds.apache.org/job/gora-trunk/1610/)
GORA-386 Gora Spark Backend Support addition to make forbidden api compliant (lewis.j.mcgibbney: rev 2ce2fda5b7f688f1c5fa9e10f7c8ff64ce99be12)

• gora-core/src/examples/java/org/apache/gora/examples/spark/SparkWordCount.java
  GORA-386 Gora Spark Backend Support - Final update of LogAnalyticsSpark to comply with forbidden-api plugin (lewis.j.mcgibbney: rev ea44388f9883218f7314ed4bec51e3f2115cf33f)
• gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalyticsSpark.java
• CHANGES.txt