Talat UYARER added a comment - 25/Sep/14 21:11 Does anybody work in this issue ? If I anyone does not work on this, Lewis John McGibbney can you assign me ?

Henry Saputra added a comment - 25/Sep/14 21:21 Talat UYARER , I just assigned it to you. Have fun

Talat UYARER added a comment - 26/Sep/14 14:06 Hi Henry Saputra and Lewis John McGibbney , Thanks for assignment. I create a patch. I use somethings from Mikhail Bernadsky 's Gora-345 mungle patch. I hope I could do it . Can you review it ?

Lewis John McGibbney added a comment - 28/Sep/14 03:49 Hi Talat UYARER , we need to upgrade MapReduce tests (and possibly the default version of Hadoop to use to build Gora aginst) to Hadoop 2.X before we upgrade HBase dependency. The same is happening here a before we released Gora 0.5 e.g. Results : Tests in error: initializationError(org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce): org/apache/hadoop/hdfs/DistributedFileSystem Tests run: 53, Failures: 0, Errors: 1, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Gora ....................................... SUCCESS [ 4.830 s] [INFO] Apache Gora :: Compiler ........................... SUCCESS [ 8.089 s] [INFO] Apache Gora :: Compiler-CLI ....................... SUCCESS [ 1.915 s] [INFO] Apache Gora :: Shims Hadoop ....................... SUCCESS [ 3.061 s] [INFO] Apache Gora :: Shims Hadoop 1.x ................... SUCCESS [ 2.539 s] [INFO] Apache Gora :: Shims Hadoop 2.x ................... SUCCESS [ 3.106 s] [INFO] Apache Gora :: Shims Distribution ................. SUCCESS [ 0.803 s] [INFO] Apache Gora :: Core ............................... FAILURE [ 26.196 s] [INFO] Apache Gora :: Accumulo ........................... SKIPPED

Lewis John McGibbney added a comment - 28/Sep/14 03:51 Can you also ensure that the module pom.xml files are not reformatted within your patch? We recently addressed this in GORA-369 and it would be really nice if we could keep the formatting consistent. Thanks so much Talat UYARER great effort.

Lewis John McGibbney added a comment - 28/Sep/14 03:52 Failing test setup due to Hadoop version 1.X and 2.X mismatch

Talat UYARER added a comment - 30/Sep/14 14:42 Hi Lewis John McGibbney , I fixed test problem and formatting problem. Now all test passed.

Renato Javier Marroquín Mogrovejo added a comment - 30/Sep/14 19:48 Hi Talat UYARER , I am finally finding the time to get to you my friend One question about your patch, shouldn't the hadoop test dependency use the <hadoop.test.version> property defined in the pom.xml? So we can keep on having the same version for testing in all the places?

Talat UYARER added a comment - 01/Oct/14 07:53 Hi Renato Javier Marroquín Mogrovejo , You are right. If we use Hadoop 2 testing We should change testing dependecy. Because of that I do not use hadoop.test.version. Should I use despite it ?

Lewis John McGibbney added a comment - 19/Oct/14 23:43 Hi Talat UYARER , I fear that we cannot use the gora-hbase module with Hadoop v1.X if we make this upgrade. The HBase version you have chosen is 0.98.6.1-hadoop2 which is great, however I think it is important that we maintain backwards compatibility if at all possible. Good work Talat.

Talat UYARER added a comment - 21/Oct/14 18:05 Hi Lewis John McGibbney , HBase 0.98 deprecates use of Hadoop 1.x, and HBase 1.0 will not support Hadoop 1.x. [1] IMHO We should upgrade Hadoop 2.x in our next version. Wdyt ? [1] http://hbase.apache.org/book.html#d0e1440

Henry Saputra added a comment - 21/Oct/14 18:23 +1 Lets tagged deprecated for Hadoop 1.x for one or two more releases. Then we could go only Hadoop 2.x.

Alfonso Nishikawa added a comment - 21/Oct/14 19:38 Hi! 0.98 is deprecated or not supported? I see contradictory information In my oppinion, if 0.98 is not supported in Hadoop 1, then HBase 0.96 is the most compatible version and to upgrade to 0.98 we should make some type of fork... as everything is getting forked because of Hadoop. If 0.98 is "only" deprecated and works with Hadoop 1, another interesting questions to answer first are: Can a 0.96 client connect to a 0.98 server? Can a 0.98 client connect to a 0.96 server using only "0.96" methods? Anyone knows?

Talat UYARER added a comment - 22/Oct/14 05:56 Hi folks, 0.98 is deprecated or not supported? I see contradictory information Alfonso Nishikawa is right. There is inconsistent situation. I wondered it and I researched it. They said : HBase 0.96.x will run on Hadoop 1.x or Hadoop 2.x. HBase 0.98 still runs on both, but HBase 0.98 deprecates use of Hadoop 1. HBase 1.x will not run on Hadoop 1. As I can understand, They support Hadoop 1.x in Hbase 0.96, in Hbase 0.98 can run on Hadoop 1.x but support for Hadoop 1.x is deprecated. If they do not support their product(0.98) in Hadoop 1.x, Why we support for backwards compatibility ? IMHO I do not support it. We can document how can change dependecies of gora for hadoop 1.x work.

Alfonso Nishikawa added a comment - 22/Oct/14 21:12 Talat UYARER : Thanks for the information! Then, it seems there is no problem to upgrade to 0.98 (at least for me, "deprecated" is fine). About developing only for Hadoop 2.x I am not much convinced. In my opinion, that would deserve a mayor version. Would be better to look for other solutions to mantain backwards compatibility. Anyway, we are still a step back until HBase 1.x is the stable release, isn't it? Personally I leave the decission for later

Renato Javier Marroquín Mogrovejo added a comment - 25/Oct/14 20:37 +1 to keep them as deprecated for couple more of releases and then going only with Hadoop 2

Lewis John McGibbney added a comment - 25/Oct/14 21:07 +1 On Saturday, October 25, 2014, Renato Javier Marroquín Mogrovejo (JIRA) < – Lewis

Talat UYARER added a comment - 26/Oct/14 08:16 +1

Talat UYARER added a comment - 13/Dec/14 14:05 I updated my last patch. Lewis John McGibbney can you review it ? Previous patch I updated testing dependencies to hadoop 2. In this patch I revert it. The patch use hbase hadoop 1 client. I upgrade hadoop 1.0.1 to hadoop 1.2.1.

Alparslan Avcı added a comment - 24/Dec/14 08:21 Talat UYARER , I've tested your patch but when running hadoop jobs, I could not see the job progress of map jobs. You can see the log below: Normal run: 14/12/23 17:04:49 INFO mapred.JobClient: map 0% reduce 0% 14/12/23 17:04:53 INFO mapred.MapTask: Starting flush of map output 14/12/23 17:04:53 INFO mapred.MapTask: Finished spill 0 14/12/23 17:04:53 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting 14/12/23 17:04:54 INFO mapred.LocalJobRunner: 14/12/23 17:04:54 INFO mapred.LocalJobRunner: 14/12/23 17:04:54 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done. 14/12/23 17:04:54 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@30fd4204 14/12/23 17:04:54 INFO mapred.LocalJobRunner: 14/12/23 17:04:54 INFO mapred.Merger: Merging 1 sorted segments 14/12/23 17:04:54 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 532529 bytes 14/12/23 17:04:54 INFO mapred.LocalJobRunner: 14/12/23 17:04:54 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as the one in the databean. 14/12/23 17:04:54 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@436daf20 14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 14/12/23 17:04:54 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 15793@alparslan 14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77a9d8b90017, negotiated timeout = 40000 14/12/23 17:04:54 INFO zookeeper.ZooKeeper: Session: 0x14a77a9d8b90017 closed 14/12/23 17:04:54 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/23 17:04:54 INFO mapreduce.GoraRecordWriter: gora.buffer.write.limit = 10000 14/12/23 17:04:54 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting 14/12/23 17:04:55 INFO mapred.JobClient: map 100% reduce 0% 14/12/23 17:04:57 INFO mapred.LocalJobRunner: reduce > reduce 14/12/23 17:04:57 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done. 14/12/23 17:04:57 WARN mapred.FileOutputCommitter: Output path is null in cleanup 14/12/23 17:04:58 INFO mapred.JobClient: map 100% reduce 100% 14/12/23 17:04:58 INFO mapred.JobClient: Job complete: job_local_0001 When your patch applied: 14/12/23 17:11:23 INFO mapred.JobClient: map 0% reduce 0% 14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b10921001e closed 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/23 17:11:24 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper ensemble=localhost:2181 14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181, baseZNode=/hbase 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77b10921001f, negotiated timeout = 40000 14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b10921001f closed 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/23 17:11:24 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper ensemble=localhost:2181 14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181, baseZNode=/hbase 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77b109210020, negotiated timeout = 40000 14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b109210020 closed 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/23 17:11:24 INFO mapred.MapTask: Processing split: org.apache.gora.mapreduce.GoraInputSplit@40b86a70 14/12/23 17:11:24 INFO mapreduce.GoraRecordReader: gora.buffer.read.limit = 10000 14/12/23 17:11:24 INFO mapred.MapTask: io.sort.mb = 100 14/12/23 17:11:24 INFO mapred.MapTask: data buffer = 79691776/99614720 14/12/23 17:11:24 INFO mapred.MapTask: record buffer = 262144/327680 14/12/23 17:11:25 INFO mapred.MapTask: Starting flush of map output 14/12/23 17:11:25 INFO mapred.MapTask: Finished spill 0 14/12/23 17:11:25 INFO mapred.Task: Task:attempt_local139853648_0001_m_000000_0 is done. And is in the process of commiting 14/12/23 17:11:25 INFO mapred.LocalJobRunner: 14/12/23 17:11:25 INFO mapred.Task: Task 'attempt_local139853648_0001_m_000000_0' done. 14/12/23 17:11:25 INFO mapred.LocalJobRunner: Finishing task: attempt_local139853648_0001_m_000000_0 14/12/23 17:11:25 INFO mapred.LocalJobRunner: Map task executor complete. 14/12/23 17:11:25 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3589085 14/12/23 17:11:25 INFO mapred.LocalJobRunner: 14/12/23 17:11:25 INFO mapred.Merger: Merging 1 sorted segments 14/12/23 17:11:25 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 532529 bytes 14/12/23 17:11:25 INFO mapred.LocalJobRunner: 14/12/23 17:11:25 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as the one in the databean. 14/12/23 17:11:25 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper ensemble=localhost:2181 14/12/23 17:11:25 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181, baseZNode=/hbase 14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77b109210021, negotiated timeout = 40000 14/12/23 17:11:25 INFO zookeeper.ZooKeeper: Session: 0x14a77b109210021 closed 14/12/23 17:11:25 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/23 17:11:25 INFO mapreduce.GoraRecordWriter: gora.buffer.write.limit = 10000 14/12/23 17:11:26 INFO mapred.Task: Task:attempt_local139853648_0001_r_000000_0 is done. And is in the process of commiting 14/12/23 17:11:26 INFO mapred.LocalJobRunner: reduce > reduce 14/12/23 17:11:26 INFO mapred.Task: Task 'attempt_local139853648_0001_r_000000_0' done. 14/12/23 17:11:26 WARN mapred.FileOutputCommitter: Output path is null in cleanup 14/12/23 17:11:26 INFO mapred.JobClient: map 0% reduce 100% 14/12/23 17:11:26 INFO mapred.JobClient: Job complete: job_local139853648_0001 Is this normal on hadoop-1.2.1? Or is there a problem caused by us?

Alparslan Avcı added a comment - 24/Dec/14 08:22 BTW, the mapreduce job results are ok, no problem occurs.

Talat UYARER added a comment - 05/Jan/15 10:17 I create a hadoop 2 version. I upgrade all dependecies to Hadoop 2. Actually I do not change a lot of things. If you review my patch I will be glad. BTW We are little bit slow for development. IMHO hadoop 1.x is deprecated and we continue with hadoop 2.x support. I can write a how to Gora work on hadoop 1 documentation.

Lewis John McGibbney added a comment - 07/Jan/15 19:11 Alparslan Avcı 14/12/23 17:11:25 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as the one in the databean. You might want to get this sorted out

Lewis John McGibbney added a comment - 08/Jan/15 06:24 - edited Talat UYARER I've checked out this patch and I think it looks really good. I've tested it on my local version of trunk and it is passing all existing tests. I also agree with your comment that the shift to having Hadoop 2.X as standard with option to switch back to 1.X is best way to go. It would be great if we could further document the Shims in the placeholder I opened up however the documentation can follow shortly after I am sure. Does anyone else have cycles to try this patch out locally and comment? It would be great to get this patch in and close it off if we can get some level of consensus. Great work Talat UYARER

Lewis John McGibbney added a comment - 08/Jan/15 19:01 Anyone else for the most recent patch?

Henry Saputra added a comment - 08/Jan/15 23:20 Works in my local env. Just small nit, we could just remove the commented maven pom.xml: <!-- <hbase.version>0.94.14</hbase.version> --> with versioning we can always revert back.

Lewis John McGibbney added a comment - 08/Jan/15 23:52 Nice catch Henry Saputra I would like to commit E.O.D if it is OK. I will be using this module this coming week and would like to get as much testing done as I can over the weekend. The HBase cluster I'll be writing in to as explained on the list is CDH, so we will see how it goes.

Lewis John McGibbney added a comment - 09/Jan/15 02:56 commit e1eb90a369851805ef8e1bd1a89f71cd1644f73b Author: Lewis John McGibbney <lewis.j....nasa.gov> Date: Thu Jan 8 18:54:42 2015 -0800 GORA-375 Upgrade to HBase 0.98 commit 30bb3af96d5c70ba2e70dca183c62fb6a49c343c Author: Lewis John McGibbney <lewis.j....nasa.gov> Date: Thu Jan 8 17:55:18 2015 -0800 GORA-375 Upgrade HBase to 0.98

Lewis John McGibbney added a comment - 09/Jan/15 02:57 great work on this one Talat UYARER Henry Saputra I made sure to remove that line for you in pom.xml Once I get a stable build on Jenkins I am going to make an announcement on the website that we've skipped to Hadoop 2.X as default implementation now with backwards support for Hadoop 1.X

Hudson added a comment - 09/Jan/15 03:09 SUCCESS: Integrated in gora-trunk #1359 (See https://builds.apache.org/job/gora-trunk/1359/ ) GORA-375 Upgrade HBase to 0.98 (lewis.j.mcgibbney: rev 30bb3af96d5c70ba2e70dca183c62fb6a49c343c) pom.xml gora-hbase/src/main/java/org/apache/gora/hbase/util/HBaseByteInterface.java gora-cassandra/pom.xml gora-core/pom.xml gora-shims-hadoop2/pom.xml gora-mongodb/pom.xml gora-shims-hadoop/pom.xml gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseMapping.java gora-shims-hadoop1/pom.xml gora-goraci/pom.xml gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseTableConnection.java gora-accumulo/pom.xml gora-core/src/test/java/org/apache/gora/mapreduce/TestGoraInputFormat.java gora-hbase/pom.xml gora-tutorial/pom.xml GORA-375 Upgrade to HBase 0.98 (lewis.j.mcgibbney: rev e1eb90a369851805ef8e1bd1a89f71cd1644f73b) CHANGES.txt