Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.5
    • Fix Version/s: 0.6
    • Component/s: gora-hbase
    • Labels:

      Description

      HBase 0.98 release is the current stable release.

      Gora should be built based on HBase 0.98.

      1. GORA-375.patch
        14 kB
        Talat UYARER
      2. GORA-375-hadoop2.patch
        27 kB
        Talat UYARER
      3. org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce.txt
        3 kB
        Lewis John McGibbney

        Activity

        Hide
        talat Talat UYARER added a comment -

        Does anybody work in this issue ? If I anyone does not work on this, Lewis John McGibbney can you assign me ?

        Show
        talat Talat UYARER added a comment - Does anybody work in this issue ? If I anyone does not work on this, Lewis John McGibbney can you assign me ?
        Hide
        hsaputra Henry Saputra added a comment -

        Talat UYARER, I just assigned it to you. Have fun

        Show
        hsaputra Henry Saputra added a comment - Talat UYARER , I just assigned it to you. Have fun
        Hide
        talat Talat UYARER added a comment -

        Hi Henry Saputra and Lewis John McGibbney,

        Thanks for assignment. I create a patch. I use somethings from Mikhail Bernadsky 's Gora-345 mungle patch. I hope I could do it . Can you review it ?

        Show
        talat Talat UYARER added a comment - Hi Henry Saputra and Lewis John McGibbney , Thanks for assignment. I create a patch. I use somethings from Mikhail Bernadsky 's Gora-345 mungle patch. I hope I could do it . Can you review it ?
        Hide
        lewismc Lewis John McGibbney added a comment -

        Hi Talat UYARER, we need to upgrade MapReduce tests (and possibly the default version of Hadoop to use to build Gora aginst) to Hadoop 2.X before we upgrade HBase dependency.
        The same is happening here a before we released Gora 0.5 e.g.

        Results :

        Tests in error:
        initializationError(org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce): org/apache/hadoop/hdfs/DistributedFileSystem

        Tests run: 53, Failures: 0, Errors: 1, Skipped: 0

        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO]
        [INFO] Apache Gora ....................................... SUCCESS [ 4.830 s]
        [INFO] Apache Gora :: Compiler ........................... SUCCESS [ 8.089 s]
        [INFO] Apache Gora :: Compiler-CLI ....................... SUCCESS [ 1.915 s]
        [INFO] Apache Gora :: Shims Hadoop ....................... SUCCESS [ 3.061 s]
        [INFO] Apache Gora :: Shims Hadoop 1.x ................... SUCCESS [ 2.539 s]
        [INFO] Apache Gora :: Shims Hadoop 2.x ................... SUCCESS [ 3.106 s]
        [INFO] Apache Gora :: Shims Distribution ................. SUCCESS [ 0.803 s]
        [INFO] Apache Gora :: Core ............................... FAILURE [ 26.196 s]
        [INFO] Apache Gora :: Accumulo ........................... SKIPPED

        Show
        lewismc Lewis John McGibbney added a comment - Hi Talat UYARER , we need to upgrade MapReduce tests (and possibly the default version of Hadoop to use to build Gora aginst) to Hadoop 2.X before we upgrade HBase dependency. The same is happening here a before we released Gora 0.5 e.g. Results : Tests in error: initializationError(org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce): org/apache/hadoop/hdfs/DistributedFileSystem Tests run: 53, Failures: 0, Errors: 1, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Gora ....................................... SUCCESS [ 4.830 s] [INFO] Apache Gora :: Compiler ........................... SUCCESS [ 8.089 s] [INFO] Apache Gora :: Compiler-CLI ....................... SUCCESS [ 1.915 s] [INFO] Apache Gora :: Shims Hadoop ....................... SUCCESS [ 3.061 s] [INFO] Apache Gora :: Shims Hadoop 1.x ................... SUCCESS [ 2.539 s] [INFO] Apache Gora :: Shims Hadoop 2.x ................... SUCCESS [ 3.106 s] [INFO] Apache Gora :: Shims Distribution ................. SUCCESS [ 0.803 s] [INFO] Apache Gora :: Core ............................... FAILURE [ 26.196 s] [INFO] Apache Gora :: Accumulo ........................... SKIPPED
        Hide
        lewismc Lewis John McGibbney added a comment -

        Can you also ensure that the module pom.xml files are not reformatted within your patch?
        We recently addressed this in GORA-369 and it would be really nice if we could keep the formatting consistent.
        Thanks so much Talat UYARER great effort.

        Show
        lewismc Lewis John McGibbney added a comment - Can you also ensure that the module pom.xml files are not reformatted within your patch? We recently addressed this in GORA-369 and it would be really nice if we could keep the formatting consistent. Thanks so much Talat UYARER great effort.
        Hide
        lewismc Lewis John McGibbney added a comment -

        Failing test setup due to Hadoop version 1.X and 2.X mismatch

        Show
        lewismc Lewis John McGibbney added a comment - Failing test setup due to Hadoop version 1.X and 2.X mismatch
        Hide
        talat Talat UYARER added a comment -

        Hi Lewis John McGibbney,

        I fixed test problem and formatting problem. Now all test passed.

        Show
        talat Talat UYARER added a comment - Hi Lewis John McGibbney , I fixed test problem and formatting problem. Now all test passed.
        Hide
        renato2099 Renato Javier Marroquín Mogrovejo added a comment -

        Hi Talat UYARER,

        I am finally finding the time to get to you my friend
        One question about your patch, shouldn't the hadoop test dependency use the <hadoop.test.version> property defined in the pom.xml? So we can keep on having the same version for testing in all the places?

        Show
        renato2099 Renato Javier Marroquín Mogrovejo added a comment - Hi Talat UYARER , I am finally finding the time to get to you my friend One question about your patch, shouldn't the hadoop test dependency use the <hadoop.test.version> property defined in the pom.xml? So we can keep on having the same version for testing in all the places?
        Hide
        talat Talat UYARER added a comment -

        Hi Renato Javier Marroquín Mogrovejo,

        You are right. If we use Hadoop 2 testing We should change testing dependecy. Because of that I do not use hadoop.test.version. Should I use despite it ?

        Show
        talat Talat UYARER added a comment - Hi Renato Javier Marroquín Mogrovejo , You are right. If we use Hadoop 2 testing We should change testing dependecy. Because of that I do not use hadoop.test.version. Should I use despite it ?
        Hide
        lewismc Lewis John McGibbney added a comment -

        Hi Talat UYARER, I fear that we cannot use the gora-hbase module with Hadoop v1.X if we make this upgrade. The HBase version you have chosen is 0.98.6.1-hadoop2 which is great, however I think it is important that we maintain backwards compatibility if at all possible.
        Good work Talat.

        Show
        lewismc Lewis John McGibbney added a comment - Hi Talat UYARER , I fear that we cannot use the gora-hbase module with Hadoop v1.X if we make this upgrade. The HBase version you have chosen is 0.98.6.1-hadoop2 which is great, however I think it is important that we maintain backwards compatibility if at all possible. Good work Talat.
        Hide
        talat Talat UYARER added a comment -

        Hi Lewis John McGibbney,

        HBase 0.98 deprecates use of Hadoop 1.x, and HBase 1.0 will not support Hadoop 1.x. [1] IMHO We should upgrade Hadoop 2.x in our next version. Wdyt ?

        [1] http://hbase.apache.org/book.html#d0e1440

        Show
        talat Talat UYARER added a comment - Hi Lewis John McGibbney , HBase 0.98 deprecates use of Hadoop 1.x, and HBase 1.0 will not support Hadoop 1.x. [1] IMHO We should upgrade Hadoop 2.x in our next version. Wdyt ? [1] http://hbase.apache.org/book.html#d0e1440
        Hide
        hsaputra Henry Saputra added a comment -

        +1

        Lets tagged deprecated for Hadoop 1.x for one or two more releases. Then we could go only Hadoop 2.x.

        Show
        hsaputra Henry Saputra added a comment - +1 Lets tagged deprecated for Hadoop 1.x for one or two more releases. Then we could go only Hadoop 2.x.
        Hide
        alfonso.nishikawa Alfonso Nishikawa added a comment -

        Hi!
        0.98 is deprecated or not supported? I see contradictory information

        In my oppinion, if 0.98 is not supported in Hadoop 1, then HBase 0.96 is the most compatible version and to upgrade to 0.98 we should make some type of fork... as everything is getting forked because of Hadoop.
        If 0.98 is "only" deprecated and works with Hadoop 1, another interesting questions to answer first are:
        Can a 0.96 client connect to a 0.98 server?
        Can a 0.98 client connect to a 0.96 server using only "0.96" methods?
        Anyone knows?

        Show
        alfonso.nishikawa Alfonso Nishikawa added a comment - Hi! 0.98 is deprecated or not supported? I see contradictory information In my oppinion, if 0.98 is not supported in Hadoop 1, then HBase 0.96 is the most compatible version and to upgrade to 0.98 we should make some type of fork... as everything is getting forked because of Hadoop. If 0.98 is "only" deprecated and works with Hadoop 1, another interesting questions to answer first are: Can a 0.96 client connect to a 0.98 server? Can a 0.98 client connect to a 0.96 server using only "0.96" methods? Anyone knows?
        Hide
        talat Talat UYARER added a comment -

        Hi folks,

        0.98 is deprecated or not supported? I see contradictory information

        Alfonso Nishikawa is right. There is inconsistent situation. I wondered it and I researched it.

        They said:

        HBase 0.96.x will run on Hadoop 1.x or Hadoop 2.x. HBase 0.98 still runs on both, but HBase 0.98 deprecates use of Hadoop 1. HBase 1.x will not run on Hadoop 1.

        As I can understand, They support Hadoop 1.x in Hbase 0.96, in Hbase 0.98 can run on Hadoop 1.x but support for Hadoop 1.x is deprecated. If they do not support their product(0.98) in Hadoop 1.x, Why we support for backwards compatibility ?

        IMHO I do not support it. We can document how can change dependecies of gora for hadoop 1.x work.

        Show
        talat Talat UYARER added a comment - Hi folks, 0.98 is deprecated or not supported? I see contradictory information Alfonso Nishikawa is right. There is inconsistent situation. I wondered it and I researched it. They said : HBase 0.96.x will run on Hadoop 1.x or Hadoop 2.x. HBase 0.98 still runs on both, but HBase 0.98 deprecates use of Hadoop 1. HBase 1.x will not run on Hadoop 1. As I can understand, They support Hadoop 1.x in Hbase 0.96, in Hbase 0.98 can run on Hadoop 1.x but support for Hadoop 1.x is deprecated. If they do not support their product(0.98) in Hadoop 1.x, Why we support for backwards compatibility ? IMHO I do not support it. We can document how can change dependecies of gora for hadoop 1.x work.
        Hide
        alfonso.nishikawa Alfonso Nishikawa added a comment -

        Talat UYARER: Thanks for the information! Then, it seems there is no problem to upgrade to 0.98 (at least for me, "deprecated" is fine).

        About developing only for Hadoop 2.x I am not much convinced. In my opinion, that would deserve a mayor version. Would be better to look for other solutions to mantain backwards compatibility. Anyway, we are still a step back until HBase 1.x is the stable release, isn't it? Personally I leave the decission for later

        Show
        alfonso.nishikawa Alfonso Nishikawa added a comment - Talat UYARER : Thanks for the information! Then, it seems there is no problem to upgrade to 0.98 (at least for me, "deprecated" is fine). About developing only for Hadoop 2.x I am not much convinced. In my opinion, that would deserve a mayor version. Would be better to look for other solutions to mantain backwards compatibility. Anyway, we are still a step back until HBase 1.x is the stable release, isn't it? Personally I leave the decission for later
        Hide
        renato2099 Renato Javier Marroquín Mogrovejo added a comment -

        +1 to keep them as deprecated for couple more of releases and then going only with Hadoop 2

        Show
        renato2099 Renato Javier Marroquín Mogrovejo added a comment - +1 to keep them as deprecated for couple more of releases and then going only with Hadoop 2
        Hide
        lewismc Lewis John McGibbney added a comment -

        +1

        On Saturday, October 25, 2014, Renato Javier Marroquín Mogrovejo (JIRA) <


        Lewis

        Show
        lewismc Lewis John McGibbney added a comment - +1 On Saturday, October 25, 2014, Renato Javier Marroquín Mogrovejo (JIRA) < – Lewis
        Hide
        talat Talat UYARER added a comment -

        +1

        Show
        talat Talat UYARER added a comment - +1
        Hide
        talat Talat UYARER added a comment -

        I updated my last patch. Lewis John McGibbney can you review it ? Previous patch I updated testing dependencies to hadoop 2. In this patch I revert it. The patch use hbase hadoop 1 client. I upgrade hadoop 1.0.1 to hadoop 1.2.1.

        Show
        talat Talat UYARER added a comment - I updated my last patch. Lewis John McGibbney can you review it ? Previous patch I updated testing dependencies to hadoop 2. In this patch I revert it. The patch use hbase hadoop 1 client. I upgrade hadoop 1.0.1 to hadoop 1.2.1.
        Hide
        alparslan.avci Alparslan Avcı added a comment -

        Talat UYARER, I've tested your patch but when running hadoop jobs, I could not see the job progress of map jobs. You can see the log below:

        Normal run:

        14/12/23 17:04:49 INFO mapred.JobClient:  map 0% reduce 0%
        14/12/23 17:04:53 INFO mapred.MapTask: Starting flush of map output
        14/12/23 17:04:53 INFO mapred.MapTask: Finished spill 0
        14/12/23 17:04:53 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
        14/12/23 17:04:54 INFO mapred.LocalJobRunner: 
        14/12/23 17:04:54 INFO mapred.LocalJobRunner: 
        14/12/23 17:04:54 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
        14/12/23 17:04:54 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@30fd4204
        14/12/23 17:04:54 INFO mapred.LocalJobRunner: 
        14/12/23 17:04:54 INFO mapred.Merger: Merging 1 sorted segments
        14/12/23 17:04:54 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 532529 bytes
        14/12/23 17:04:54 INFO mapred.LocalJobRunner: 
        14/12/23 17:04:54 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as the one in the databean.
        14/12/23 17:04:54 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@436daf20
        14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
        14/12/23 17:04:54 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 15793@alparslan
        14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
        14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77a9d8b90017, negotiated timeout = 40000
        14/12/23 17:04:54 INFO zookeeper.ZooKeeper: Session: 0x14a77a9d8b90017 closed
        14/12/23 17:04:54 INFO zookeeper.ClientCnxn: EventThread shut down
        14/12/23 17:04:54 INFO mapreduce.GoraRecordWriter: gora.buffer.write.limit = 10000
        14/12/23 17:04:54 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
        14/12/23 17:04:55 INFO mapred.JobClient:  map 100% reduce 0%
        14/12/23 17:04:57 INFO mapred.LocalJobRunner: reduce > reduce
        14/12/23 17:04:57 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
        14/12/23 17:04:57 WARN mapred.FileOutputCommitter: Output path is null in cleanup
        14/12/23 17:04:58 INFO mapred.JobClient:  map 100% reduce 100%
        14/12/23 17:04:58 INFO mapred.JobClient: Job complete: job_local_0001
        

        When your patch applied:

        14/12/23 17:11:23 INFO mapred.JobClient:  map 0% reduce 0%
        14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b10921001e closed
        14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down
        14/12/23 17:11:24 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper ensemble=localhost:2181
        14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181, baseZNode=/hbase
        14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
        14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
        14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77b10921001f, negotiated timeout = 40000
        14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b10921001f closed
        14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down
        14/12/23 17:11:24 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper ensemble=localhost:2181
        14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181, baseZNode=/hbase
        14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
        14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
        14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77b109210020, negotiated timeout = 40000
        14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b109210020 closed
        14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down
        14/12/23 17:11:24 INFO mapred.MapTask: Processing split: org.apache.gora.mapreduce.GoraInputSplit@40b86a70
        14/12/23 17:11:24 INFO mapreduce.GoraRecordReader: gora.buffer.read.limit = 10000
        14/12/23 17:11:24 INFO mapred.MapTask: io.sort.mb = 100
        14/12/23 17:11:24 INFO mapred.MapTask: data buffer = 79691776/99614720
        14/12/23 17:11:24 INFO mapred.MapTask: record buffer = 262144/327680
        14/12/23 17:11:25 INFO mapred.MapTask: Starting flush of map output
        14/12/23 17:11:25 INFO mapred.MapTask: Finished spill 0
        14/12/23 17:11:25 INFO mapred.Task: Task:attempt_local139853648_0001_m_000000_0 is done. And is in the process of commiting
        14/12/23 17:11:25 INFO mapred.LocalJobRunner: 
        14/12/23 17:11:25 INFO mapred.Task: Task 'attempt_local139853648_0001_m_000000_0' done.
        14/12/23 17:11:25 INFO mapred.LocalJobRunner: Finishing task: attempt_local139853648_0001_m_000000_0
        14/12/23 17:11:25 INFO mapred.LocalJobRunner: Map task executor complete.
        14/12/23 17:11:25 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3589085
        14/12/23 17:11:25 INFO mapred.LocalJobRunner: 
        14/12/23 17:11:25 INFO mapred.Merger: Merging 1 sorted segments
        14/12/23 17:11:25 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 532529 bytes
        14/12/23 17:11:25 INFO mapred.LocalJobRunner: 
        14/12/23 17:11:25 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as the one in the databean.
        14/12/23 17:11:25 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper ensemble=localhost:2181
        14/12/23 17:11:25 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181, baseZNode=/hbase
        14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
        14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
        14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77b109210021, negotiated timeout = 40000
        14/12/23 17:11:25 INFO zookeeper.ZooKeeper: Session: 0x14a77b109210021 closed
        14/12/23 17:11:25 INFO zookeeper.ClientCnxn: EventThread shut down
        14/12/23 17:11:25 INFO mapreduce.GoraRecordWriter: gora.buffer.write.limit = 10000
        14/12/23 17:11:26 INFO mapred.Task: Task:attempt_local139853648_0001_r_000000_0 is done. And is in the process of commiting
        14/12/23 17:11:26 INFO mapred.LocalJobRunner: reduce > reduce
        14/12/23 17:11:26 INFO mapred.Task: Task 'attempt_local139853648_0001_r_000000_0' done.
        14/12/23 17:11:26 WARN mapred.FileOutputCommitter: Output path is null in cleanup
        14/12/23 17:11:26 INFO mapred.JobClient:  map 0% reduce 100%
        14/12/23 17:11:26 INFO mapred.JobClient: Job complete: job_local139853648_0001
        

        Is this normal on hadoop-1.2.1? Or is there a problem caused by us?

        Show
        alparslan.avci Alparslan Avcı added a comment - Talat UYARER , I've tested your patch but when running hadoop jobs, I could not see the job progress of map jobs. You can see the log below: Normal run: 14/12/23 17:04:49 INFO mapred.JobClient: map 0% reduce 0% 14/12/23 17:04:53 INFO mapred.MapTask: Starting flush of map output 14/12/23 17:04:53 INFO mapred.MapTask: Finished spill 0 14/12/23 17:04:53 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting 14/12/23 17:04:54 INFO mapred.LocalJobRunner: 14/12/23 17:04:54 INFO mapred.LocalJobRunner: 14/12/23 17:04:54 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done. 14/12/23 17:04:54 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@30fd4204 14/12/23 17:04:54 INFO mapred.LocalJobRunner: 14/12/23 17:04:54 INFO mapred.Merger: Merging 1 sorted segments 14/12/23 17:04:54 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 532529 bytes 14/12/23 17:04:54 INFO mapred.LocalJobRunner: 14/12/23 17:04:54 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as the one in the databean. 14/12/23 17:04:54 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@436daf20 14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 14/12/23 17:04:54 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 15793@alparslan 14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77a9d8b90017, negotiated timeout = 40000 14/12/23 17:04:54 INFO zookeeper.ZooKeeper: Session: 0x14a77a9d8b90017 closed 14/12/23 17:04:54 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/23 17:04:54 INFO mapreduce.GoraRecordWriter: gora.buffer.write.limit = 10000 14/12/23 17:04:54 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting 14/12/23 17:04:55 INFO mapred.JobClient: map 100% reduce 0% 14/12/23 17:04:57 INFO mapred.LocalJobRunner: reduce > reduce 14/12/23 17:04:57 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done. 14/12/23 17:04:57 WARN mapred.FileOutputCommitter: Output path is null in cleanup 14/12/23 17:04:58 INFO mapred.JobClient: map 100% reduce 100% 14/12/23 17:04:58 INFO mapred.JobClient: Job complete: job_local_0001 When your patch applied: 14/12/23 17:11:23 INFO mapred.JobClient: map 0% reduce 0% 14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b10921001e closed 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/23 17:11:24 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper ensemble=localhost:2181 14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181, baseZNode=/hbase 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77b10921001f, negotiated timeout = 40000 14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b10921001f closed 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/23 17:11:24 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper ensemble=localhost:2181 14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181, baseZNode=/hbase 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77b109210020, negotiated timeout = 40000 14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b109210020 closed 14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/23 17:11:24 INFO mapred.MapTask: Processing split: org.apache.gora.mapreduce.GoraInputSplit@40b86a70 14/12/23 17:11:24 INFO mapreduce.GoraRecordReader: gora.buffer.read.limit = 10000 14/12/23 17:11:24 INFO mapred.MapTask: io.sort.mb = 100 14/12/23 17:11:24 INFO mapred.MapTask: data buffer = 79691776/99614720 14/12/23 17:11:24 INFO mapred.MapTask: record buffer = 262144/327680 14/12/23 17:11:25 INFO mapred.MapTask: Starting flush of map output 14/12/23 17:11:25 INFO mapred.MapTask: Finished spill 0 14/12/23 17:11:25 INFO mapred.Task: Task:attempt_local139853648_0001_m_000000_0 is done. And is in the process of commiting 14/12/23 17:11:25 INFO mapred.LocalJobRunner: 14/12/23 17:11:25 INFO mapred.Task: Task 'attempt_local139853648_0001_m_000000_0' done. 14/12/23 17:11:25 INFO mapred.LocalJobRunner: Finishing task: attempt_local139853648_0001_m_000000_0 14/12/23 17:11:25 INFO mapred.LocalJobRunner: Map task executor complete. 14/12/23 17:11:25 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3589085 14/12/23 17:11:25 INFO mapred.LocalJobRunner: 14/12/23 17:11:25 INFO mapred.Merger: Merging 1 sorted segments 14/12/23 17:11:25 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 532529 bytes 14/12/23 17:11:25 INFO mapred.LocalJobRunner: 14/12/23 17:11:25 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as the one in the databean. 14/12/23 17:11:25 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper ensemble=localhost:2181 14/12/23 17:11:25 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181, baseZNode=/hbase 14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a77b109210021, negotiated timeout = 40000 14/12/23 17:11:25 INFO zookeeper.ZooKeeper: Session: 0x14a77b109210021 closed 14/12/23 17:11:25 INFO zookeeper.ClientCnxn: EventThread shut down 14/12/23 17:11:25 INFO mapreduce.GoraRecordWriter: gora.buffer.write.limit = 10000 14/12/23 17:11:26 INFO mapred.Task: Task:attempt_local139853648_0001_r_000000_0 is done. And is in the process of commiting 14/12/23 17:11:26 INFO mapred.LocalJobRunner: reduce > reduce 14/12/23 17:11:26 INFO mapred.Task: Task 'attempt_local139853648_0001_r_000000_0' done. 14/12/23 17:11:26 WARN mapred.FileOutputCommitter: Output path is null in cleanup 14/12/23 17:11:26 INFO mapred.JobClient: map 0% reduce 100% 14/12/23 17:11:26 INFO mapred.JobClient: Job complete: job_local139853648_0001 Is this normal on hadoop-1.2.1? Or is there a problem caused by us?
        Hide
        alparslan.avci Alparslan Avcı added a comment -

        BTW, the mapreduce job results are ok, no problem occurs.

        Show
        alparslan.avci Alparslan Avcı added a comment - BTW, the mapreduce job results are ok, no problem occurs.
        Hide
        talat Talat UYARER added a comment -

        I create a hadoop 2 version. I upgrade all dependecies to Hadoop 2. Actually I do not change a lot of things. If you review my patch I will be glad. BTW We are little bit slow for development. IMHO hadoop 1.x is deprecated and we continue with hadoop 2.x support. I can write a how to Gora work on hadoop 1 documentation.

        Show
        talat Talat UYARER added a comment - I create a hadoop 2 version. I upgrade all dependecies to Hadoop 2. Actually I do not change a lot of things. If you review my patch I will be glad. BTW We are little bit slow for development. IMHO hadoop 1.x is deprecated and we continue with hadoop 2.x support. I can write a how to Gora work on hadoop 1 documentation.
        Hide
        lewismc Lewis John McGibbney added a comment -

        Alparslan Avcı

        14/12/23 17:11:25 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as the one in the databean.
        

        You might want to get this sorted out

        Show
        lewismc Lewis John McGibbney added a comment - Alparslan Avcı 14/12/23 17:11:25 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as the one in the databean. You might want to get this sorted out
        Hide
        lewismc Lewis John McGibbney added a comment - - edited

        Talat UYARER I've checked out this patch and I think it looks really good. I've tested it on my local version of trunk and it is passing all existing tests. I also agree with your comment that the shift to having Hadoop 2.X as standard with option to switch back to 1.X is best way to go.
        It would be great if we could further document the Shims in the placeholder I opened up however the documentation can follow shortly after I am sure.
        Does anyone else have cycles to try this patch out locally and comment? It would be great to get this patch in and close it off if we can get some level of consensus.
        Great work Talat UYARER

        Show
        lewismc Lewis John McGibbney added a comment - - edited Talat UYARER I've checked out this patch and I think it looks really good. I've tested it on my local version of trunk and it is passing all existing tests. I also agree with your comment that the shift to having Hadoop 2.X as standard with option to switch back to 1.X is best way to go. It would be great if we could further document the Shims in the placeholder I opened up however the documentation can follow shortly after I am sure. Does anyone else have cycles to try this patch out locally and comment? It would be great to get this patch in and close it off if we can get some level of consensus. Great work Talat UYARER
        Hide
        lewismc Lewis John McGibbney added a comment -

        Anyone else for the most recent patch?

        Show
        lewismc Lewis John McGibbney added a comment - Anyone else for the most recent patch?
        Hide
        hsaputra Henry Saputra added a comment -

        Works in my local env.

        Just small nit, we could just remove the commented maven pom.xml:
        <!-- <hbase.version>0.94.14</hbase.version> -->
        with versioning we can always revert back.

        Show
        hsaputra Henry Saputra added a comment - Works in my local env. Just small nit, we could just remove the commented maven pom.xml: <!-- <hbase.version>0.94.14</hbase.version> --> with versioning we can always revert back.
        Hide
        lewismc Lewis John McGibbney added a comment -

        Nice catch Henry Saputra I would like to commit E.O.D if it is OK. I will be using this module this coming week and would like to get as much testing done as I can over the weekend.
        The HBase cluster I'll be writing in to as explained on the list is CDH, so we will see how it goes.

        Show
        lewismc Lewis John McGibbney added a comment - Nice catch Henry Saputra I would like to commit E.O.D if it is OK. I will be using this module this coming week and would like to get as much testing done as I can over the weekend. The HBase cluster I'll be writing in to as explained on the list is CDH, so we will see how it goes.
        Hide
        lewismc Lewis John McGibbney added a comment -

        commit e1eb90a369851805ef8e1bd1a89f71cd1644f73b
        Author: Lewis John McGibbney <lewis.j....nasa.gov>
        Date: Thu Jan 8 18:54:42 2015 -0800

        GORA-375 Upgrade to HBase 0.98

        commit 30bb3af96d5c70ba2e70dca183c62fb6a49c343c
        Author: Lewis John McGibbney <lewis.j....nasa.gov>
        Date: Thu Jan 8 17:55:18 2015 -0800

        GORA-375 Upgrade HBase to 0.98

        Show
        lewismc Lewis John McGibbney added a comment - commit e1eb90a369851805ef8e1bd1a89f71cd1644f73b Author: Lewis John McGibbney <lewis.j....nasa.gov> Date: Thu Jan 8 18:54:42 2015 -0800 GORA-375 Upgrade to HBase 0.98 commit 30bb3af96d5c70ba2e70dca183c62fb6a49c343c Author: Lewis John McGibbney <lewis.j....nasa.gov> Date: Thu Jan 8 17:55:18 2015 -0800 GORA-375 Upgrade HBase to 0.98
        Hide
        lewismc Lewis John McGibbney added a comment -

        great work on this one Talat UYARER
        Henry Saputra I made sure to remove that line for you in pom.xml
        Once I get a stable build on Jenkins I am going to make an announcement on the website that we've skipped to Hadoop 2.X as default implementation now with backwards support for Hadoop 1.X

        Show
        lewismc Lewis John McGibbney added a comment - great work on this one Talat UYARER Henry Saputra I made sure to remove that line for you in pom.xml Once I get a stable build on Jenkins I am going to make an announcement on the website that we've skipped to Hadoop 2.X as default implementation now with backwards support for Hadoop 1.X
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in gora-trunk #1359 (See https://builds.apache.org/job/gora-trunk/1359/)
        GORA-375 Upgrade HBase to 0.98 (lewis.j.mcgibbney: rev 30bb3af96d5c70ba2e70dca183c62fb6a49c343c)

        • pom.xml
        • gora-hbase/src/main/java/org/apache/gora/hbase/util/HBaseByteInterface.java
        • gora-cassandra/pom.xml
        • gora-core/pom.xml
        • gora-shims-hadoop2/pom.xml
        • gora-mongodb/pom.xml
        • gora-shims-hadoop/pom.xml
        • gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java
        • gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseMapping.java
        • gora-shims-hadoop1/pom.xml
        • gora-goraci/pom.xml
        • gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseTableConnection.java
        • gora-accumulo/pom.xml
        • gora-core/src/test/java/org/apache/gora/mapreduce/TestGoraInputFormat.java
        • gora-hbase/pom.xml
        • gora-tutorial/pom.xml
          GORA-375 Upgrade to HBase 0.98 (lewis.j.mcgibbney: rev e1eb90a369851805ef8e1bd1a89f71cd1644f73b)
        • CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in gora-trunk #1359 (See https://builds.apache.org/job/gora-trunk/1359/ ) GORA-375 Upgrade HBase to 0.98 (lewis.j.mcgibbney: rev 30bb3af96d5c70ba2e70dca183c62fb6a49c343c) pom.xml gora-hbase/src/main/java/org/apache/gora/hbase/util/HBaseByteInterface.java gora-cassandra/pom.xml gora-core/pom.xml gora-shims-hadoop2/pom.xml gora-mongodb/pom.xml gora-shims-hadoop/pom.xml gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseMapping.java gora-shims-hadoop1/pom.xml gora-goraci/pom.xml gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseTableConnection.java gora-accumulo/pom.xml gora-core/src/test/java/org/apache/gora/mapreduce/TestGoraInputFormat.java gora-hbase/pom.xml gora-tutorial/pom.xml GORA-375 Upgrade to HBase 0.98 (lewis.j.mcgibbney: rev e1eb90a369851805ef8e1bd1a89f71cd1644f73b) CHANGES.txt

          People

          • Assignee:
            talat Talat UYARER
            Reporter:
            yuzhihong@gmail.com Ted Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development