Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3
    • Fix Version/s: 0.4
    • Component/s: gora-hbase
    • Labels:
      None

      Description

      We haven't touched the HBase versioning in a good while. When a new user heads over to the HBase site, they are directed to the 'stable' release which is currently sitting at 0.94.4.

      I realise that we have (legacy) support for the 0.90.X branch of HBase, but from what I can see, there is no current justification for this decision and it is also not within any strategic short/medium/long term objectives of Gora.

      This issue should

      *Enable us to discuss what Hbase branch we wish to support moving forward
      *Actually implement the upgrade which gathers most consensus.

        Issue Links

          Activity

          Hide
          Lewis John McGibbney added a comment -

          Ultimately merged and committed to trunk codebase @r1586888

          Show
          Lewis John McGibbney added a comment - Ultimately merged and committed to trunk codebase @r1586888
          Hide
          Lewis John McGibbney added a comment -

          Hi Sertug. In all honesty, you can try chaging the pom.xml hbase artifact version, compile the code and start from there. This is more or less an entire upgrade of the (client/abstract) HBase api in Gora.
          If you have time and effort then even a start along the lines Henry discusses would be ideal... I think.

          Show
          Lewis John McGibbney added a comment - Hi Sertug. In all honesty, you can try chaging the pom.xml hbase artifact version, compile the code and start from there. This is more or less an entire upgrade of the (client/abstract) HBase api in Gora. If you have time and effort then even a start along the lines Henry discusses would be ideal... I think.
          Hide
          Sertug Kaya added a comment -

          Hi all;
          We also would like to be able to work with Gora 0.3 and HBase 0.94.9 .
          Is there anything that you can point out so we can contribute to this issue?

          Show
          Sertug Kaya added a comment - Hi all; We also would like to be able to work with Gora 0.3 and HBase 0.94.9 . Is there anything that you can point out so we can contribute to this issue?
          Hide
          yasin tamer added a comment -

          Hi Henry Saputra we also want to use Gora 0.3 with Hbase 0.94.9, i wonder is there any approximate time for finishing this development?

          Show
          yasin tamer added a comment - Hi Henry Saputra we also want to use Gora 0.3 with Hbase 0.94.9, i wonder is there any approximate time for finishing this development?
          Hide
          Henry Saputra added a comment - - edited

          I propose we go with HBase 0.94.8 (or .9) with Avro 1.7 so we could leverage work for GORA-94. Enis sadi HBAse client does not have dependency on Avro so we could exclude from pom dependency tree.

          Oracle NoSQL and Voldermort is not part of Gora yet so they can wait till dynamic classloader for each module is implemented.

          Show
          Henry Saputra added a comment - - edited I propose we go with HBase 0.94.8 (or .9) with Avro 1.7 so we could leverage work for GORA-94 . Enis sadi HBAse client does not have dependency on Avro so we could exclude from pom dependency tree. Oracle NoSQL and Voldermort is not part of Gora yet so they can wait till dynamic classloader for each module is implemented.
          Hide
          Scott Stults added a comment -

          The other thing is, we'd be able to support multiple versions of the same store, like both HBase 0.92 and 0.94. So maybe if we go that route we should also introduce a store version number somewhere.

          Show
          Scott Stults added a comment - The other thing is, we'd be able to support multiple versions of the same store, like both HBase 0.92 and 0.94. So maybe if we go that route we should also introduce a store version number somewhere.
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          We also have to keep in mind our different Avro version dependencies:

          HBase 0.94.8 : avro-1.5.3
          Oracle Nosql : avro-1.6.3
          Voldemort : avro-1.4.0

          I think what Scott proposed over in GORA-209 for changing the way class loading was being done could also work in here. I mean we need different versions of Avro in different projects, and our supported projects might not want to share the same version.
          What do you guys think?

          Show
          Renato Javier Marroquín Mogrovejo added a comment - We also have to keep in mind our different Avro version dependencies: HBase 0.94.8 : avro-1.5.3 Oracle Nosql : avro-1.6.3 Voldemort : avro-1.4.0 I think what Scott proposed over in GORA-209 for changing the way class loading was being done could also work in here. I mean we need different versions of Avro in different projects, and our supported projects might not want to share the same version. What do you guys think?
          Hide
          Lewis John McGibbney added a comment -

          This is a great point Alfonso.
          We might introduce more problems by using most recent Avro stuff. Not every project (from within the hadoop ecosystem) uses cutting edge Avro stuff. My personal goal is to deal with the Avro stuff in line wiht this HBase upgrade... these seem to be the two aspects of Gora which are keeping us back right now.
          I am not HBase literate... I don't know the API... but I am more than willing to get my hands dirty.
          If you guys can decide upon a strategy to get this working... I will chip in my bit if and where I can.

          Show
          Lewis John McGibbney added a comment - This is a great point Alfonso. We might introduce more problems by using most recent Avro stuff. Not every project (from within the hadoop ecosystem) uses cutting edge Avro stuff. My personal goal is to deal with the Avro stuff in line wiht this HBase upgrade... these seem to be the two aspects of Gora which are keeping us back right now. I am not HBase literate... I don't know the API... but I am more than willing to get my hands dirty. If you guys can decide upon a strategy to get this working... I will chip in my bit if and where I can.
          Hide
          Alfonso Nishikawa added a comment -

          Hi.
          @Enis: Great info. Thank you very much!
          What Avro version do we choose? I find fine 1.5.3 ... but no reasons

          Show
          Alfonso Nishikawa added a comment - Hi. @Enis: Great info. Thank you very much! What Avro version do we choose? I find fine 1.5.3 ... but no reasons
          Hide
          Enis Soztutar added a comment -

          I would suggest going with the HBase 0.94 releases once we are at it. 0.92 and 0.94 versions are mostly compatible at least that is the plan. There were some minor issues, that were solved in later versions of 0.92.x and 0.94.x releases. From Gora's perspective, Gora can move to 0.94.9 (coming out shortly). The jars between 0.92 and 0.94 are swappable as well. So any client wanting to go with 0.92 can still replace the jars.

          The bigger concern is avro jars. HBase depends on avro 1.5.3. I don't think you can swap with 1.7. However, from client side, HBase should not depend on Avro. So we can exclude that dependency from Hbase.

          Show
          Enis Soztutar added a comment - I would suggest going with the HBase 0.94 releases once we are at it. 0.92 and 0.94 versions are mostly compatible at least that is the plan. There were some minor issues, that were solved in later versions of 0.92.x and 0.94.x releases. From Gora's perspective, Gora can move to 0.94.9 (coming out shortly). The jars between 0.92 and 0.94 are swappable as well. So any client wanting to go with 0.92 can still replace the jars. The bigger concern is avro jars. HBase depends on avro 1.5.3. I don't think you can swap with 1.7. However, from client side, HBase should not depend on Avro. So we can exclude that dependency from Hbase.
          Hide
          Henry Saputra added a comment -

          My only concern is that the upgrade dependencies are painful process and if we could just move to HBase 0.94 client I would say we take that hard route now.

          Enis Soztutar could probably help us answering this question. The other issue is the transitive dependencies of such upgrade. I believe Lewis John McGibbney mentioned about ZK issue with Oracle data store.

          I am proposing creating GORA-201 branch to start this development.

          Show
          Henry Saputra added a comment - My only concern is that the upgrade dependencies are painful process and if we could just move to HBase 0.94 client I would say we take that hard route now. Enis Soztutar could probably help us answering this question. The other issue is the transitive dependencies of such upgrade. I believe Lewis John McGibbney mentioned about ZK issue with Oracle data store. I am proposing creating GORA-201 branch to start this development.
          Hide
          Alfonso Nishikawa added a comment - - edited

          Hi all.
          My thoughts about this issue are: "upgrade to the minimum client which allows all our desired functionality and has compatibility with subsequent versions, and check if client libraries are interchangeable".
          For example, just answer questions like:

          • Is HBase client 0.92.0 compatible with server 0.94.4?
          • Can HBase client 0.92.0 be substituted with client jar 0.94.4?
          • Is HBase client 0.92.0 compatible with server 0.94.7?
          • Can HBase client jar 0.92.0 be substituted with client jar 0.94.7?
          • Can Avro 1.5.3 be substituted with jar Avro 1.7?

          In this way we could assure it will work with a range of versions just changing the dependence. A detail is that, for example, we would not have the last functionalities from HBase client 0.94.7, but this does not matter since we (by definition) don't use them (just for example, functionalities up to 0.92.0).

          Show
          Alfonso Nishikawa added a comment - - edited Hi all. My thoughts about this issue are: "upgrade to the minimum client which allows all our desired functionality and has compatibility with subsequent versions, and check if client libraries are interchangeable". For example, just answer questions like: Is HBase client 0.92.0 compatible with server 0.94.4? Can HBase client 0.92.0 be substituted with client jar 0.94.4? Is HBase client 0.92.0 compatible with server 0.94.7? Can HBase client jar 0.92.0 be substituted with client jar 0.94.7? Can Avro 1.5.3 be substituted with jar Avro 1.7? In this way we could assure it will work with a range of versions just changing the dependence. A detail is that, for example, we would not have the last functionalities from HBase client 0.94.7, but this does not matter since we (by definition) don't use them (just for example, functionalities up to 0.92.0).
          Hide
          Henry Saputra added a comment -

          I will drive the HBase upgrade initiative. Looks like lot of blocker issues start showing up because of this

          Show
          Henry Saputra added a comment - I will drive the HBase upgrade initiative. Looks like lot of blocker issues start showing up because of this
          Hide
          Lewis John McGibbney added a comment -

          The hbase dep IGora seems to stagnated somewhat.
          I am not keeping up with hbase but do know that we use an old api.
          Eds avro patch is another kettle of post 0.3 fish but linked to this indeed.

          Show
          Lewis John McGibbney added a comment - The hbase dep IGora seems to stagnated somewhat. I am not keeping up with hbase but do know that we use an old api. Eds avro patch is another kettle of post 0.3 fish but linked to this indeed.
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Any ideas on this one? Should we start looking into Ed's Avro patch in order to start upgrading libraries?

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Any ideas on this one? Should we start looking into Ed's Avro patch in order to start upgrading libraries?
          Hide
          Lewis John McGibbney added a comment -

          One more reason for us to upgrade HBase API usage/dependency in Gora

          http://www.mail-archive.com/user%40nutch.apache.org/msg08700.html

          Show
          Lewis John McGibbney added a comment - One more reason for us to upgrade HBase API usage/dependency in Gora http://www.mail-archive.com/user%40nutch.apache.org/msg08700.html

            People

            • Assignee:
              Henry Saputra
              Reporter:
              Lewis John McGibbney
            • Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development