Uploaded image for project: 'Apache Gora'
  1. Apache Gora
  2. GORA-217

[GSoC2013] Oracle NoSQL datastore for Gora

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.9
    • Component/s: gora-oracle
    • Labels:

      Description

      Research the Oracle NoSQL Community Edition Data model, features and functionality
      Analyze existing gora-datastore implementations E.g. gora-cassandra, gora-hbase, gora-sql, gora-accumulo and gora-dynamodb
      compare all of the above to determine when the gora-oraclenosql data store should be used as an alternative to other
      Review the Oracle NoSQL Database API and identify areas of key importance
      Document the conclusions in the form of the new Java coded gora-oraclenosql datastore which will be attached to the Gora wiki and actively reviewed throughout the project.

      Very useful link
      http://docs.oracle.com/cd/NOSQL/html/GettingStartedGuide/

      1. GSoC2013_Final_Project_Report.pdf
        1.16 MB
        Apostolos Giannakidis
      2. immm_2012_4_10_20050.pdf
        343 kB
        Lewis John McGibbney
      3. Gora-Oracle NoSQL - Project Proposal.docx
        32 kB
        Apostolos Giannakidis

        Activity

        Hide
        lewismc Lewis John McGibbney added a comment -

        Hi Apostolos thanks for adding this.
        You can see TheASF's GSoC2013 submissions below
        http://s.apache.org/0Xh

        Show
        lewismc Lewis John McGibbney added a comment - Hi Apostolos thanks for adding this. You can see TheASF's GSoC2013 submissions below http://s.apache.org/0Xh
        Hide
        db43 Dybra Grande added a comment -

        I am really interested in learning and implementing Oracle NoSQL datastore for Gora.

        I have 2 questions:

        1. The research, analyzing and review of information – will this need to be submitted?

        2. I would like to get more information on this project. Who can i talk to?

        Show
        db43 Dybra Grande added a comment - I am really interested in learning and implementing Oracle NoSQL datastore for Gora. I have 2 questions: 1. The research, analyzing and review of information – will this need to be submitted? 2. I would like to get more information on this project. Who can i talk to?
        Hide
        lewismc Lewis John McGibbney added a comment -

        Hi Dybra, yes you can apply for the project no problem. Please see the Gora archives [0] both user@ and dev@ and check for my previous emails on this topic. You have until the 4th of May to submit your proposal to Google Melange. There is also details on this on the mailing lists.
        You should also check out the Gora wiki and create a wiki page for your proposal.
        Thank you
        Lewis

        [0] http://www.mail-archive.com/user%40gora.apache.org/

        Show
        lewismc Lewis John McGibbney added a comment - Hi Dybra, yes you can apply for the project no problem. Please see the Gora archives [0] both user@ and dev@ and check for my previous emails on this topic. You have until the 4th of May to submit your proposal to Google Melange. There is also details on this on the mailing lists. You should also check out the Gora wiki and create a wiki page for your proposal. Thank you Lewis [0] http://www.mail-archive.com/user%40gora.apache.org/
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment - - edited

        Added a draft of my proposal for the Gora-Oracle NoSQL GSoC project.

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - - edited Added a draft of my proposal for the Gora-Oracle NoSQL GSoC project.
        Hide
        alfonso.nishikawa Alfonso Nishikawa added a comment -

        Hi, Apostolos. Very interesting draft. I have nothing to remark, so cheer up!

        Show
        alfonso.nishikawa Alfonso Nishikawa added a comment - Hi, Apostolos. Very interesting draft. I have nothing to remark, so cheer up!
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        Hey Alfonso.

        Thank you for your reply. I am glad that you find interesting my proposal.
        I would be interested in any ideas for expanding the scope so that the Gora community could benefit more from this project.

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - Hey Alfonso. Thank you for your reply. I am glad that you find interesting my proposal. I would be interested in any ideas for expanding the scope so that the Gora community could benefit more from this project.
        Hide
        lewismc Lewis John McGibbney added a comment -

        Hi Apostolis, Please get this in to Melange ASAP. Thanks

        Show
        lewismc Lewis John McGibbney added a comment - Hi Apostolis, Please get this in to Melange ASAP. Thanks
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        Hey Lewis,

        My proposal is officially submitted!

        Here is the URL:
        https://google-melange.appspot.com/gsoc/proposal/review/google/gsoc2013/apgiannakidis/1

        I look forward to your feedback and comments!

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - Hey Lewis, My proposal is officially submitted! Here is the URL: https://google-melange.appspot.com/gsoc/proposal/review/google/gsoc2013/apgiannakidis/1 I look forward to your feedback and comments!
        Hide
        lewismc Lewis John McGibbney added a comment -

        Paper explaining Oracle NoSQL

        Show
        lewismc Lewis John McGibbney added a comment - Paper explaining Oracle NoSQL
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        Project accepted in GSoc! Let's go!!!

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - Project accepted in GSoc! Let's go!!!
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        A small post that I wrote while I was experimenting with the Oracle NoSQL API:
        http://www.giannakidis.info/post/54370539362/experimenting-with-kvstore-multigetiterator

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - A small post that I wrote while I was experimenting with the Oracle NoSQL API: http://www.giannakidis.info/post/54370539362/experimenting-with-kvstore-multigetiterator
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        Another small post that I wrote that document the existing projects that integrate with Oracle NoSQL database:
        http://www.giannakidis.info/post/54652821873/integrating-with-oracle-nosql-existing-solutions

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - Another small post that I wrote that document the existing projects that integrate with Oracle NoSQL database: http://www.giannakidis.info/post/54652821873/integrating-with-oracle-nosql-existing-solutions
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        Steve O'Hearn, author of the very good book: "OCA Oracle Database SQL Expert Exam Guide: Exam 1Z0-047", wrote a very beautiful blog post about this project. Check it out: http://blog.corbinian.com/node/89

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - Steve O'Hearn, author of the very good book: "OCA Oracle Database SQL Expert Exam Guide: Exam 1Z0-047", wrote a very beautiful blog post about this project. Check it out: http://blog.corbinian.com/node/89
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment - - edited

        This is the Git repos for this project. This is already known to my mentor, Lewis, and I am also sharing it with the rest of the community.
        I would be grateful for your feedback.

        https://github.com/maestros/gora-oraclenosql/tree/master/gora-oracle

        Best regards,
        Apos

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - - edited This is the Git repos for this project. This is already known to my mentor, Lewis, and I am also sharing it with the rest of the community. I would be grateful for your feedback. https://github.com/maestros/gora-oraclenosql/tree/master/gora-oracle Best regards, Apos
        Hide
        lewismc Lewis John McGibbney added a comment -

        Code is coming on very nicely.

        Show
        lewismc Lewis John McGibbney added a comment - Code is coming on very nicely.
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        Hello all,

        Just an update on this.
        I have pushed the latest code that implements the Query and Delete operations. This is a big milestone in this project as most of the test cases now complete successfully.

        As usual, you can find the source code in the github repos:
        https://github.com/maestros/gora-oraclenosql/tree/master/gora-oracle

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - Hello all, Just an update on this. I have pushed the latest code that implements the Query and Delete operations. This is a big milestone in this project as most of the test cases now complete successfully. As usual, you can find the source code in the github repos: https://github.com/maestros/gora-oraclenosql/tree/master/gora-oracle
        Hide
        renato2099 Renato Javier Marroquín Mogrovejo added a comment -

        Hi Apos,

        This is great! You are doing great progress!
        I just noticed that in class [1] there are some constants used in as TimeUnit such as "MILLISECONDS", "DAYS", etc, that I think should go in a different file (OracleStoreConstants? maybe?) as they come directly from the properties file, so users can find more easily those values and we can modify them without actually going touching the OracleDataStore class.

        [1] https://github.com/maestros/gora-oraclenosql/blob/master/gora-oracle/src/main/java/org/apache/gora/oracle/store/OracleStore.java

        Show
        renato2099 Renato Javier Marroquín Mogrovejo added a comment - Hi Apos, This is great! You are doing great progress! I just noticed that in class [1] there are some constants used in as TimeUnit such as "MILLISECONDS", "DAYS", etc, that I think should go in a different file (OracleStoreConstants? maybe?) as they come directly from the properties file, so users can find more easily those values and we can modify them without actually going touching the OracleDataStore class. [1] https://github.com/maestros/gora-oraclenosql/blob/master/gora-oracle/src/main/java/org/apache/gora/oracle/store/OracleStore.java
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        Nice one Renato.
        The change, as per your suggestion, has been committed. Thank you very much!

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - Nice one Renato. The change, as per your suggestion, has been committed. Thank you very much!
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        Union support was also added to the datastore. This means that ALL test cases now pass with success, with the exception of the testGetPartitions(). Any feedback for the produced code so far would be very much appreciated.

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - Union support was also added to the datastore. This means that ALL test cases now pass with success, with the exception of the testGetPartitions(). Any feedback for the produced code so far would be very much appreciated.
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        A new blog post that I wrote regarding the basics of Apache Gora and the data model that I used for the Gora-Oracle data store.

        http://www.giannakidis.info/post/58960528544/apache-gora-data-modelling-and-data-bean-to-oracle

        Cheers,
        Apos

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - A new blog post that I wrote regarding the basics of Apache Gora and the data model that I used for the Gora-Oracle data store. http://www.giannakidis.info/post/58960528544/apache-gora-data-modelling-and-data-bean-to-oracle Cheers, Apos
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment -

        I attached the final project report for this project. This is the main deliverable, as far as the documentation part is concerned.

        I would like to thank Lewis John Mcgibbney, Alfonso Nishikawa, Renato Marroquín Mogrovejo and the rest of the Apache Gora community for their support.

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - I attached the final project report for this project. This is the main deliverable, as far as the documentation part is concerned. I would like to thank Lewis John Mcgibbney, Alfonso Nishikawa, Renato Marroquín Mogrovejo and the rest of the Apache Gora community for their support.
        Hide
        renato2099 Renato Javier Marroquín Mogrovejo added a comment -

        Hey Apos,

        This is for sure an awesome work! Congrats!!!
        I didn't have to go over your whole document, but it'd be great if at least part of all that awesome documentation could go into the site, so we could help newcomers into Gora (:
        And another question is about the resultCache functionality you wrote. Could you point me to the code where you wrote this? I think this is great but it is something we should all evaluate as we should try to be coherent across all data store implementations.
        Once again, awesome work Apos! and hope you can stick around in Gora to keep it moving forward (:

        Show
        renato2099 Renato Javier Marroquín Mogrovejo added a comment - Hey Apos, This is for sure an awesome work! Congrats!!! I didn't have to go over your whole document, but it'd be great if at least part of all that awesome documentation could go into the site, so we could help newcomers into Gora (: And another question is about the resultCache functionality you wrote. Could you point me to the code where you wrote this? I think this is great but it is something we should all evaluate as we should try to be coherent across all data store implementations. Once again, awesome work Apos! and hope you can stick around in Gora to keep it moving forward (:
        Hide
        lewismc Lewis John McGibbney added a comment -

        I shadow Renato's comments entirely, great work Apos.
        A long summer with good focused work and now the code and documentation is at a level of quality that is worthy of congratulations.
        It's been great working on this project with you and I hope you enjoyed it as much as I did.
        Next...
        Get the code into the codebase

        Show
        lewismc Lewis John McGibbney added a comment - I shadow Renato's comments entirely, great work Apos. A long summer with good focused work and now the code and documentation is at a level of quality that is worthy of congratulations. It's been great working on this project with you and I hope you enjoyed it as much as I did. Next... Get the code into the codebase
        Hide
        ap.giannakidis Apostolos Giannakidis added a comment - - edited

        Hey all,

        Sorry for the late reply.

        @Renato Javier Marroquín Mogrovejo, the Result Cache is really in a "draft" version, as I am not 100% sure that it is implemented in a proper way. I need to allocate more time for this to write some test cases to validate its functionality. I must also stress test the whole module and benchmark its performance with and without the Result Cache mechanism in order to see if it actually contributes to the overall performance of the module. I didn't have the time to do this properly within the GSoC timeframe. As for the actual code, you can see the cached object in the OracleQuery class: https://github.com/maestros/gora-oraclenosql/blob/master/gora-oracle/src/main/java/org/apache/gora/oracle/query/OracleQuery.java#L44. The OracleStore.execute() method knows about the cached object and handles it accordingly: https://github.com/maestros/gora-oraclenosql/blob/master/gora-oracle/src/main/java/org/apache/gora/oracle/store/OracleStore.java#L1088. The motive for this feature was the fact that I saw that some test cases call the store.execute() method several times in a raw, with the same query. For such reasons, there is no need to execute the same query again and again, if the query was successfully executed in the first execute() invocation.

        @Lewis John McGibbney and rest of the community, I have been in contact with the Oracle NoSQL database development team and they informed me that they decided to release their kvclient.jar under the Apache License Version 2.0 in their new release, which is planned in a some months from now. This is great news for this project as it resolves the license incompatibility issue and allows this module to be incorporated into Gora's codebase!!

        Show
        ap.giannakidis Apostolos Giannakidis added a comment - - edited Hey all, Sorry for the late reply. @ Renato Javier Marroquín Mogrovejo , the Result Cache is really in a "draft" version, as I am not 100% sure that it is implemented in a proper way. I need to allocate more time for this to write some test cases to validate its functionality. I must also stress test the whole module and benchmark its performance with and without the Result Cache mechanism in order to see if it actually contributes to the overall performance of the module. I didn't have the time to do this properly within the GSoC timeframe. As for the actual code, you can see the cached object in the OracleQuery class: https://github.com/maestros/gora-oraclenosql/blob/master/gora-oracle/src/main/java/org/apache/gora/oracle/query/OracleQuery.java#L44 . The OracleStore.execute() method knows about the cached object and handles it accordingly: https://github.com/maestros/gora-oraclenosql/blob/master/gora-oracle/src/main/java/org/apache/gora/oracle/store/OracleStore.java#L1088 . The motive for this feature was the fact that I saw that some test cases call the store.execute() method several times in a raw, with the same query. For such reasons, there is no need to execute the same query again and again, if the query was successfully executed in the first execute() invocation. @ Lewis John McGibbney and rest of the community, I have been in contact with the Oracle NoSQL database development team and they informed me that they decided to release their kvclient.jar under the Apache License Version 2.0 in their new release, which is planned in a some months from now. This is great news for this project as it resolves the license incompatibility issue and allows this module to be incorporated into Gora's codebase!!
        Hide
        lewismc Lewis John McGibbney added a comment -

        Thanks for heads up Apos.
        From my point of view (having been out for a while) I am back online to get the Avro upgrade done. After that I will be working on getting this module integrated into trunk. I will be working closely with Apos to propose a patch for this module so that we can get it in to trunk.
        Regarding a release for 0.4... lets go to dev@ and we can discuss the timeframe (soon) + which issues we wish to address.
        Thanks Apos + others.

        Show
        lewismc Lewis John McGibbney added a comment - Thanks for heads up Apos. From my point of view (having been out for a while) I am back online to get the Avro upgrade done. After that I will be working on getting this module integrated into trunk. I will be working closely with Apos to propose a patch for this module so that we can get it in to trunk. Regarding a release for 0.4... lets go to dev@ and we can discuss the timeframe (soon) + which issues we wish to address. Thanks Apos + others.

          People

          • Assignee:
            ap.giannakidis Apostolos Giannakidis
            Reporter:
            ap.giannakidis Apostolos Giannakidis
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:

              Development