Details

      Description

      Support 10gen MongoDB datastore in GORA.

      1. org.apache.gora.mongodb.store.TestMongoStore.txt
        44 kB
        Lewis John McGibbney
      2. gora-mongodb-src.tar.gz
        43 kB
        Fabien Poulard
      3. GORA-199v3.patch
        136 kB
        Damien Raude-Morvan
      4. GORA-199v2.patch
        174 kB
        Lewis John McGibbney
      5. GORA-199.patch
        174 kB
        Lewis John McGibbney
      6. GORA-119-trunk.patch
        134 kB
        Talat UYARER

        Issue Links

          Activity

          Hide
          Damien Raude-Morvan added a comment -

          HI Lewis John McGibbney and Henry Saputra !

          Thank you all for your help on getting this properly integrated into Gora!
          I hope to be able to improve this support in coming weeks/months.

          Regards,

          Show
          Damien Raude-Morvan added a comment - HI Lewis John McGibbney and Henry Saputra ! Thank you all for your help on getting this properly integrated into Gora! I hope to be able to improve this support in coming weeks/months. Regards,
          Hide
          Henry Saputra added a comment -

          W000t!! As Lewis had mentioned, thanks for the contributions and patience Damien Raude-Morvan

          Show
          Henry Saputra added a comment - W000t!! As Lewis had mentioned, thanks for the contributions and patience Damien Raude-Morvan
          Hide
          Lewis John McGibbney added a comment -

          A late note, I pulled the commit around 5 minutes ago directly from the branch you have been working on at
          https://github.com/Dictanova/gora --> GORA_199 so it is up-to-date with trunk and your local changes as of ~9 hours ago.

          Show
          Lewis John McGibbney added a comment - A late note, I pulled the commit around 5 minutes ago directly from the branch you have been working on at https://github.com/Dictanova/gora --> GORA_199 so it is up-to-date with trunk and your local changes as of ~9 hours ago.
          Hide
          Lewis John McGibbney added a comment -

          commit e39e9fbcdb020fed1e039a0e7f05ffb356709997
          Merge: a39cfe4 8ac3bf5
          Author: Lewis John McGibbney <lewis.j.mcgibbney@jpl.nasa.gov>
          Date: Wed Jun 4 09:16:44 2014 -0700

          Merge branch 'GORA_199'

          Thank you so much Damien Raude-Morvan for your patience here.
          If you can, please open tickets for further improvements to the module on the Jira and we can address in due course.
          Great work folks.

          Show
          Lewis John McGibbney added a comment - commit e39e9fbcdb020fed1e039a0e7f05ffb356709997 Merge: a39cfe4 8ac3bf5 Author: Lewis John McGibbney <lewis.j.mcgibbney@jpl.nasa.gov> Date: Wed Jun 4 09:16:44 2014 -0700 Merge branch 'GORA_199' Thank you so much Damien Raude-Morvan for your patience here. If you can, please open tickets for further improvements to the module on the Jira and we can address in due course. Great work folks.
          Hide
          Henry Saputra added a comment -

          HI Damien Raude-Morvan, yes we do have to add more descriptive code formatting (maybe template for eclipse or idea) but you can check out https://cwiki.apache.org/confluence/display/GORA/Apache+Gora+Coding+Style for high level guide

          Still need to talk to INFRA to get my Wiki username added as contributor to Apache gora wiki =(

          Show
          Henry Saputra added a comment - HI Damien Raude-Morvan , yes we do have to add more descriptive code formatting (maybe template for eclipse or idea) but you can check out https://cwiki.apache.org/confluence/display/GORA/Apache+Gora+Coding+Style for high level guide Still need to talk to INFRA to get my Wiki username added as contributor to Apache gora wiki =(
          Hide
          Talat UYARER added a comment -

          Hi Damien Raude-Morvan,

          Could you please provide a template for code formating (for Eclipse or IntelliJ IDEA maybe ?)

          You can use Nutch's Eclipse Code Format file. That is in the main folder of Nutch. We wait your improvements

          Show
          Talat UYARER added a comment - Hi Damien Raude-Morvan , Could you please provide a template for code formating (for Eclipse or IntelliJ IDEA maybe ?) You can use Nutch's Eclipse Code Format file. That is in the main folder of Nutch. We wait your improvements
          Hide
          Damien Raude-Morvan added a comment -

          Hi Lewis John McGibbney,

          • I've updated version in GoraMongodbTestDriver to use latest PRODUCTION release
          • I've removed reference to Nutch stuff for my branch
          • Could you please provide a template for code formating (for Eclipse or IntelliJ IDEA maybe ?)
          • TestMongoStore: references to StateManager are in comments but yeah, I need to work a bit on this one to update it for trunk
          Show
          Damien Raude-Morvan added a comment - Hi Lewis John McGibbney , I've updated version in GoraMongodbTestDriver to use latest PRODUCTION release I've removed reference to Nutch stuff for my branch Could you please provide a template for code formating (for Eclipse or IntelliJ IDEA maybe ?) TestMongoStore: references to StateManager are in comments but yeah, I need to work a bit on this one to update it for trunk
          Hide
          Lewis John McGibbney added a comment -

          Great Talat UYARER some comments

          • .version(Version.Main.V2_6) in GoraMongodbTestDriver should be upraded
          • There are many references to Nutch stuff... we should not have this... this was explained in the above conversation
          • Code formatting is not correct.
          • TestMongoStore needs more work. There are references to StateManager!!! Also I doubt thus patch will pass tests... we need to fix that if we want to commit it prior to UNION and filter support.
            I'll not work on these issues this weekend as I have other stuff to do (also I'm updating the Gora site) but will work on them next week.
            Thanks for updating the patch.
          Show
          Lewis John McGibbney added a comment - Great Talat UYARER some comments .version(Version.Main.V2_6) in GoraMongodbTestDriver should be upraded There are many references to Nutch stuff... we should not have this... this was explained in the above conversation Code formatting is not correct. TestMongoStore needs more work. There are references to StateManager!!! Also I doubt thus patch will pass tests... we need to fix that if we want to commit it prior to UNION and filter support. I'll not work on these issues this weekend as I have other stuff to do (also I'm updating the Gora site) but will work on them next week. Thanks for updating the patch.
          Hide
          Talat UYARER added a comment -

          Hi All,

          I create a patch for our trunk. Now It can compile. But I don't work for Union and Filter support. This patch generated for migrating Damien Raude-Morvan's work to our trunk.

          Damien Raude-Morvan, I have time at Sunday, Can I work about Union and filter support ?

          Show
          Talat UYARER added a comment - Hi All, I create a patch for our trunk. Now It can compile. But I don't work for Union and Filter support. This patch generated for migrating Damien Raude-Morvan 's work to our trunk. Damien Raude-Morvan , I have time at Sunday, Can I work about Union and filter support ?
          Hide
          Lewis John McGibbney added a comment -

          I'm a big +1 for this. We are just in to development mode again (as if we left it!!!) so filters and UNION type support can wait

          Show
          Lewis John McGibbney added a comment - I'm a big +1 for this. We are just in to development mode again (as if we left it!!!) so filters and UNION type support can wait
          Hide
          Lewis John McGibbney added a comment -

          Damien Raude-Morvan, please DO NOT go out of your way to produce a patch for SVN.
          We are going to VOTE to move Gora to Git... it should not be a pain for people like yourself to contribute your great code contributions to Gora.
          We really appreciate you contributions and patience.
          I'll update this thread once we've VOTE;d.

          Show
          Lewis John McGibbney added a comment - Damien Raude-Morvan , please DO NOT go out of your way to produce a patch for SVN. We are going to VOTE to move Gora to Git... it should not be a pain for people like yourself to contribute your great code contributions to Gora. We really appreciate you contributions and patience. I'll update this thread once we've VOTE;d.
          Hide
          Damien Raude-Morvan added a comment -

          Okay, so I'll try to finalize a proper patch during this evening and submit it to this JIRA issue.

          Cheers,

          Show
          Damien Raude-Morvan added a comment - Okay, so I'll try to finalize a proper patch during this evening and submit it to this JIRA issue. Cheers,
          Hide
          Henry Saputra added a comment -

          +1 for Talat's idea. We could drop Union and Filter support in separate patch for MongoDD store.

          Show
          Henry Saputra added a comment - +1 for Talat's idea. We could drop Union and Filter support in separate patch for MongoDD store.
          Hide
          Talat UYARER added a comment -

          Hi Damien Raude-Morvan,

          1/ new handling of union type. Is there a guideline a how its managed by other store in Gora ?

          Every store handle with different way. In Cassandra store use Union Index for stroing Union type, In Hbase and Solr they support only nullable type Union ["null",type]. IMHO you should a patch that compatible with trunk. We can help you for the UNION and Filter support

          Show
          Talat UYARER added a comment - Hi Damien Raude-Morvan , 1/ new handling of union type. Is there a guideline a how its managed by other store in Gora ? Every store handle with different way. In Cassandra store use Union Index for stroing Union type, In Hbase and Solr they support only nullable type Union ["null",type] . IMHO you should a patch that compatible with trunk. We can help you for the UNION and Filter support
          Hide
          Damien Raude-Morvan added a comment -

          Hi Henry Saputra and Lewis John McGibbney,

          If possible, I would like to avoid exchanging patch (ie. exporting git without prefix) but if you prefer to work with patch I'll try to adapt
          I'll postpone filter enabled scan to a latter release I think. So only blocking issue is to get UNION support fixed (I've already worked in it previously to GORA_94 merge)

          I've added link to my Gora Gradle Plugin as a comment in GORA-277.

          Cheers,

          Show
          Damien Raude-Morvan added a comment - Hi Henry Saputra and Lewis John McGibbney , If possible, I would like to avoid exchanging patch (ie. exporting git without prefix) but if you prefer to work with patch I'll try to adapt I'll postpone filter enabled scan to a latter release I think. So only blocking issue is to get UNION support fixed (I've already worked in it previously to GORA_94 merge) I've added link to my Gora Gradle Plugin as a comment in GORA-277 . Cheers,
          Hide
          Henry Saputra added a comment -

          Hopefully this link will help to generate SVN compatible patch from Github : http://wiki.apache.org/hadoop/GitAndHadoop

          Show
          Henry Saputra added a comment - Hopefully this link will help to generate SVN compatible patch from Github : http://wiki.apache.org/hadoop/GitAndHadoop
          Hide
          Lewis John McGibbney added a comment -

          Hi Damien Raude-Morvan, can you can please provide a SVN compatible patch for Gora trunk and we can all help to get this in to the trunk codebase.
          I saw your update of the module. Thank you for your persistence on this... it has been a long time coming and will be an excellent contribuition.
          Out of curiosity... do you still have code for the gradel plugin? Maybe we could push this in to trunk!?
          wdyt?

          Show
          Lewis John McGibbney added a comment - Hi Damien Raude-Morvan , can you can please provide a SVN compatible patch for Gora trunk and we can all help to get this in to the trunk codebase. I saw your update of the module. Thank you for your persistence on this... it has been a long time coming and will be an excellent contribuition. Out of curiosity... do you still have code for the gradel plugin? Maybe we could push this in to trunk!? wdyt?
          Hide
          Henry Saputra added a comment -

          CC Alfonso Nishikawa, because he was leading effort to support Union in HBase.
          I believe the support is put in the gora-core on the persistent module so the new datastore should get the support for free?

          For filter enable you can take a look https://issues.apache.org/jira/browse/GORA-119 which has discussions and patch to see how it is implemented for HBase store.

          Show
          Henry Saputra added a comment - CC Alfonso Nishikawa , because he was leading effort to support Union in HBase. I believe the support is put in the gora-core on the persistent module so the new datastore should get the support for free? For filter enable you can take a look https://issues.apache.org/jira/browse/GORA-119 which has discussions and patch to see how it is implemented for HBase store.
          Hide
          Damien Raude-Morvan added a comment -

          Hi all,

          I've worked a bit on merging back trunk into my GORA_199 branch on github
          Current WiP can be seen here : https://github.com/Dictanova/gora/compare/apache:trunk...GORA_199

          I might need some help to understand :
          1/ new handling of union type. Is there a guideline a how its managed by other store in Gora ?
          2/ how to implement filter enable scan (which might be really usefull to enable full power of MongoDB backend)

          Regards,

          Show
          Damien Raude-Morvan added a comment - Hi all, I've worked a bit on merging back trunk into my GORA_199 branch on github Current WiP can be seen here : https://github.com/Dictanova/gora/compare/apache:trunk...GORA_199 I might need some help to understand : 1/ new handling of union type. Is there a guideline a how its managed by other store in Gora ? 2/ how to implement filter enable scan (which might be really usefull to enable full power of MongoDB backend) Regards,
          Hide
          Henry Saputra added a comment -

          HI Damien Raude-Morvan, apologize again for us being late to drive GORA-94. Like Talat just mentioned, the fixes now merge to trunk so would love to get this patch going again.

          Sincerely again for the delay.

          Show
          Henry Saputra added a comment - HI Damien Raude-Morvan , apologize again for us being late to drive GORA-94 . Like Talat just mentioned, the fixes now merge to trunk so would love to get this patch going again. Sincerely again for the delay.
          Hide
          Talat UYARER added a comment -

          Hi Damien Raude-Morvan,

          We released Gora 0.4 Can you try merge with trunk ? If You don't enough time I can help you.

          Show
          Talat UYARER added a comment - Hi Damien Raude-Morvan , We released Gora 0.4 Can you try merge with trunk ? If You don't enough time I can help you.
          Hide
          Henry Saputra added a comment -

          HI Damien, thanks so much for the updated patch and I am so sorry for the very late response.

          I prefer to wait for GORA-94 to merge to trunk then deliver both MongoDB and Lucene stores to Gora.
          We would like to stabilize the core first before adding new storage support hence less changes needed to propagate to all store specific code.

          Apologize again for the delay but we do look more contributions from you.

          Show
          Henry Saputra added a comment - HI Damien, thanks so much for the updated patch and I am so sorry for the very late response. I prefer to wait for GORA-94 to merge to trunk then deliver both MongoDB and Lucene stores to Gora. We would like to stabilize the core first before adding new storage support hence less changes needed to propagate to all store specific code. Apologize again for the delay but we do look more contributions from you.
          Hide
          Damien Raude-Morvan added a comment -

          Hi there!

          I've attached a new patch of my GORA-199 branch (MongoDB Store implementation). I've updated code to reflect that
          1/ I'm waiting for GORA-66 for range query support
          2/ MongoStore don't support fields which use union on 3 differents type

          With those latest changes, all remaining tests are successully executed.
          Let me know if you want to include GORA-94 before this patch so that I can rebase my branch on GORA-94

          Regards,

          Show
          Damien Raude-Morvan added a comment - Hi there! I've attached a new patch of my GORA-199 branch (MongoDB Store implementation). I've updated code to reflect that 1/ I'm waiting for GORA-66 for range query support 2/ MongoStore don't support fields which use union on 3 differents type With those latest changes, all remaining tests are successully executed. Let me know if you want to include GORA-94 before this patch so that I can rebase my branch on GORA-94 Regards,
          Hide
          Henry Saputra added a comment -

          We want GORA-94 for next release I suppose? The HBase upgrade depended on this too.

          Maybe we could get this next and Lucene integration store right after GORA-94 merge to trunk.

          Show
          Henry Saputra added a comment - We want GORA-94 for next release I suppose? The HBase upgrade depended on this too. Maybe we could get this next and Lucene integration store right after GORA-94 merge to trunk.
          Hide
          Lewis John McGibbney added a comment -

          Hi folks, can I suggest the following options for this issue...

          • We test and commit the patch
          • We wait until the next release of Gora (0.4) and then make asserted efforts to review and commit this issue along with the Lucene store recently contributed by Scott Stults.
            Both options have pro's and con's.
            If we commit right now then it means merging work on GORA-94 is going to be more of a pain
            At the end of the day, everything will eventually be upgraded to Avro 1.7.X anyway...
          Show
          Lewis John McGibbney added a comment - Hi folks, can I suggest the following options for this issue... We test and commit the patch We wait until the next release of Gora (0.4) and then make asserted efforts to review and commit this issue along with the Lucene store recently contributed by Scott Stults . Both options have pro's and con's. If we commit right now then it means merging work on GORA-94 is going to be more of a pain At the end of the day, everything will eventually be upgraded to Avro 1.7.X anyway...
          Hide
          lufeng added a comment -

          Hi Damien

          I have patched you v2 version and run the mvn package, it pass the unit test. Where is the 2 last test failure, I did not find them. Can you point them out clearly. Thanks.

          Show
          lufeng added a comment - Hi Damien I have patched you v2 version and run the mvn package, it pass the unit test. Where is the 2 last test failure, I did not find them. Can you point them out clearly. Thanks.
          Hide
          Damien Raude-Morvan added a comment -

          Hi Lewis,

          I lack time to be able to fully work on this issue, I've just uploaded a new patch rebased on current trunk (there is only 2 last test failures).

          Show
          Damien Raude-Morvan added a comment - Hi Lewis, I lack time to be able to fully work on this issue, I've just uploaded a new patch rebased on current trunk (there is only 2 last test failures).
          Hide
          Lewis John McGibbney added a comment -

          Hi Damien, is there anything you want to add to this patch for the time being?
          AFAIK, we have a reasonably comprehensive test suite here, semantics of many operations seems to be sound.
          WHat is missing is an up-to-date patch proposal for trunk to tie in with your comments on the 12/Jul.
          I would really like to seriously review and get this datastore in.

          Show
          Lewis John McGibbney added a comment - Hi Damien, is there anything you want to add to this patch for the time being? AFAIK, we have a reasonably comprehensive test suite here, semantics of many operations seems to be sound. WHat is missing is an up-to-date patch proposal for trunk to tie in with your comments on the 12/Jul. I would really like to seriously review and get this datastore in.
          Hide
          Lewis John McGibbney added a comment -

          Dynamite Damien. If you could keep us updated with the state of the patch/development then it would be greatly appreciated.

          Show
          Lewis John McGibbney added a comment - Dynamite Damien. If you could keep us updated with the state of the patch/development then it would be greatly appreciated.
          Hide
          Damien Raude-Morvan added a comment -

          For the record, my current work is visible on Github :
          https://github.com/drazzib/gora/compare/apache:trunk...GORA_199

          I've fixed another batch of issues, and now :

          Tests run: 39, Failures: 5, Errors: 4, Skipped: 0

          Show
          Damien Raude-Morvan added a comment - For the record, my current work is visible on Github : https://github.com/drazzib/gora/compare/apache:trunk...GORA_199 I've fixed another batch of issues, and now : — Tests run: 39, Failures: 5, Errors: 4, Skipped: 0 —
          Hide
          Lewis John McGibbney added a comment -

          Hi Damien, thanks for this.
          Comments...

          • We need to decide whether we want to retain the Nutch stuff within examples. TBH I am not fussed, however as it is your patch, and if you can clean it up, then I would most certainly back putting it in to the examples directory with a README to explain the example to users
          • Building on the above, if you wish to include but comment out a Nutch dependency within the mongodb pom.xml then it is up to users/devs whether they wish to uncomment it and run the full mongo test suite then that is fine by me as well.
          • I have attached the current failing tests for this module. We need to address these most certainly or at least override and @Ignore them with reasoning as to why they are failing if this patch is to make it in to the codebase.

          This is another huge step forward. Thanks for persisting with this one. I am looking forward to getting this in to the Gora codebase.

          Show
          Lewis John McGibbney added a comment - Hi Damien, thanks for this. Comments... We need to decide whether we want to retain the Nutch stuff within examples. TBH I am not fussed, however as it is your patch, and if you can clean it up, then I would most certainly back putting it in to the examples directory with a README to explain the example to users Building on the above, if you wish to include but comment out a Nutch dependency within the mongodb pom.xml then it is up to users/devs whether they wish to uncomment it and run the full mongo test suite then that is fine by me as well. I have attached the current failing tests for this module. We need to address these most certainly or at least override and @Ignore them with reasoning as to why they are failing if this patch is to make it in to the codebase. This is another huge step forward. Thanks for persisting with this one. I am looking forward to getting this in to the Gora codebase.
          Hide
          Damien Raude-Morvan added a comment -

          Hi,

          I've updated gora-mongodb patch with somes fixes :

          • Add missing src/test/conf/gora.properties to avoid NullPointerException
          • Avoid crash in MongoDBQuery if missing fields
          • Start support of UNION (very early stage)
          • Fix behavior of schemaExists / schemaCreate
          Show
          Damien Raude-Morvan added a comment - Hi, I've updated gora-mongodb patch with somes fixes : Add missing src/test/conf/gora.properties to avoid NullPointerException Avoid crash in MongoDBQuery if missing fields Start support of UNION (very early stage) Fix behavior of schemaExists / schemaCreate
          Hide
          Lewis John McGibbney added a comment -

          If we could get this sorted out then it would be great to get the code in to trunk, get it tested and stabilized before we push 0.4.

          Show
          Lewis John McGibbney added a comment - If we could get this sorted out then it would be great to get the code in to trunk, get it tested and stabilized before we push 0.4.
          Hide
          Lewis John McGibbney added a comment -

          Hi Damien, I uploaded a new patch which attempt sot get the correct gora-mongo-mapping.xml, however I do not think that it is correct as I get NPE when I attempt to run the test suite against the module. I debugged this down to the creation of the employeeStore in the core tests and right enough this is where the GoraException is thrown.
          Here is the mapping file I've tried so far, can you please comment.

          <gora-orm>
             
            <class name="org.apache.gora.examples.generated.Employee" keyClass="java.lang.String" document="frontier">
              <field name="name" docfield="name" type="string"/>
              <field name="dateOfBirth" docfield="dateOfBirth" type="int64"/>
              <field name="ssn" docfield="ssn" type="string"/>
              <field name="salary" docfield="salary" type="int32"/>
              <field name="boss" docfield="boss" type="document"/>
              <field name="webpage" docfield="webpage" type="document"/>
            </class>
          
            <class name="org.apache.gora.examples.generated.WebPage" keyClass="java.lang.String" document="webpage">
              <field name="url" docfield="url" type="string"/>
              <field name="content" docfield="content" type="binary"/>
              <field name="parsedContent" docfield="pContent" type="binary"/>
              <field name="outlinks" docfield="links.out" type="document"/>
              <field name="metadata" docfield="metadata" type="document"/>
            </class>
            
            <class name="org.apache.gora.examples.generated.TokenDatum" keyClass="java.lang.String" keyspace="TokenDatum">
              <field name="count" docfield="count" qualifier="int32"/>
            </class>
            
          
          </gora-orm>
          
          Show
          Lewis John McGibbney added a comment - Hi Damien, I uploaded a new patch which attempt sot get the correct gora-mongo-mapping.xml, however I do not think that it is correct as I get NPE when I attempt to run the test suite against the module. I debugged this down to the creation of the employeeStore in the core tests and right enough this is where the GoraException is thrown. Here is the mapping file I've tried so far, can you please comment. <gora-orm> <class name= "org.apache.gora.examples.generated.Employee" keyClass= "java.lang.String" document= "frontier" > <field name= "name" docfield= "name" type= "string" /> <field name= "dateOfBirth" docfield= "dateOfBirth" type= "int64" /> <field name= "ssn" docfield= "ssn" type= "string" /> <field name= "salary" docfield= "salary" type= "int32" /> <field name= "boss" docfield= "boss" type= "document" /> <field name= "webpage" docfield= "webpage" type= "document" /> </class> <class name= "org.apache.gora.examples.generated.WebPage" keyClass= "java.lang.String" document= "webpage" > <field name= "url" docfield= "url" type= "string" /> <field name= "content" docfield= "content" type= "binary" /> <field name= "parsedContent" docfield= "pContent" type= "binary" /> <field name= "outlinks" docfield= "links.out" type= "document" /> <field name= "metadata" docfield= "metadata" type= "document" /> </class> <class name= "org.apache.gora.examples.generated.TokenDatum" keyClass= "java.lang.String" keyspace= "TokenDatum" > <field name= "count" docfield= "count" qualifier= "int32" /> </class> </gora-orm>
          Hide
          Lewis John McGibbney added a comment -

          Patch for trunk.
          I added a test driver for Mongodb which uses a nice ASLv2.0 licensed testing project for embedding MongoDB server for our tests.
          https://github.com/flapdoodle-oss/embedmongo.flapdoodle.de

          Right now we need to specify the correct mapping file so that the mongodb tests tie in with the ones from core. This is the next step in getting this patch in to trunk!

          For the time being I've disabled most of the Nutch stuff as we do not wish to be relying upon Nutch within Gora. We can add some of this at a later stage depending on what folks think.

          Please try this patch out and improve where you guys can.

          Great patch BTW.

          Show
          Lewis John McGibbney added a comment - Patch for trunk. I added a test driver for Mongodb which uses a nice ASLv2.0 licensed testing project for embedding MongoDB server for our tests. https://github.com/flapdoodle-oss/embedmongo.flapdoodle.de Right now we need to specify the correct mapping file so that the mongodb tests tie in with the ones from core. This is the next step in getting this patch in to trunk! For the time being I've disabled most of the Nutch stuff as we do not wish to be relying upon Nutch within Gora. We can add some of this at a later stage depending on what folks think. Please try this patch out and improve where you guys can. Great patch BTW.
          Hide
          Damien Raude-Morvan added a comment -

          Use "git format-patch -1 --no-prefix" to export patch from Git.

          Show
          Damien Raude-Morvan added a comment - Use "git format-patch -1 --no-prefix" to export patch from Git.
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Thank you very much Damien! I will do my best to review it as soon as possible (:

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Thank you very much Damien! I will do my best to review it as soon as possible (:
          Hide
          Lewis John McGibbney added a comment -

          Hi Damien, thank you for this patch, it's really starting to look like the business
          Can you please generate a clean patch as follows
          git diff --no-prefix trunk > GORA-199.patch

          This means that we can apply it against SVN where it will hopefully, eventually be committed.
          Thank you v much for the effort We are nearly there with this one.
          I also noted that you have a whole bunch of additional tests which is great, hopefully we can retain many of them and possibly test them against other datastores.

          Show
          Lewis John McGibbney added a comment - Hi Damien, thank you for this patch, it's really starting to look like the business Can you please generate a clean patch as follows git diff --no-prefix trunk > GORA-199 .patch This means that we can apply it against SVN where it will hopefully, eventually be committed. Thank you v much for the effort We are nearly there with this one. I also noted that you have a whole bunch of additional tests which is great, hopefully we can retain many of them and possibly test them against other datastores.
          Hide
          Damien Raude-Morvan added a comment -

          Hi,

          I've posted updated support for MongoDB (see 0001-GORA-199-MongoDB-Support-in-Gora.patch).
          I think i've applied all recommandations (headers, maven build, eclipse code format...).

          Could someone review this in details and provide some feedbacks ?

          Regards,

          Show
          Damien Raude-Morvan added a comment - Hi, I've posted updated support for MongoDB (see 0001- GORA-199 -MongoDB-Support-in-Gora.patch). I think i've applied all recommandations (headers, maven build, eclipse code format...). Could someone review this in details and provide some feedbacks ? Regards,
          Hide
          Damien Raude-Morvan added a comment -

          Support for MongoDB in Gora

          Show
          Damien Raude-Morvan added a comment - Support for MongoDB in Gora
          Hide
          Lewis John McGibbney added a comment -

          OK

          {bq} - We can prepare an updated, Apache licenced, patch
          Please, this merely consists of ensuring that every file contains the ASLv2.0 license header. You will see them on all of the other Java files and will also see them on all of the XML and other properties files you may have within your patch.
          {bq}

          Maybe submit ICLA or CCLA for copyright assignement ?
          If you guys would like to keep working on/with Gora we would really, really like to have you aboard. At Apache, we operate on a meritocratic model where people that contribute earn respect which should be rewarded. Merit is dished out loving by the Project Management Committee :0). Based on becoming a committer, and obtaining write access to the Gora codebase and other things, you would then be asked to file an ICLA or something similar, however this would be based on the PMC holding some kind of VOTE on your contributions to the Gora code.

          {bq} Migrate to Maven instead of Gradle build
          Yes please, this would be awesome and would really help integration. You can pull one of the existing module POM's for an idea about the inheritance model we use within Gora. Versioning should be held within the parent pom.xml. Apart from that, the design is pretty straight forward.
          {bq}

          Apply official ASF code-format (can you give me link to XML configuration for formater ?)
          There is an eclipse code formatter at the link below. You can import this in to Eclipse and use it to format your patches.
          http://svn.apache.org/repos/asf/gora/committers/

          Thanks Lewis

          Show
          Lewis John McGibbney added a comment - OK {bq} - We can prepare an updated, Apache licenced, patch Please, this merely consists of ensuring that every file contains the ASLv2.0 license header. You will see them on all of the other Java files and will also see them on all of the XML and other properties files you may have within your patch. {bq} Maybe submit ICLA or CCLA for copyright assignement ? If you guys would like to keep working on/with Gora we would really, really like to have you aboard. At Apache, we operate on a meritocratic model where people that contribute earn respect which should be rewarded. Merit is dished out loving by the Project Management Committee :0). Based on becoming a committer, and obtaining write access to the Gora codebase and other things, you would then be asked to file an ICLA or something similar, however this would be based on the PMC holding some kind of VOTE on your contributions to the Gora code. {bq} Migrate to Maven instead of Gradle build Yes please, this would be awesome and would really help integration. You can pull one of the existing module POM's for an idea about the inheritance model we use within Gora. Versioning should be held within the parent pom.xml. Apart from that, the design is pretty straight forward. {bq} Apply official ASF code-format (can you give me link to XML configuration for formater ?) There is an eclipse code formatter at the link below. You can import this in to Eclipse and use it to format your patches. http://svn.apache.org/repos/asf/gora/committers/ Thanks Lewis
          Hide
          Damien Raude-Morvan added a comment -

          Right now, our gora-mongodb driver is licenced under same proprietary licence as all our internal code.
          But our aim is to release it to ASF. Here is current plan :

          • We can prepare an updated, Apache licenced, patch
          • Maybe submit ICLA or CCLA for copyright assignement ?
          • Migrate to Maven instead of Gradle build
          • Apply official ASF code-format (can you give me link to XML configuration for formater ?)

          Is that ok for you ?

          Show
          Damien Raude-Morvan added a comment - Right now, our gora-mongodb driver is licenced under same proprietary licence as all our internal code. But our aim is to release it to ASF. Here is current plan : We can prepare an updated, Apache licenced, patch Maybe submit ICLA or CCLA for copyright assignement ? Migrate to Maven instead of Gradle build Apply official ASF code-format (can you give me link to XML configuration for formater ?) Is that ok for you ?
          Hide
          Lewis John McGibbney added a comment -

          .Hi Damien. Nice, really nice. Would really like to test and integrate the patch.
          Can you please comment on the mongo code you are implementing with regards to its license compatability?
          Thank you so much

          Show
          Lewis John McGibbney added a comment - .Hi Damien. Nice, really nice. Would really like to test and integrate the patch. Can you please comment on the mongo code you are implementing with regards to its license compatability? Thank you so much
          Hide
          Damien Raude-Morvan added a comment -

          Hi all,

          I'm working @ Dictanova with Fabien (aka. initial patch submiter) and we have an updated gora-mongodb module with bugfixes and some new features.
          For now, we are using Gora 0.2.1 / Nutch 2.1 but I'm planning to update to Gora 0.3 / Nutch 0.3 in coming weeks.

          I'll try to prepare an updated patch that build against trunk.

          Regards,

          Show
          Damien Raude-Morvan added a comment - Hi all, I'm working @ Dictanova with Fabien (aka. initial patch submiter) and we have an updated gora-mongodb module with bugfixes and some new features. For now, we are using Gora 0.2.1 / Nutch 2.1 but I'm planning to update to Gora 0.3 / Nutch 0.3 in coming weeks. I'll try to prepare an updated patch that build against trunk. Regards,
          Hide
          valera radu added a comment -

          Thank you for the advices,i had hoped what this will be easier to do, as soon as it will work on my machine i will commit changes

          Show
          valera radu added a comment - Thank you for the advices,i had hoped what this will be easier to do, as soon as it will work on my machine i will commit changes
          Hide
          Lewis John McGibbney added a comment -

          We would need to use the java driver for Mongo DB Henry. This is ASL licensed
          https://github.com/mongodb/mongo-java-driver/blob/master/LICENSE.txt
          This does of course cause some overhead for folks wanting to implement this as it means a fundamental shake up of the existing codebase as posted by Fabien. It does however mean that there is an option is anyone wishes to pursue it.

          Show
          Lewis John McGibbney added a comment - We would need to use the java driver for Mongo DB Henry. This is ASL licensed https://github.com/mongodb/mongo-java-driver/blob/master/LICENSE.txt This does of course cause some overhead for folks wanting to implement this as it means a fundamental shake up of the existing codebase as posted by Fabien. It does however mean that there is an option is anyone wishes to pursue it.
          Hide
          Henry Saputra added a comment - - edited

          How do we plan to test this with mongodb server? The server license is GNU AGPL v3.0 which cannot be included in ASF project: http://www.apache.org/licenses/GPL-compatibility.html

          Show
          Henry Saputra added a comment - - edited How do we plan to test this with mongodb server? The server license is GNU AGPL v3.0 which cannot be included in ASF project: http://www.apache.org/licenses/GPL-compatibility.html
          Hide
          Lewis John McGibbney added a comment -

          Hi Valera. There is a bit fo work to be done here to get this working against trunk.
          I would advise you to pull trunk [0], add a module pom.xml (you can base it simply on one of the existing module implementations) then build the module so that is becomes part of the Gora build.
          The current source provided by Fabien includes some Gradle build configuration (which we would probably remove) prior to this being implemented in to the codebase, this is simply because we rely upon Maven for the Gora build lifecycle.
          After that there seems to be quite a bit of Nutch related material, which again we would most likely remove unless there is some value to the testing scenario...
          If you are able to do any of this and submit a patch against trunk then it would be excellent.
          [0] http://svn.apache.org/repos/asf/gora/trunk/

          Show
          Lewis John McGibbney added a comment - Hi Valera. There is a bit fo work to be done here to get this working against trunk. I would advise you to pull trunk [0] , add a module pom.xml (you can base it simply on one of the existing module implementations) then build the module so that is becomes part of the Gora build. The current source provided by Fabien includes some Gradle build configuration (which we would probably remove) prior to this being implemented in to the codebase, this is simply because we rely upon Maven for the Gora build lifecycle. After that there seems to be quite a bit of Nutch related material, which again we would most likely remove unless there is some value to the testing scenario... If you are able to do any of this and submit a patch against trunk then it would be excellent. [0] http://svn.apache.org/repos/asf/gora/trunk/
          Hide
          valera radu added a comment - - edited

          Hi,
          Could someone please provide instructions to apply the patch?

          Show
          valera radu added a comment - - edited Hi, Could someone please provide instructions to apply the patch?
          Hide
          Alfonso Nishikawa added a comment -

          Great job with the patch. Thank you very much for your effort!

          I saw something about "a record in a record". Is it something impossible? Needs serialization? Am I wrong with the idea that MongoDB supports natively nesting?

          I would comment only careless points:

          • Just tell about the tabulators with 2 spaces (I think it is the style in Gora code). It's a pitty since your code is much beauty snif
          • I would keep MongoMappingBuilder out of MongoStore (any reason to have it there? just my opinion)

          This is a great starting point for a full MongoDB support Congrats.

          Show
          Alfonso Nishikawa added a comment - Great job with the patch. Thank you very much for your effort! I saw something about "a record in a record". Is it something impossible? Needs serialization? Am I wrong with the idea that MongoDB supports natively nesting? I would comment only careless points: Just tell about the tabulators with 2 spaces (I think it is the style in Gora code). It's a pitty since your code is much beauty snif I would keep MongoMappingBuilder out of MongoStore (any reason to have it there? just my opinion) This is a great starting point for a full MongoDB support Congrats.
          Hide
          Fabien Poulard added a comment -

          I guess the «Create Patch» functionality doesn't work as I expected. Here is the source code archive anyway.

          Show
          Fabien Poulard added a comment - I guess the «Create Patch» functionality doesn't work as I expected. Here is the source code archive anyway.
          Hide
          Fabien Poulard added a comment -

          Here is the whole implementation of the MongoDB implementation. Keep in mind this is more a POC than a final implementation. The code definitely needs some refactoring (especially the MongoDB Map/Array/Record serialization part). Please review the code.

          Also this code was developped by myself within the company I co-funded. My goal is clearly to contribute it also I know that Apache usually asks for specific paper in order to make such a contribution from a company. How should I proceed ?

          The building process makes use of gradle and need a gradle plugin I developped to automatically compile Gora schemas. You'll find the code of this plugin at github :
          https://github.com/Dictanova/gora-gradle-plugin

          Show
          Fabien Poulard added a comment - Here is the whole implementation of the MongoDB implementation. Keep in mind this is more a POC than a final implementation. The code definitely needs some refactoring (especially the MongoDB Map/Array/Record serialization part). Please review the code. Also this code was developped by myself within the company I co-funded. My goal is clearly to contribute it also I know that Apache usually asks for specific paper in order to make such a contribution from a company. How should I proceed ? The building process makes use of gradle and need a gradle plugin I developped to automatically compile Gora schemas. You'll find the code of this plugin at github : https://github.com/Dictanova/gora-gradle-plugin
          Hide
          Lewis John McGibbney added a comment -

          @Fabien, I totally missed these issues as I have recently moved to batch digests for all of the mailing lists I subscribe to. Please upload your patch here and we can begin to review. I understand from your threads elsewhere that you do not have the tests working with the TestDriver scenario, however this is not a huge problem.
          FYI we currently use Maven for the build lifecycle so maybe we can add that functionality as well.
          Finally, please feel free to explain a bit about your avro-gradle-plugin and how if possible we can use this within Gora. My understanding of Groovy is limited and Rails even less so please be gentle
          Thank you and apologies again as this one seems to have passed slipped right through the net.

          Show
          Lewis John McGibbney added a comment - @Fabien, I totally missed these issues as I have recently moved to batch digests for all of the mailing lists I subscribe to. Please upload your patch here and we can begin to review. I understand from your threads elsewhere that you do not have the tests working with the TestDriver scenario, however this is not a huge problem. FYI we currently use Maven for the build lifecycle so maybe we can add that functionality as well. Finally, please feel free to explain a bit about your avro-gradle-plugin and how if possible we can use this within Gora. My understanding of Groovy is limited and Rails even less so please be gentle Thank you and apologies again as this one seems to have passed slipped right through the net.
          Hide
          Fabien Poulard added a comment -

          Hi Lewis,

          Yes, as I posted on the mailing list, I have a functional implementation running Nutch. My question on the mailing list was : how to share the code ? It definitely need some reviewing as I'm no expert in Gora or Avro (or even MongoDB for that matter). I'm not sure a big patch with all the code would be the best move.

          Show
          Fabien Poulard added a comment - Hi Lewis, Yes, as I posted on the mailing list, I have a functional implementation running Nutch. My question on the mailing list was : how to share the code ? It definitely need some reviewing as I'm no expert in Gora or Avro (or even MongoDB for that matter). I'm not sure a big patch with all the code would be the best move.
          Hide
          Lewis John McGibbney added a comment -

          Hi Fabien, out of curiosity, have you been working on this at all? Is there any code written?

          Show
          Lewis John McGibbney added a comment - Hi Fabien, out of curiosity, have you been working on this at all? Is there any code written?

            People

            • Assignee:
              Damien Raude-Morvan
              Reporter:
              Fabien Poulard
            • Votes:
              4 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development