Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: SDB
    • Labels:

      Description

      Add support for normalized storage in SDB for MonetDB - http://www.monetdb.org/

        Activity

        Hide
        ultan ocarroll added a comment -

        Add support for normalized storage on MonetDB. Only tested so far as SDBConfig creates the schema without error. More work needed....

        Show
        ultan ocarroll added a comment - Add support for normalized storage on MonetDB. Only tested so far as SDBConfig creates the schema without error. More work needed....
        Hide
        ultan ocarroll added a comment -

        Fixed patch - doesnt remove classes that were in fact modified !
        Doesnt contain any Unit Test code - use at your own risk !

        Show
        ultan ocarroll added a comment - Fixed patch - doesnt remove classes that were in fact modified ! Doesnt contain any Unit Test code - use at your own risk !
        Hide
        Paolo Castagna added a comment -

        Thanks.

        As I said on jena-users mailing list (http://markmail.org/message/4vzfo4ytfuijbnrf): a patch without tests is better than no patch at all.
        However, tests are very important and they should be considered part of the patch.

        When you submit a patch also, make sure you flag: "Grant license to ASF for inclusion in ASF works (as per the Apache License)" if you wish to get your patch contributed to the project.
        This is really important, as we cannot accept patches without your consent.

        Show
        Paolo Castagna added a comment - Thanks. As I said on jena-users mailing list ( http://markmail.org/message/4vzfo4ytfuijbnrf): a patch without tests is better than no patch at all. However, tests are very important and they should be considered part of the patch. When you submit a patch also, make sure you flag: "Grant license to ASF for inclusion in ASF works (as per the Apache License)" if you wish to get your patch contributed to the project. This is really important, as we cannot accept patches without your consent.
        Hide
        Andy Seaborne added a comment -

        Thanks very much for the patch.

        Testing for a specific database is mostly scripted; it's not done with JUnit code directly but some code that builds JUnit tests.

        The file testing/store-list.ttl contains the details of the store to be tested. (store-simple-list.ttl is for the layout1 design which is not used for real).

        SDBModelGraphTestSuite contains the configuration of ...sdb.test.model.Test"store"Model.java.

        Show
        Andy Seaborne added a comment - Thanks very much for the patch. Testing for a specific database is mostly scripted; it's not done with JUnit code directly but some code that builds JUnit tests. The file testing/store-list.ttl contains the details of the store to be tested. (store-simple-list.ttl is for the layout1 design which is not used for real). SDBModelGraphTestSuite contains the configuration of ...sdb.test.model.Test"store"Model.java.
        Hide
        ultan ocarroll added a comment -

        Hi Andy,

        Thanks for the info : I suspected as much from my brief glimpse. I'll
        def get around to it, Im interested to see how it fairs.

        Show
        ultan ocarroll added a comment - Hi Andy, Thanks for the info : I suspected as much from my brief glimpse. I'll def get around to it, Im interested to see how it fairs.
        Hide
        ultan ocarroll added a comment -

        BTW - while I'm thinking about it, is there a fixture data set for these
        tests ?

        Show
        ultan ocarroll added a comment - BTW - while I'm thinking about it, is there a fixture data set for these tests ?
        Hide
        Andy Seaborne added a comment -

        The tests need an empty database but one that exists and is formatted (tables exist). (not knowing MonetDB I may be using the wrong language).

        The store-list.ttl names a description file (assembler) usually in StoreDesc directory using relative URIs e.g. <StoreDesc/mysql-hash.ttl>, that is relative to the store-list.ttl file. so testing/StoreDesc/mysql-hash.ttl.

        Show
        Andy Seaborne added a comment - The tests need an empty database but one that exists and is formatted (tables exist). (not knowing MonetDB I may be using the wrong language). The store-list.ttl names a description file (assembler) usually in StoreDesc directory using relative URIs e.g. <StoreDesc/mysql-hash.ttl>, that is relative to the store-list.ttl file. so testing/StoreDesc/mysql-hash.ttl.
        Hide
        ultan ocarroll added a comment -

        Patch needs some more work, as loading failing on temp table creation. Will update this weekend and include the unit tests as well. License to be granted also.

        Show
        ultan ocarroll added a comment - Patch needs some more work, as loading failing on temp table creation. Will update this weekend and include the unit tests as well. License to be granted also.
        Hide
        Damian Steer added a comment -

        Thanks for all your work on this Ultan. I have monetdb installed and have tried your patch. Looking good so far.

        Show
        Damian Steer added a comment - Thanks for all your work on this Ultan. I have monetdb installed and have tried your patch. Looking good so far.
        Hide
        ultan ocarroll added a comment -

        Hi Damian - watch out, danger danger ! I've come across some issues with
        MonetDB temp tables that I'm trying to sort out. Happens during unit
        test, on second group of tests when ensureTempTables is called for a
        second time. Have mailed Monet, (CCd Andy), awaiting a response.

        Show
        ultan ocarroll added a comment - Hi Damian - watch out, danger danger ! I've come across some issues with MonetDB temp tables that I'm trying to sort out. Happens during unit test, on second group of tests when ensureTempTables is called for a second time. Have mailed Monet, (CCd Andy), awaiting a response.
        Show
        ultan ocarroll added a comment - MonetDB question : http://sourceforge.net/mailarchive/forum.php?thread_name=4E9C962D.9070505%40eircom.net&forum_name=monetdb-users
        Hide
        ultan ocarroll added a comment -

        No answers from MonetDB. Either Im doing something too stupid to raise an answer or its a problem no-one is interested in. Saw mention of it in a post from 2009 also. So for now, I expect the only way to get the unit tests to pass is to perform as an admin user. Schema can then be authorised by a non-admin user for loading and general usage, assuming temp tables not (re)used during normal CRUD operations. Will give this a go Thu at the earliest.

        Show
        ultan ocarroll added a comment - No answers from MonetDB. Either Im doing something too stupid to raise an answer or its a problem no-one is interested in. Saw mention of it in a post from 2009 also. So for now, I expect the only way to get the unit tests to pass is to perform as an admin user. Schema can then be authorised by a non-admin user for loading and general usage, assuming temp tables not (re)used during normal CRUD operations. Will give this a go Thu at the earliest.
        Hide
        ultan ocarroll added a comment -

        Having moved over to using an admin user, the unit tests could successfully create temp tables (ordinary users cannot, I have found out, officially), and I got passed a few more tests. However, when the script gets to Optional - 1 it fails, as the query seems to crash the server "SQLException: End of Stream"

        1) Any objections to sharing the hash schema and queries with Monet ? Fabian Groffen will take a look.
        2) Want the code as it currently stands, uploaded ?

        I'm going to move on to Vertica, [1] now that its just gone community edition, and its also based around CStore. I also have access to a Vertica appliance for a short while. Will open an issue here once I've built the code and unit tests this time, first time !

        [1] http://www.vertica.com/community-edition-beta-signup/

        Show
        ultan ocarroll added a comment - Having moved over to using an admin user, the unit tests could successfully create temp tables (ordinary users cannot, I have found out, officially), and I got passed a few more tests. However, when the script gets to Optional - 1 it fails, as the query seems to crash the server "SQLException: End of Stream" 1) Any objections to sharing the hash schema and queries with Monet ? Fabian Groffen will take a look. 2) Want the code as it currently stands, uploaded ? I'm going to move on to Vertica, [1] now that its just gone community edition, and its also based around CStore. I also have access to a Vertica appliance for a short while. Will open an issue here once I've built the code and unit tests this time, first time ! [1] http://www.vertica.com/community-edition-beta-signup/
        Hide
        ultan ocarroll added a comment -

        Have submitted 2 bugs on MonetDB

        connection/server crash : http://bugs.monetdb.org/show_bug.cgi?id=2915
        temp tables: http://bugs.monetdb.org/show_bug.cgi?id=2916

        Show
        ultan ocarroll added a comment - Have submitted 2 bugs on MonetDB connection/server crash : http://bugs.monetdb.org/show_bug.cgi?id=2915 temp tables: http://bugs.monetdb.org/show_bug.cgi?id=2916
        Hide
        ultan ocarroll added a comment -

        Just tried with the Dec 2011 SP2 release of MonetDB and the crash has been resolved and all of the unit tests pass. Once I get my head back into all this (having been away some time) Ill get a patch together.

        Hash test results
        ------------------------
        Test: Term-01
        Test: Term-02 - plain string
        Test: Term-03
        Test: Term-04
        Test: Term-05 - unknown type
        Test: Term-06
        Test: Term-07
        Test: Term-08
        Test: Term-09
        Test: Term-10
        Test: Term-11
        Test: Term-12
        Test: Vars-1
        Test: Vars-2
        Test: Vars-3
        Test: Vars-4
        Test: Vars-5
        Test: Unicode-1
        Test: Unicode-2
        Test: Unicode-3
        Test: Unicode-4
        Test: Unicode-5
        Test: Basic - Empty pattern, all variables
        Test: Basic - Empty pattern -name variable
        Test: Basic - 1 - Single triple pattern
        Test: Basic - 2 - Two triple patterns, common subject pair, object-subject path
        Test: Basic - 3 - Three triple patterns, common subject
        Test: Triple - 1
        Test: Triple - 2
        Test: Triple - 3
        Test: Triple - 4
        Test: Optional - 1
        Test: Optional - 2
        Test: Optional - 3
        Test: Optional - 4
        Test: Coalesce - 1
        Test: Coalesce - 2
        Test: Coalesce - 3
        Test: Partially optimized - 1
        Test: Partially optimized - 2
        Test: Partially optimized - 3
        Test: Partially optimized - 4
        Test: Regex-1
        Test: Regex-2
        Test: Regex-3
        Test: Regex-opt-1
        Test: Regex-opt-2
        Test: Regex-opt-3
        Test: Regex-opt-4
        Test: Equals-1
        Test: Equals-2
        Test: Nested Optionals - 1
        Test: Nested Optionals - 2
        Test: Optional-filter - 1
        Test: Filter-nested - 1
        Test: Filter-nested - 2
        Test: Structure - 1
        Test: Structure - 2
        Test: Structure - 3
        Test: Structure - 4
        Test: Structure - 5
        Test: Structure - 6
        Test: Structure - 10
        Test: Quads-1
        Test: Quads-2
        Test: Quads-2a
        Test: Merge-1
        Test: Merge-2
        Test: Merge-3
        Test: Merge-4
        Test: Merge-5
        Test: Merge-6
        Test: Merge-A
        Test: Merge-B
        Test: Slice 1
        Test: Slice 2
        Test: Slice 3
        Test: Slice 4
        Test: Slice 5
        Test: Slice 6
        Test: Distinct 1
        Test: Distinct 2
        Tests = 82 : Successes = 82 : Errors = 0 : Failures = 0

        Index Test Results
        ----------------------------
        Test: Term-01
        Test: Term-02 - plain string
        Test: Term-03
        Test: Term-04
        Test: Term-05 - unknown type
        Test: Term-06
        Test: Term-07
        Test: Term-08
        Test: Term-09
        Test: Term-10
        Test: Term-11
        Test: Term-12
        Test: Vars-1
        Test: Vars-2
        Test: Vars-3
        Test: Vars-4
        Test: Vars-5
        Test: Unicode-1
        Test: Unicode-2
        Test: Unicode-3
        Test: Unicode-4
        Test: Unicode-5
        Test: Basic - Empty pattern, all variables
        Test: Basic - Empty pattern -name variable
        Test: Basic - 1 - Single triple pattern
        Test: Basic - 2 - Two triple patterns, common subject pair, object-subject path
        Test: Basic - 3 - Three triple patterns, common subject
        Test: Triple - 1
        Test: Triple - 2
        Test: Triple - 3
        Test: Triple - 4
        Test: Optional - 1
        Test: Optional - 2
        Test: Optional - 3
        Test: Optional - 4
        Test: Coalesce - 1
        Test: Coalesce - 2
        Test: Coalesce - 3
        Test: Partially optimized - 1
        Test: Partially optimized - 2
        Test: Partially optimized - 3
        Test: Partially optimized - 4
        Test: Regex-1
        Test: Regex-2
        Test: Regex-3
        Test: Regex-opt-1
        Test: Regex-opt-2
        Test: Regex-opt-3
        Test: Regex-opt-4
        Test: Equals-1
        Test: Equals-2
        Test: Nested Optionals - 1
        Test: Nested Optionals - 2
        Test: Optional-filter - 1
        Test: Filter-nested - 1
        Test: Filter-nested - 2
        Test: Structure - 1
        Test: Structure - 2
        Test: Structure - 3
        Test: Structure - 4
        Test: Structure - 5
        Test: Structure - 6
        Test: Structure - 10
        Test: Quads-1
        Test: Quads-2
        Test: Quads-2a
        Test: Merge-1
        Test: Merge-2
        Test: Merge-3
        Test: Merge-4
        Test: Merge-5
        Test: Merge-6
        Test: Merge-A
        Test: Merge-B
        Test: Slice 1
        Test: Slice 2
        Test: Slice 3
        Test: Slice 4
        Test: Slice 5
        Test: Slice 6
        Test: Distinct 1
        Test: Distinct 2
        Tests = 82 : Successes = 82 : Errors = 0 : Failures = 0

        Show
        ultan ocarroll added a comment - Just tried with the Dec 2011 SP2 release of MonetDB and the crash has been resolved and all of the unit tests pass. Once I get my head back into all this (having been away some time) Ill get a patch together. Hash test results ------------------------ Test: Term-01 Test: Term-02 - plain string Test: Term-03 Test: Term-04 Test: Term-05 - unknown type Test: Term-06 Test: Term-07 Test: Term-08 Test: Term-09 Test: Term-10 Test: Term-11 Test: Term-12 Test: Vars-1 Test: Vars-2 Test: Vars-3 Test: Vars-4 Test: Vars-5 Test: Unicode-1 Test: Unicode-2 Test: Unicode-3 Test: Unicode-4 Test: Unicode-5 Test: Basic - Empty pattern, all variables Test: Basic - Empty pattern -name variable Test: Basic - 1 - Single triple pattern Test: Basic - 2 - Two triple patterns, common subject pair, object-subject path Test: Basic - 3 - Three triple patterns, common subject Test: Triple - 1 Test: Triple - 2 Test: Triple - 3 Test: Triple - 4 Test: Optional - 1 Test: Optional - 2 Test: Optional - 3 Test: Optional - 4 Test: Coalesce - 1 Test: Coalesce - 2 Test: Coalesce - 3 Test: Partially optimized - 1 Test: Partially optimized - 2 Test: Partially optimized - 3 Test: Partially optimized - 4 Test: Regex-1 Test: Regex-2 Test: Regex-3 Test: Regex-opt-1 Test: Regex-opt-2 Test: Regex-opt-3 Test: Regex-opt-4 Test: Equals-1 Test: Equals-2 Test: Nested Optionals - 1 Test: Nested Optionals - 2 Test: Optional-filter - 1 Test: Filter-nested - 1 Test: Filter-nested - 2 Test: Structure - 1 Test: Structure - 2 Test: Structure - 3 Test: Structure - 4 Test: Structure - 5 Test: Structure - 6 Test: Structure - 10 Test: Quads-1 Test: Quads-2 Test: Quads-2a Test: Merge-1 Test: Merge-2 Test: Merge-3 Test: Merge-4 Test: Merge-5 Test: Merge-6 Test: Merge-A Test: Merge-B Test: Slice 1 Test: Slice 2 Test: Slice 3 Test: Slice 4 Test: Slice 5 Test: Slice 6 Test: Distinct 1 Test: Distinct 2 Tests = 82 : Successes = 82 : Errors = 0 : Failures = 0 Index Test Results ---------------------------- Test: Term-01 Test: Term-02 - plain string Test: Term-03 Test: Term-04 Test: Term-05 - unknown type Test: Term-06 Test: Term-07 Test: Term-08 Test: Term-09 Test: Term-10 Test: Term-11 Test: Term-12 Test: Vars-1 Test: Vars-2 Test: Vars-3 Test: Vars-4 Test: Vars-5 Test: Unicode-1 Test: Unicode-2 Test: Unicode-3 Test: Unicode-4 Test: Unicode-5 Test: Basic - Empty pattern, all variables Test: Basic - Empty pattern -name variable Test: Basic - 1 - Single triple pattern Test: Basic - 2 - Two triple patterns, common subject pair, object-subject path Test: Basic - 3 - Three triple patterns, common subject Test: Triple - 1 Test: Triple - 2 Test: Triple - 3 Test: Triple - 4 Test: Optional - 1 Test: Optional - 2 Test: Optional - 3 Test: Optional - 4 Test: Coalesce - 1 Test: Coalesce - 2 Test: Coalesce - 3 Test: Partially optimized - 1 Test: Partially optimized - 2 Test: Partially optimized - 3 Test: Partially optimized - 4 Test: Regex-1 Test: Regex-2 Test: Regex-3 Test: Regex-opt-1 Test: Regex-opt-2 Test: Regex-opt-3 Test: Regex-opt-4 Test: Equals-1 Test: Equals-2 Test: Nested Optionals - 1 Test: Nested Optionals - 2 Test: Optional-filter - 1 Test: Filter-nested - 1 Test: Filter-nested - 2 Test: Structure - 1 Test: Structure - 2 Test: Structure - 3 Test: Structure - 4 Test: Structure - 5 Test: Structure - 6 Test: Structure - 10 Test: Quads-1 Test: Quads-2 Test: Quads-2a Test: Merge-1 Test: Merge-2 Test: Merge-3 Test: Merge-4 Test: Merge-5 Test: Merge-6 Test: Merge-A Test: Merge-B Test: Slice 1 Test: Slice 2 Test: Slice 3 Test: Slice 4 Test: Slice 5 Test: Slice 6 Test: Distinct 1 Test: Distinct 2 Tests = 82 : Successes = 82 : Errors = 0 : Failures = 0
        Hide
        ultan ocarroll added a comment -

        Introduces dependency on MonetDB jdbc client jar : http://dev.monetdb.org/downloads/Java/Dec2011-SP2/jdbcclient.jar

        Show
        ultan ocarroll added a comment - Introduces dependency on MonetDB jdbc client jar : http://dev.monetdb.org/downloads/Java/Dec2011-SP2/jdbcclient.jar
        Hide
        ultan ocarroll added a comment -

        Updated patch file. Created in Eclipse, manually removed .classpath index entry in patch file.
        Applied patch to secondary check out.

        Show
        ultan ocarroll added a comment - Updated patch file. Created in Eclipse, manually removed .classpath index entry in patch file. Applied patch to secondary check out.

          People

          • Assignee:
            Unassigned
            Reporter:
            ultan ocarroll
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development