JDO
  1. JDO
  2. JDO-652

Provision of a typesafe refactor-friendly query capability for JDOQL

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: JDO 3.2
    • Component/s: api, specification, tck
    • Labels:
      None

      Description

      There are various querying capabilities of this type around. JPA2 has its Criteria query API. Third party solutions like QueryDSL also exist, in its case providing a JDOQL implementation (as well as JPQL, and HQL). We should seriously consider introducing something along these lines in the JDO2.4 timeframe.

      There is a comparison of JPA Criteria with QueryDSL over at
      http://source.mysema.com/forum/mvnforum/viewthread_thread,49

      1. typesafe.patch
        65 kB
        Andy Jefferson
      2. typesafe_manifest.patch
        0.5 kB
        Andy Jefferson

        Issue Links

          Activity

          Hide
          Timo Westkämper added a comment -

          We at Mysema are interested in a standardization of the dynamic query metamodel of Querydsl. Providing something similar in JDO is a good first step.

          The reference documentation of Querydsl describes the JDOQL usage :
          http://source.mysema.com/static/querydsl/1.5.0/reference/html/ch02.html#jdoql_integration

          Show
          Timo Westkämper added a comment - We at Mysema are interested in a standardization of the dynamic query metamodel of Querydsl. Providing something similar in JDO is a good first step. The reference documentation of Querydsl describes the JDOQL usage : http://source.mysema.com/static/querydsl/1.5.0/reference/html/ch02.html#jdoql_integration
          Show
          Andy Jefferson added a comment - If anyone is interested in contributing ideas etc, please go to http://datanucleus.blogspot.com/2010/06/jdo-typesafe-refactorable-queries.html and http://blog.mysema.com/2010/06/typesafe-queries-for-datanucleus.html
          Hide
          Matthew T. Adams added a comment -

          If some form of the following two Sun RFEs were actually implemented, we could have a fully typesafe query system:

          http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5043025
          http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6915224

          Perhaps we should start getting people to vote for them.

          Show
          Matthew T. Adams added a comment - If some form of the following two Sun RFEs were actually implemented, we could have a fully typesafe query system: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5043025 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6915224 Perhaps we should start getting people to vote for them.
          Show
          Andy Jefferson added a comment - See also http://www.datanucleus.org/servlet/jira/browse/NUCCORE-545
          Hide
          Matthew T. Adams added a comment -

          While I like the current discussion on the fluent query api and think that we should continue to pursue this idea, I'm afraid that its post facto nature might be a hindrance to its adoption. That is, a developer has to compile (and enhance, I suspect) his PC objects in order to get the fluent query objects; he can't develop persistent objects and use the fluent query ones in the same module if they require compilation and enhancement. It's kind of a chicken-and-egg problem, and the only solution I see is to develop the persistent objects in one module and use fluent query objects in a separate one that depends on the module with the persistent objects.

          I think that developers will want to have query objects and methods available to them in the same module; the only way I see that happening is to leverage technologies like Spring Roo, which limits the developers choice of IDE.

          Again, I support the fluent query api as discussed in this issue, but I think we should also create a conventional query api that bites the bullet and, unfortunately, uses strings for field names that has overloads that take java.lang.reflect.Field objects, which opens the door in the event that typesafe means of reflective access to fields as requested in Sun bugs 5043025 and 6915224 becomes a reality.

          It did occur to me while writing this that the enhancer could be leveraged somehow to optionally examine the field names given in the conventional query api to do field name existence checking. While it's not typesafe at java compile time, it could at least be typesafe at enhancement time, which most people do one right after the other.

          Thoughts?

          Show
          Matthew T. Adams added a comment - While I like the current discussion on the fluent query api and think that we should continue to pursue this idea, I'm afraid that its post facto nature might be a hindrance to its adoption. That is, a developer has to compile (and enhance, I suspect) his PC objects in order to get the fluent query objects; he can't develop persistent objects and use the fluent query ones in the same module if they require compilation and enhancement. It's kind of a chicken-and-egg problem, and the only solution I see is to develop the persistent objects in one module and use fluent query objects in a separate one that depends on the module with the persistent objects. I think that developers will want to have query objects and methods available to them in the same module; the only way I see that happening is to leverage technologies like Spring Roo, which limits the developers choice of IDE. Again, I support the fluent query api as discussed in this issue, but I think we should also create a conventional query api that bites the bullet and, unfortunately, uses strings for field names that has overloads that take java.lang.reflect.Field objects, which opens the door in the event that typesafe means of reflective access to fields as requested in Sun bugs 5043025 and 6915224 becomes a reality. It did occur to me while writing this that the enhancer could be leveraged somehow to optionally examine the field names given in the conventional query api to do field name existence checking. While it's not typesafe at java compile time, it could at least be typesafe at enhancement time, which most people do one right after the other. Thoughts?
          Hide
          Andy Jefferson added a comment -

          For the QueryDSL approach : a user would not have to compile their model classes before creating the query classes. The compile triggers a visit to the enhancer processor, which creates the query class source files, and then model + query classes are compiled together. A bytecode enhancer is not involved. at. all. The user can develop their app in an IDE as they normally would ... model is designed ... and its autocompile triggers creation of query classes ... and then both can be used in the persistence code.

          Show
          Andy Jefferson added a comment - For the QueryDSL approach : a user would not have to compile their model classes before creating the query classes. The compile triggers a visit to the enhancer processor, which creates the query class source files, and then model + query classes are compiled together. A bytecode enhancer is not involved. at. all. The user can develop their app in an IDE as they normally would ... model is designed ... and its autocompile triggers creation of query classes ... and then both can be used in the persistence code.
          Hide
          Timo Westkämper added a comment -

          Matthew, could you give some code examples how a fluent Query API using the not implemented Java features could look like?

          If I understand correctly it provides only typesafe shortcuts to direct fields and methods. So you end up with a static metamodel like in JPA 2 Criteria. Using a static metamodel for query construction is quite verbose as you will know if you have used JPA 2 Criteria.

          Show
          Timo Westkämper added a comment - Matthew, could you give some code examples how a fluent Query API using the not implemented Java features could look like? If I understand correctly it provides only typesafe shortcuts to direct fields and methods. So you end up with a static metamodel like in JPA 2 Criteria. Using a static metamodel for query construction is quite verbose as you will know if you have used JPA 2 Criteria.
          Hide
          Matthew T. Adams added a comment -

          @Andy: Based on your response, I must be misunderstanding the QueryDSL approach. My understanding is that a prerequisite to the creation of the QueryDSL objects is your object model. If the two can be developed in a conventional IDE, please show or explain to me how. A conventional, field name-based query api would not require the objects to exist beforehand, unless it used the as-yet-nonextant language features that we'd like to have.

          Show
          Matthew T. Adams added a comment - @Andy: Based on your response, I must be misunderstanding the QueryDSL approach. My understanding is that a prerequisite to the creation of the QueryDSL objects is your object model. If the two can be developed in a conventional IDE, please show or explain to me how. A conventional, field name-based query api would not require the objects to exist beforehand, unless it used the as-yet-nonextant language features that we'd like to have.
          Hide
          Matthew T. Adams added a comment -

          @Timo: The code examples that I would propose look similar to what they do in JPA2 criteria queries, except the replacement of field name strings with the newly proposed syntax and appropriate modifications in the spirit of JDO collection navigation vs. JPA's "join" syntax. Depending on the design of JDO's query api, it may or may not be verbose.

          Show
          Matthew T. Adams added a comment - @Timo: The code examples that I would propose look similar to what they do in JPA2 criteria queries, except the replacement of field name strings with the newly proposed syntax and appropriate modifications in the spirit of JDO collection navigation vs. JPA's "join" syntax. Depending on the design of JDO's query api, it may or may not be verbose.
          Hide
          Andy Jefferson added a comment -

          @Matthew, you open Eclipse (and set it up to have the annotation processor enabled). You write a model class ("Person"). When pressing "Save" Eclipse will attempt to compile it. Prior to compilation all files to be compiled that have the required annotations (@PersistenceCapable) are passed to the annotation processor(s). The annotation processor generates the equivalent query source code (QPerson). The compiler then compiles all classes (model + query).

          When compilation has finished the user starts to write a persistence class. They can reference Person and QPerson since both are compiled classes.

          Show
          Andy Jefferson added a comment - @Matthew, you open Eclipse (and set it up to have the annotation processor enabled). You write a model class ("Person"). When pressing "Save" Eclipse will attempt to compile it. Prior to compilation all files to be compiled that have the required annotations (@PersistenceCapable) are passed to the annotation processor(s). The annotation processor generates the equivalent query source code (QPerson). The compiler then compiles all classes (model + query). When compilation has finished the user starts to write a persistence class. They can reference Person and QPerson since both are compiled classes.
          Hide
          Matthew T. Adams added a comment -

          @Andy, would this require the use of javac 1.6.x in order to work without the user having to configure anything by default? I don't mind that it would, I just want to be clear. How could this work in a 1.5 environment?

          Show
          Matthew T. Adams added a comment - @Andy, would this require the use of javac 1.6.x in order to work without the user having to configure anything by default? I don't mind that it would, I just want to be clear. How could this work in a 1.5 environment?
          Hide
          Andy Jefferson added a comment -

          @Matthew, annotation processing with 1.6 will work out of the box. Use with 1.5 will require use of "APT" as per http://download.oracle.com/docs/cd/E17476_01/javase/1.5.0/docs/guide/apt/index.html

          Show
          Andy Jefferson added a comment - @Matthew, annotation processing with 1.6 will work out of the box. Use with 1.5 will require use of "APT" as per http://download.oracle.com/docs/cd/E17476_01/javase/1.5.0/docs/guide/apt/index.html
          Hide
          Matthew T. Adams added a comment -

          Ok, then. I don't have an objection to requiring that folks use 1.6 in their dev environment if they want the benefit of minimal configuration to get query classes. We can still support 1.5 if the users want to configure APT.

          Show
          Matthew T. Adams added a comment - Ok, then. I don't have an objection to requiring that folks use 1.6 in their dev environment if they want the benefit of minimal configuration to get query classes. We can still support 1.5 if the users want to configure APT.
          Hide
          Andy Jefferson added a comment - - edited

          Re: developing model and query objects, if someone wants to give it a try they can put
          "datanucleus-jdo-query.jar" (available in the DN nightly repo) in the Eclipse annotation processor jar list, enable JDK1.6 compliance, and also have "datanucleus-core.jar" in the CLASSPATH. They can then create a model class (e.g Product.java) and start writing persistence code that refers to QProduct, such as
          QProduct qp = QProduct.candidate;
          and start making references to "qp" and its various methods ... see http://datanucleus.blogspot.com/2010/07/jdo-typesafe-refactorable-queries.html for the expressions. This is the basis for the various clauses of the proposed query API

          Note this is not currently a full working prototype, it just allows creation of the dynamic query objects and can demonstrate the access of fields, use of methods and relations etc. An example of the API as currently in DataNucleus SVN

          Query<Product> q = pm.newTypesafeQuery(Product.class);
          QProduct cand = QProduct.candidate;
          q.filter(cand.value.lt(40.00).and(cand.name.startsWith("Wal"))).orderBy(cand.name.asc());
          List<Product> results = q.executeList();

          The "Query" object here is a different class to the javax.jdo.Query, could be renamed TypesafeQuery for example. The user can also type "q.toString()" to see the single-string equivalent.

          Show
          Andy Jefferson added a comment - - edited Re: developing model and query objects, if someone wants to give it a try they can put "datanucleus-jdo-query.jar" (available in the DN nightly repo) in the Eclipse annotation processor jar list, enable JDK1.6 compliance, and also have "datanucleus-core.jar" in the CLASSPATH. They can then create a model class (e.g Product.java) and start writing persistence code that refers to QProduct, such as QProduct qp = QProduct.candidate; and start making references to "qp" and its various methods ... see http://datanucleus.blogspot.com/2010/07/jdo-typesafe-refactorable-queries.html for the expressions. This is the basis for the various clauses of the proposed query API Note this is not currently a full working prototype, it just allows creation of the dynamic query objects and can demonstrate the access of fields, use of methods and relations etc. An example of the API as currently in DataNucleus SVN Query<Product> q = pm.newTypesafeQuery(Product.class); QProduct cand = QProduct.candidate; q.filter(cand.value.lt(40.00).and(cand.name.startsWith("Wal"))).orderBy(cand.name.asc()); List<Product> results = q.executeList(); The "Query" object here is a different class to the javax.jdo.Query, could be renamed TypesafeQuery for example. The user can also type "q.toString()" to see the single-string equivalent.
          Hide
          Andy Jefferson added a comment -

          Some examples of what you can do with DataNucleus prototype API
          http://datanucleus.blogspot.com/2010/11/jdo-typesafe-queries-part-3-examples.html

          Show
          Andy Jefferson added a comment - Some examples of what you can do with DataNucleus prototype API http://datanucleus.blogspot.com/2010/11/jdo-typesafe-queries-part-3-examples.html
          Hide
          Andy Jefferson added a comment -

          Javadocs for what is proposed to be "javax.jdo.query" are at
          http://www.datanucleus.org/javadocs/core/2.2/org/datanucleus/query/typesafe/package-summary.html

          DN SVN trunk (and soon to be 2.2 M3) now has a functioning JDOQL "typesafe" query API.

          Amongst the areas to discuss
          1. Naming convention for generated "Query" classes. Currently prefixed by "Q" in this prototype
          2. Way of obtaining candidate, parameter, variable, subquery. If the user wants to access a field then they need to cast to the expression type. See the blog entry for details. Ideally would like to avoid casting
          3. Can we extend it to provide something for JDOQL and JPQL. That is, make use of the "language" argument that JDO typically accepts on queries (not implemented in the prototype) and have TypesafeJDOQLQuery, TypesafeJPQLQuery (for example).

          Show
          Andy Jefferson added a comment - Javadocs for what is proposed to be "javax.jdo.query" are at http://www.datanucleus.org/javadocs/core/2.2/org/datanucleus/query/typesafe/package-summary.html DN SVN trunk (and soon to be 2.2 M3) now has a functioning JDOQL "typesafe" query API. Amongst the areas to discuss 1. Naming convention for generated "Query" classes. Currently prefixed by "Q" in this prototype 2. Way of obtaining candidate, parameter, variable, subquery. If the user wants to access a field then they need to cast to the expression type. See the blog entry for details. Ideally would like to avoid casting 3. Can we extend it to provide something for JDOQL and JPQL. That is, make use of the "language" argument that JDO typically accepts on queries (not implemented in the prototype) and have TypesafeJDOQLQuery, TypesafeJPQLQuery (for example).
          Hide
          Matthew T. Adams added a comment -

          Andy wrote:
          1. Naming convention for generated "Query" classes. Currently prefixed by "Q" in this prototype
          2. Way of obtaining candidate, parameter, variable, subquery. If the user wants to access a field then they need to cast to the expression type. See the blog entry for details. Ideally would like to avoid casting


          RE 1: My personal recommendation for the the "Query" classes is to append the word Query to end of the type. Instead of QProduct, it would be ProductQuery.

          RE 2: I've been following your blog entries on this development work, and I agree on avoiding casting. Would it be possible to provide a method in the Q class that the user can call instead of casting? Perhaps a getField() method in the right place that returns a java.lang.reflect.Field. I'm not sure where the right place is yet, but in that method, the cast can take place and at least the user wouldn't have to do any casting. Thoughts?

          Show
          Matthew T. Adams added a comment - Andy wrote: 1. Naming convention for generated "Query" classes. Currently prefixed by "Q" in this prototype 2. Way of obtaining candidate, parameter, variable, subquery. If the user wants to access a field then they need to cast to the expression type. See the blog entry for details. Ideally would like to avoid casting RE 1: My personal recommendation for the the "Query" classes is to append the word Query to end of the type. Instead of QProduct, it would be ProductQuery. RE 2: I've been following your blog entries on this development work, and I agree on avoiding casting. Would it be possible to provide a method in the Q class that the user can call instead of casting? Perhaps a getField() method in the right place that returns a java.lang.reflect.Field. I'm not sure where the right place is yet, but in that method, the cast can take place and at least the user wouldn't have to do any casting. Thoughts?
          Hide
          Andy Jefferson added a comment -

          Thx for your feedback Matthew.
          I actually updated DN SVN this morning with some changes for casting, and also the most recent blog entry examples to match. The casting isn't there now (at least on the examples I've worked through). The main things on this change are

          • use static method on Query class to get the candidate.
          • use static method on Query class to get variable of that type
          • use xxxParameter methods on TypesafeQuery to get the XXXExpression for the parameter (e.g stringParameter(paramName), doubleParameter(paramName).
          • multiple select() methods on TypesafeSubquery so we can get the correct XXXExpression there too.
            Step in the right direction at least
          Show
          Andy Jefferson added a comment - Thx for your feedback Matthew. I actually updated DN SVN this morning with some changes for casting, and also the most recent blog entry examples to match. The casting isn't there now (at least on the examples I've worked through). The main things on this change are use static method on Query class to get the candidate. use static method on Query class to get variable of that type use xxxParameter methods on TypesafeQuery to get the XXXExpression for the parameter (e.g stringParameter(paramName), doubleParameter(paramName). multiple select() methods on TypesafeSubquery so we can get the correct XXXExpression there too. Step in the right direction at least
          Hide
          Andy Jefferson added a comment -

          Blog entry comparing JDO "Typesafe" against JPA "Criteria"
          http://datanucleus.blogspot.com/2010/11/jdo-typesafe-vs-jpa-criteria.html

          Show
          Andy Jefferson added a comment - Blog entry comparing JDO "Typesafe" against JPA "Criteria" http://datanucleus.blogspot.com/2010/11/jdo-typesafe-vs-jpa-criteria.html
          Hide
          Matthew T. Adams added a comment -

          I admit to not having followed the design discussions too closely, but there was one line that tripped me up when reading your blog. It was the last example, that of a subselect:

          /* 1 */ TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
          /* 2 */ QProduct cand = QProduct.candidate();
          /* 3 */ TypesafeSubquery tqsub = tq.subquery(Product.class, "q");
          /* 4 */ QProduct candsub = QProduct.candidate("q");
          /* 5 */ List<Product> results = tq.filter(cand.value.lt(tqsub.select(candsub.value.avg()))).executeList();

          On line 3, I expected to see TypesafeSubquery as a generic declaration, TypesafeSubquery<Product>. I realize that that since it's coming from its superselect instance (of type TypesafeQuery<Product>), the subselect can know the type of the candidate class of its superselect.

          Question 1: Why is the declaration not "TypesafeSubquery<Product>"?

          Question 2: If the subselect queries the same candidate class as the superselect, could we not provide an overload for TypesafeQuery<T>'s subquery method that has the signature "TypesafeSubquery<T> subquery(string name)"?

          Show
          Matthew T. Adams added a comment - I admit to not having followed the design discussions too closely, but there was one line that tripped me up when reading your blog. It was the last example, that of a subselect: /* 1 */ TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class); /* 2 */ QProduct cand = QProduct.candidate(); /* 3 */ TypesafeSubquery tqsub = tq.subquery(Product.class, "q"); /* 4 */ QProduct candsub = QProduct.candidate("q"); /* 5 */ List<Product> results = tq.filter(cand.value.lt(tqsub.select(candsub.value.avg()))).executeList(); On line 3, I expected to see TypesafeSubquery as a generic declaration, TypesafeSubquery<Product>. I realize that that since it's coming from its superselect instance (of type TypesafeQuery<Product>), the subselect can know the type of the candidate class of its superselect. Question 1: Why is the declaration not "TypesafeSubquery<Product>"? Question 2: If the subselect queries the same candidate class as the superselect, could we not provide an overload for TypesafeQuery<T>'s subquery method that has the signature "TypesafeSubquery<T> subquery(string name)"?
          Hide
          Matthew T. Adams added a comment -

          Small name change suggestion. Instead of the term "TypesafeQuery", how about "TypedQuery"? The fact that it's type-safe is implicit by virtue of its being typed. Saying that it's typed is sufficient.

          Show
          Matthew T. Adams added a comment - Small name change suggestion. Instead of the term "TypesafeQuery", how about "TypedQuery"? The fact that it's type-safe is implicit by virtue of its being typed. Saying that it's typed is sufficient.
          Hide
          Andy Jefferson added a comment -

          Re: 1. subselect declared type. Well the interface is written as <T> so I could have added <Product> in the declaration but it adds nothing in this example since we don't use that info here.
          Re: 2. extra subquery(String) method. Yes certainly.
          Re: naming. I saw that JPA called theirs that, so decided to distance myself :-P

          Show
          Andy Jefferson added a comment - Re: 1. subselect declared type. Well the interface is written as <T> so I could have added <Product> in the declaration but it adds nothing in this example since we don't use that info here. Re: 2. extra subquery(String) method. Yes certainly. Re: naming. I saw that JPA called theirs that, so decided to distance myself :-P
          Hide
          Andy Jefferson added a comment -

          Other thing to consider : what type of access do we want to persistable members (field/properties) ? There are two options, clearly

          1. fields - "candidate.myField" - the only problem with this is that they have to be initialised in the "Query" class, and if we have recursion then this just recurses in the constructor of the query class. So we'd need a way of delimiting the depth of the possible querying so we don't get stack overflows. DN 2.2 M3 imposes a depth of 5.
          e.g tq.orderBy(cand.name.asc());

          2. properties - "candidate.myField()" - this has no such drawback. DN 2.2 M3 allows this if you specify the javac argument "-AqueryMode=PROPERTY".
          e.g tq.orderBy(cand.name().asc());

          Recommend the latter option, but allow implementations to provide both if desired

          Show
          Andy Jefferson added a comment - Other thing to consider : what type of access do we want to persistable members (field/properties) ? There are two options, clearly 1. fields - "candidate.myField" - the only problem with this is that they have to be initialised in the "Query" class, and if we have recursion then this just recurses in the constructor of the query class. So we'd need a way of delimiting the depth of the possible querying so we don't get stack overflows. DN 2.2 M3 imposes a depth of 5. e.g tq.orderBy(cand.name.asc()); 2. properties - "candidate.myField()" - this has no such drawback. DN 2.2 M3 allows this if you specify the javac argument "-AqueryMode=PROPERTY". e.g tq.orderBy(cand.name().asc()); Recommend the latter option, but allow implementations to provide both if desired
          Hide
          Andy Jefferson added a comment -

          Proposed javax.jdo.query API interfaces. All present in latest release of DataNucleus 2.2M3

          Show
          Andy Jefferson added a comment - Proposed javax.jdo.query API interfaces. All present in latest release of DataNucleus 2.2M3
          Hide
          Andy Jefferson added a comment -

          Any comments ? Everyone ok with this being checked in as-is ?

          Show
          Andy Jefferson added a comment - Any comments ? Everyone ok with this being checked in as-is ?
          Hide
          Andy Jefferson added a comment -

          Update to MANIFEST so we export the javax.jdo.query package

          Show
          Andy Jefferson added a comment - Update to MANIFEST so we export the javax.jdo.query package
          Hide
          Matthew T. Adams added a comment -

          After discussing on the conf call Fri Apr 9, we considered the separation of the query definition from the persistence manager connected to which the query definition is executed. It became evident that there is a separation of concerns between the definition or criteria of the query and the PM against which it should execute.

          Instead of a TypesafeQuery<T>, perhaps the class could have a name that reflects the fact that it's a query definition, like TypesafeQueryDefinition<T>, TypesafeQueryCriteria<T>, or, considering that we're in Java and type safety is inherent, I'd move to drop the prefix "Typesafe" from the name and go with just something like QueryCritieria<T>. The currently proposed "Q" classes could be called XxxCriteria ("TeamCriteria" in this example).

          It should also be abled to be obtained by the PMF or the PM. New PMF methods:

          // proposed new methods on PMF
          1. <T> QueryCriteria<T> newQueryCriteria(Class<T> clazz);
          2. Various overloads to execute the given QueryCriteria using the PM given by PMF.getPersistenceManagerProxy()

          New methods could be added to PM to allow for the execution of the query using that PM:

          // methods on PM
          1. <T> QueryCriteria<T> newQueryCriteria(Class<T> clazz);
          2. Various overloads to execute the given QueryCriteria using this PM.

          I see two options for methods that execute these queries. First, only put the execution methods (execute, update, delete) on the PM to keep the number of overloads down, or second, provide overloaded execution methods (execute, update, delete) on the QueryCriteria classes, one set of which takes a PM and delegates to its appropriate execution method, and another set of which does not take a PM and delegates to the PM given by PMF.getPersistenceManagerProxy(). The choice of which option comes down to how many overloaded methods there would be.

          Additionally, if such overloaded execution methods are provided on the QueryCriteria<T> class, should the ones taking no PM execute against the PM returned from PM.getPersistenceManagerProxy() or, if the QueryCriteria was obtained from a PM, the PM from which it came?

          Show
          Matthew T. Adams added a comment - After discussing on the conf call Fri Apr 9, we considered the separation of the query definition from the persistence manager connected to which the query definition is executed. It became evident that there is a separation of concerns between the definition or criteria of the query and the PM against which it should execute. Instead of a TypesafeQuery<T>, perhaps the class could have a name that reflects the fact that it's a query definition, like TypesafeQueryDefinition<T>, TypesafeQueryCriteria<T>, or, considering that we're in Java and type safety is inherent, I'd move to drop the prefix "Typesafe" from the name and go with just something like QueryCritieria<T>. The currently proposed "Q" classes could be called XxxCriteria ("TeamCriteria" in this example). It should also be abled to be obtained by the PMF or the PM. New PMF methods: // proposed new methods on PMF 1. <T> QueryCriteria<T> newQueryCriteria(Class<T> clazz); 2. Various overloads to execute the given QueryCriteria using the PM given by PMF.getPersistenceManagerProxy() New methods could be added to PM to allow for the execution of the query using that PM: // methods on PM 1. <T> QueryCriteria<T> newQueryCriteria(Class<T> clazz); 2. Various overloads to execute the given QueryCriteria using this PM. I see two options for methods that execute these queries. First, only put the execution methods (execute, update, delete) on the PM to keep the number of overloads down, or second, provide overloaded execution methods (execute, update, delete) on the QueryCriteria classes, one set of which takes a PM and delegates to its appropriate execution method, and another set of which does not take a PM and delegates to the PM given by PMF.getPersistenceManagerProxy(). The choice of which option comes down to how many overloaded methods there would be. Additionally, if such overloaded execution methods are provided on the QueryCriteria<T> class, should the ones taking no PM execute against the PM returned from PM.getPersistenceManagerProxy() or, if the QueryCriteria was obtained from a PM, the PM from which it came?
          Hide
          Andy Jefferson added a comment -

          Re: Adding methods to PMF to define the query
          No problem with that (as mentioned before on the jdo-dev list some time ago).

          Re: Calling things XXXCriteria
          -1. This is a query, just like one formed from a single-string, or declarative API, in a form that can be executed. There is nothing in the dictionary definition of "criteria" that means it is somehow more suitable for this type of query. The difference to a single-string query, or a query generated by the declarative API is that is typesafe and refactorable. All queries (whether single-string,declarative, or via this API) have "criteria". Just because some other API calls its API "Criteria" is not a reason in itself, and could be argued a reason not to, to avoid possible confusion.

          Re: putting methods on the PM
          Are you planning on doing that for a normal Query too? because they have no inherent need to be created from a PM either. Consistency please.

          Show
          Andy Jefferson added a comment - Re: Adding methods to PMF to define the query No problem with that (as mentioned before on the jdo-dev list some time ago). Re: Calling things XXXCriteria -1. This is a query, just like one formed from a single-string, or declarative API, in a form that can be executed. There is nothing in the dictionary definition of "criteria" that means it is somehow more suitable for this type of query. The difference to a single-string query, or a query generated by the declarative API is that is typesafe and refactorable. All queries (whether single-string,declarative, or via this API) have "criteria". Just because some other API calls its API "Criteria" is not a reason in itself, and could be argued a reason not to, to avoid possible confusion. Re: putting methods on the PM Are you planning on doing that for a normal Query too? because they have no inherent need to be created from a PM either. Consistency please.
          Hide
          Matthew T. Adams added a comment -

          Before I forget, there was another issue about how bulk update & delete affect any callbacks that PC classes or listeners may implement. We discussed on the call that it would be beneficial for the developer to make a conscious decision to decide whether or not instance or listener callbacks should be invoked by requiring all of the bulk update(..) and delete(..) methods to take a boolean indicating whether callbacks should be invoked or not.

          This would also be a new feature for the existing deletion by query feature, since the specification currently requires that predeletion callbacks be invoked.

          In this way, the developer is required to tell JDO that he knows about the callbacks and either wants or doesn't want to subvert the object model.

          Show
          Matthew T. Adams added a comment - Before I forget, there was another issue about how bulk update & delete affect any callbacks that PC classes or listeners may implement. We discussed on the call that it would be beneficial for the developer to make a conscious decision to decide whether or not instance or listener callbacks should be invoked by requiring all of the bulk update(..) and delete(..) methods to take a boolean indicating whether callbacks should be invoked or not. This would also be a new feature for the existing deletion by query feature, since the specification currently requires that predeletion callbacks be invoked. In this way, the developer is required to tell JDO that he knows about the callbacks and either wants or doesn't want to subvert the object model.
          Hide
          Michael Bouschen added a comment -

          I looked at the JDOQL "typesafe" query API.

          The expression syntax is looking good, but I have some questions about the handling of query parameters and specifying the query result. My concern is that the typesafe query API is using a very different aproach compared to the regular JDO queries.

          • Parameter handling:
            The typesafe query API uses a method setParameter to set the parameter value, meaning the parameters values are part of the state of the TypesafeQuery instance. With JDO queries parameter values are passed as arguments to the Query.execute call and not stored in the Query state.
          • Query result specification:
            The typesafe query API used a special execute method executeResultList to specify the query result, e.g. tq.executeResultList(String.class, false, cand.lastname). At first glance this looks like an execute call taking parameter values, but it is not. The reason might be, that this aproach allows tempating on the query result. An alternative would be specifying the query result type as a tempate argument of the TypesafeQuery instance, in addition to the candidate class.

          What do you think?

          Show
          Michael Bouschen added a comment - I looked at the JDOQL "typesafe" query API. The expression syntax is looking good, but I have some questions about the handling of query parameters and specifying the query result. My concern is that the typesafe query API is using a very different aproach compared to the regular JDO queries. Parameter handling: The typesafe query API uses a method setParameter to set the parameter value, meaning the parameters values are part of the state of the TypesafeQuery instance. With JDO queries parameter values are passed as arguments to the Query.execute call and not stored in the Query state. Query result specification: The typesafe query API used a special execute method executeResultList to specify the query result, e.g. tq.executeResultList(String.class, false, cand.lastname). At first glance this looks like an execute call taking parameter values, but it is not. The reason might be, that this aproach allows tempating on the query result. An alternative would be specifying the query result type as a tempate argument of the TypesafeQuery instance, in addition to the candidate class. What do you think?
          Hide
          Andy Jefferson added a comment -

          Re: Query result specification
          There are two approaches ... constructor argument, or execute argument. If you do as constructor argument then you can't change the result class thereafter. With "normal" API you can change the result class/result definition after construction (i.e reuse Query objects). Consequently I don't see a sensible alternative to what is already there.

          Re: Parameter specification
          Why is it so important to pass the parameter values in to execute() ? Obviously we could add an argument (Map?, array?) to executeXXX() but I don't see an alternative to passing the result in also. What difference does it make if a parameter is "state"?

          Show
          Andy Jefferson added a comment - Re: Query result specification There are two approaches ... constructor argument, or execute argument. If you do as constructor argument then you can't change the result class thereafter. With "normal" API you can change the result class/result definition after construction (i.e reuse Query objects). Consequently I don't see a sensible alternative to what is already there. Re: Parameter specification Why is it so important to pass the parameter values in to execute() ? Obviously we could add an argument (Map?, array?) to executeXXX() but I don't see an alternative to passing the result in also. What difference does it make if a parameter is "state"?
          Hide
          Craig L Russell added a comment -

          I like the direction of this api. My comments here might have been expressed by others; I don't claim ownership of them.

          I like the naming of the Q classes generated during pre-processing. I consider the requirement to have the persistent classes annotated to be a distraction though.

          I think we need to consider reuse of the query artifacts by different persistence managers, and also the possibility that the query definition might be stored as a named query. This leads to needing a factory defined on the PMF, not just the PM. The factory might not be needed on the PM depending on other considerations (discussed below).

          The same query definition could be used by different projections. We could make this explicit in the api by adding a method to the query definition that returns a different interface like QueryProjection<R> where R is the class of the result. And the query definition interface contains a superset of the methods of the query projection interface, reflecting that the default query projection returns instances of the candidate class.

          It isn't clear from the external view whether there is a check to be sure that the filter applies to properties of the candidate class. I expect that we can make sure that this is the case by templating the expressions on both the type of the candidate class and the type of the property.

          In order to execute a query, three things are needed: the query definition (including the projection definition), the persistence manager, and the parameters. I believe that there is value in allowing parameters to be bound very late, during the execution of the query. But I also see value in allowing parameters to be bound to a different instance entirely.

          If set individually, properties should be type-checked as well. Using the setParameter(name, value) method on the query definition, I don't see how to check the type. Instead of name, perhaps a method taking the parameter itself instead of the name.

          It will be good to integrate this new query api into the existing query, but I'll leave that discussion for later (perhaps once we agree what the new query should look like).

          Thinking out loud,

          QueryDefinition<T> extends QueryProjection<T>

          { void filter(Predicate<T>); QueryProjection<P> project(Class<P>, projectionList...); get/set Unmodifiable(); }

          QueryProjection<P> {
          }

          QueryExecution<P>

          { setParameter(Parameter<Q>, Q); List<P> executeList(Object... parameters); P executeUnique(Object... parameters); List<P> executeList(Map parameters); P executeUnique(Map parameters); get/set ReadTimeout(int millis); get/set WriteTimeout(int millis); get/set Range(int low, int high); get/set IgnoreCache(boolean); get/set SerializeRead(boolean); }

          PersistenceManagerFactory

          { ... QueryDefinition<T> newQueryDefinition(Class<T> candidateClass), boolean unique); void storeNamedQuery(String name, QueryProjection<?> query); QueryProjection<?> getNamedQuery(String name); }

          PersistenceManager {
          ...
          List<P> executeList(QueryProjection<P>, Object... parameters);
          P executeUnique(QueryProjection<P>, Object... parameters);
          QueryExecution<P> newQueryExecution(ExecutionProjection<P>);

          Show
          Craig L Russell added a comment - I like the direction of this api. My comments here might have been expressed by others; I don't claim ownership of them. I like the naming of the Q classes generated during pre-processing. I consider the requirement to have the persistent classes annotated to be a distraction though. I think we need to consider reuse of the query artifacts by different persistence managers, and also the possibility that the query definition might be stored as a named query. This leads to needing a factory defined on the PMF, not just the PM. The factory might not be needed on the PM depending on other considerations (discussed below). The same query definition could be used by different projections. We could make this explicit in the api by adding a method to the query definition that returns a different interface like QueryProjection<R> where R is the class of the result. And the query definition interface contains a superset of the methods of the query projection interface, reflecting that the default query projection returns instances of the candidate class. It isn't clear from the external view whether there is a check to be sure that the filter applies to properties of the candidate class. I expect that we can make sure that this is the case by templating the expressions on both the type of the candidate class and the type of the property. In order to execute a query, three things are needed: the query definition (including the projection definition), the persistence manager, and the parameters. I believe that there is value in allowing parameters to be bound very late, during the execution of the query. But I also see value in allowing parameters to be bound to a different instance entirely. If set individually, properties should be type-checked as well. Using the setParameter(name, value) method on the query definition, I don't see how to check the type. Instead of name, perhaps a method taking the parameter itself instead of the name. It will be good to integrate this new query api into the existing query, but I'll leave that discussion for later (perhaps once we agree what the new query should look like). Thinking out loud, QueryDefinition<T> extends QueryProjection<T> { void filter(Predicate<T>); QueryProjection<P> project(Class<P>, projectionList...); get/set Unmodifiable(); } QueryProjection<P> { } QueryExecution<P> { setParameter(Parameter<Q>, Q); List<P> executeList(Object... parameters); P executeUnique(Object... parameters); List<P> executeList(Map parameters); P executeUnique(Map parameters); get/set ReadTimeout(int millis); get/set WriteTimeout(int millis); get/set Range(int low, int high); get/set IgnoreCache(boolean); get/set SerializeRead(boolean); } PersistenceManagerFactory { ... QueryDefinition<T> newQueryDefinition(Class<T> candidateClass), boolean unique); void storeNamedQuery(String name, QueryProjection<?> query); QueryProjection<?> getNamedQuery(String name); } PersistenceManager { ... List<P> executeList(QueryProjection<P>, Object... parameters); P executeUnique(QueryProjection<P>, Object... parameters); QueryExecution<P> newQueryExecution(ExecutionProjection<P>);
          Hide
          Andy Jefferson added a comment -

          In the patch, "TypesafeQuery" needs renaming to JDOQLTypedQuery, or JDOQLTypesafeQuery to reflect the fact that it is for JDOQL only (unlike javax.jdo.Query which can be used for any query language).

          Show
          Andy Jefferson added a comment - In the patch, "TypesafeQuery" needs renaming to JDOQLTypedQuery, or JDOQLTypesafeQuery to reflect the fact that it is for JDOQL only (unlike javax.jdo.Query which can be used for any query language).
          Hide
          Andy Jefferson added a comment -

          As already said, there are two parts to this.
          1). Generation of a (JDOQL) query using "Q" classes in a typesafe manner. This revolves around the XXXExpression classes in the patch that was provided 4+ years ago. Since I have seen no negative comments on this unless something is received to the contrary soon then I will commit this in the next week (into the package "javax.jdo.query").

          2). The structure of the JDOQLTypedQuery class, and how it ties in to the query execution process. This needs to make use of whatever interface for QueryExecution so that a JDOQLTypedQuery can be executed using the same methods as a traditional Query. Personally I'm in favour of having a QueryExecution (or equivalent name) interface, and not changing too much more ... i.e let the user create a query like they do now pm.newQuery or pm.newTypedQuery and then execute them using methods in this interface, hence minimising the API change for users.

          Show
          Andy Jefferson added a comment - As already said, there are two parts to this. 1). Generation of a (JDOQL) query using "Q" classes in a typesafe manner. This revolves around the XXXExpression classes in the patch that was provided 4+ years ago. Since I have seen no negative comments on this unless something is received to the contrary soon then I will commit this in the next week (into the package "javax.jdo.query"). 2). The structure of the JDOQLTypedQuery class, and how it ties in to the query execution process. This needs to make use of whatever interface for QueryExecution so that a JDOQLTypedQuery can be executed using the same methods as a traditional Query. Personally I'm in favour of having a QueryExecution (or equivalent name) interface, and not changing too much more ... i.e let the user create a query like they do now pm.newQuery or pm.newTypedQuery and then execute them using methods in this interface, hence minimising the API change for users.
          Hide
          Andy Jefferson added a comment -

          There are (at least) 2 options for the execution part.
          1. Allow creation of Query on the PMF, PM, and then execute on the PM.
          Query q = pmf.createQuery(...)
          pm1.executeQuery(q);
          pm2.executeQuery(q);
          This has complications of the fact that the Query takes its FetchPlan from the PM that creates it (which may be modified from the default FetchPlan) yet with this has to start from a default FetchPlan, it means people will have to make large changes to their existing code (or use deprecated methods all over the place).

          2. Create a Query like now, on the PM, and have the generic Query (what we have now) and the new JDOQLTypedQuery implement QueryExecutor. So we can have consistent execute methods across the Query and JDOQLTypedQuery. Not sure we can do the same with QueryDefinition (other than some of the basic getter/setter stuff) since the JDOQLTypedQuery will be using the javax.jdo.query.XXXExpression classes in its filter(), groupBy(), orderBy(), ... methods and the generic one uses Strings.

          For the first one, who has this use-case of requiring a Query to be useable across PMs? A JDO impl will cache the query+compilation already, PM can save it as named and access the named query in another PM.

          Other requirements ?

          • JDOQLTypedQuery.toString() should return the equivalent single-string JDOQL, so can be converted in that direction.
          • do we need to be able to convert from (JDOQL) single-string to JDOQLTypedQuery?

          SVN now has the JDOQLTypedQuery (and Subquery) added for reference/discussion.

          Show
          Andy Jefferson added a comment - There are (at least) 2 options for the execution part. 1. Allow creation of Query on the PMF, PM, and then execute on the PM. Query q = pmf.createQuery(...) pm1.executeQuery(q); pm2.executeQuery(q); This has complications of the fact that the Query takes its FetchPlan from the PM that creates it (which may be modified from the default FetchPlan) yet with this has to start from a default FetchPlan, it means people will have to make large changes to their existing code (or use deprecated methods all over the place). 2. Create a Query like now, on the PM, and have the generic Query (what we have now) and the new JDOQLTypedQuery implement QueryExecutor. So we can have consistent execute methods across the Query and JDOQLTypedQuery. Not sure we can do the same with QueryDefinition (other than some of the basic getter/setter stuff) since the JDOQLTypedQuery will be using the javax.jdo.query.XXXExpression classes in its filter(), groupBy(), orderBy(), ... methods and the generic one uses Strings. For the first one, who has this use-case of requiring a Query to be useable across PMs? A JDO impl will cache the query+compilation already, PM can save it as named and access the named query in another PM. Other requirements ? JDOQLTypedQuery.toString() should return the equivalent single-string JDOQL, so can be converted in that direction. do we need to be able to convert from (JDOQL) single-string to JDOQLTypedQuery? SVN now has the JDOQLTypedQuery (and Subquery) added for reference/discussion.
          Hide
          Craig L Russell added a comment -

          In order to make use of the new QueryExecution interface it will be useful to make the Query interface templated.
          <T> Query<T> ...

          Show
          Craig L Russell added a comment - In order to make use of the new QueryExecution interface it will be useful to make the Query interface templated. <T> Query<T> ...
          Hide
          Andy Jefferson added a comment - - edited

          What remains on this issue :

          1. spec : I don't mind contributing some text. The problem is that most of Ch14 is based around javax.jdo.Query. What I suggest would be updates to 14.1 to mention about JDOQLTypedQuery, an update to Ch14.5 to add the newJDOQLTypedQuery() method. Then insert a new section 14.10 before "Examples" to discuss solely the JDOQLTypedQuery interface. Finally update the "Examples" section to add some examples at the end. Other ideas? Alternately put the new section after examples, and have its examples inline with the various method descriptions.

          2. tck : this needs tests. I am not the best person to write these since I've written the majority of the API, and the implementation. Volunteer?

          Show
          Andy Jefferson added a comment - - edited What remains on this issue : 1. spec : I don't mind contributing some text. The problem is that most of Ch14 is based around javax.jdo.Query. What I suggest would be updates to 14.1 to mention about JDOQLTypedQuery, an update to Ch14.5 to add the newJDOQLTypedQuery() method. Then insert a new section 14.10 before "Examples" to discuss solely the JDOQLTypedQuery interface. Finally update the "Examples" section to add some examples at the end. Other ideas? Alternately put the new section after examples, and have its examples inline with the various method descriptions. 2. tck : this needs tests. I am not the best person to write these since I've written the majority of the API, and the implementation. Volunteer?
          Hide
          Andy Jefferson added a comment -

          The other option to 1. spec is to create a new chapter 15 straight after "14. Query". The only problem is, I've no idea how these JDO_master.odm and the various chapters are linked, so if people prefer that then maybe someone who knows can create a start point for this new Chapter and I'll update it with the necessary description

          Show
          Andy Jefferson added a comment - The other option to 1. spec is to create a new chapter 15 straight after "14. Query". The only problem is, I've no idea how these JDO_master.odm and the various chapters are linked, so if people prefer that then maybe someone who knows can create a start point for this new Chapter and I'll update it with the necessary description
          Hide
          Craig L Russell added a comment -

          Changing the structure of the specification would mean a really lot of work that I'm not sure is worth the effort.

          I'd suggest adding a section in Chapter 14 that deals with the new query strategy.

          I'll take a look at making the initial changes.

          Show
          Craig L Russell added a comment - Changing the structure of the specification would mean a really lot of work that I'm not sure is worth the effort. I'd suggest adding a section in Chapter 14 that deals with the new query strategy. I'll take a look at making the initial changes.
          Hide
          Tilmann Zäschke added a comment -

          In https://issues.apache.org/jira/browse/JDO-749, ANdy wore in response to my question about streams/lambdas in queries:
          And about yet another 'redesign' of the query API, we have waited nearly 3 years for this one from the provision of the DN TypesafeQuery API with little input from others. Bear in mind that I wouldn't be contributing much time, so if you want it then you need to provide time/ideas/implementations/tests.

          Sorry, I only really joined the discussions here a few months ago, I don't know the history, but I certainly don't want to be disruptive.

          I thought about streams/lambdas a bit more, and I think one key issue is that the API will look very different from JDOQL or the proposed typesafe API, it simply will have to look like the Java 8 stream API (filter(), reduce(), sort(), forEach(), ...).
          Also, it will require considerable effort to even verify feasibility, I guess at least another 6 months at least. The result may be that it actually doesn't really work or makes sense.

          I don't know what the JDO 3.2 timeline is, but I suspect it may be best to follow through with the current proposal.
          I suppose the disadvantage of duplicating the query at a later point with a stream API is smaller than waiting for an API that may not even work. I guess it boils down to the argument that having a 'very good' solution now is preferable to potentially 'perfect' solution that may never come.

          Show
          Tilmann Zäschke added a comment - In https://issues.apache.org/jira/browse/JDO-749 , ANdy wore in response to my question about streams/lambdas in queries: And about yet another 'redesign' of the query API, we have waited nearly 3 years for this one from the provision of the DN TypesafeQuery API with little input from others. Bear in mind that I wouldn't be contributing much time, so if you want it then you need to provide time/ideas/implementations/tests. Sorry, I only really joined the discussions here a few months ago, I don't know the history, but I certainly don't want to be disruptive. I thought about streams/lambdas a bit more, and I think one key issue is that the API will look very different from JDOQL or the proposed typesafe API, it simply will have to look like the Java 8 stream API ( filter() , reduce() , sort() , forEach() , ...). Also, it will require considerable effort to even verify feasibility, I guess at least another 6 months at least. The result may be that it actually doesn't really work or makes sense. I don't know what the JDO 3.2 timeline is, but I suspect it may be best to follow through with the current proposal. I suppose the disadvantage of duplicating the query at a later point with a stream API is smaller than waiting for an API that may not even work. I guess it boils down to the argument that having a 'very good' solution now is preferable to potentially 'perfect' solution that may never come.
          Hide
          Andy Jefferson added a comment -

          If you want a better query API utilising Java 8 language features (and there are clearly lots of ideas around about how it could look, e.g Jinq) i'd encourage you to develop a proof of concept and ask for ideas as your concept starts to take shape - having something usable to visualise it is the best way of getting feedback. Feel free to clone the DN datanucleus-api-jdo and datanucleus-java8 plugins so it is readily visible, and if it comes to something it could initially become part of DN and then later JDO? i.e the same steps as we went through with the JDOQLTypedQuery.

          Show
          Andy Jefferson added a comment - If you want a better query API utilising Java 8 language features (and there are clearly lots of ideas around about how it could look, e.g Jinq) i'd encourage you to develop a proof of concept and ask for ideas as your concept starts to take shape - having something usable to visualise it is the best way of getting feedback. Feel free to clone the DN datanucleus-api-jdo and datanucleus-java8 plugins so it is readily visible, and if it comes to something it could initially become part of DN and then later JDO? i.e the same steps as we went through with the JDOQLTypedQuery.
          Hide
          Tilmann Zäschke added a comment -

          Yes, that sounds good, and thanks for the link to Jinq. That is what I had in mind, except that they seem to load everything into the client first, which I think is not useful except for very small datasets...

          Show
          Tilmann Zäschke added a comment - Yes, that sounds good, and thanks for the link to Jinq. That is what I had in mind, except that they seem to load everything into the client first, which I think is not useful except for very small datasets...

            People

            • Assignee:
              Unassigned
              Reporter:
              Andy Jefferson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development