Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-834

StackOverflowError getting predicates from the metadata provider

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0-incubating
    • Component/s: None
    • Labels:
      None

      Description

      The problem was discovered using the latest SNAPSHOT of 1.4 for a QA run in Hive. In Hive, it can be reproduced using semijoin.q. In particular, the query is the following:

      explain select a.key from t3 a left semi join t2 b on a.value = b.value where a.key > 100
      

      The (partial) stack trace of the Exception is:

      ...
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(ChainedRelMetadataProvider.java:109)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.CachingRelMetadataProvider$CachingInvocationHandler.invoke(CachingRelMetadataProvider.java:132)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:484)
              at org.apache.calcite.rel.metadata.RelMdPredicates.getPredicates(RelMdPredicates.java:186)
              at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke(ReflectiveRelMetadataProvider.java:182)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(ChainedRelMetadataProvider.java:109)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.CachingRelMetadataProvider$CachingInvocationHandler.invoke(CachingRelMetadataProvider.java:132)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(ChainedRelMetadataProvider.java:109)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.CachingRelMetadataProvider$CachingInvocationHandler.invoke(CachingRelMetadataProvider.java:132)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:484)
              at org.apache.calcite.rel.metadata.RelMdPredicates.getPredicates(RelMdPredicates.java:186)
              at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke(ReflectiveRelMetadataProvider.java:182)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(ChainedRelMetadataProvider.java:109)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.CachingRelMetadataProvider$CachingInvocationHandler.invoke(CachingRelMetadataProvider.java:132)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(ChainedRelMetadataProvider.java:109)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.CachingRelMetadataProvider$CachingInvocationHandler.invoke(CachingRelMetadataProvider.java:132)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:484)
              at org.apache.calcite.rel.metadata.RelMdPredicates.getPredicates(RelMdPredicates.java:186)
              at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke(ReflectiveRelMetadataProvider.java:182)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(ChainedRelMetadataProvider.java:109)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.CachingRelMetadataProvider$CachingInvocationHandler.invoke(CachingRelMetadataProvider.java:132)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.ChainedRelMetadataProvider$ChainedInvocationHandler.invoke(ChainedRelMetadataProvider.java:109)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.calcite.rel.metadata.CachingRelMetadataProvider$CachingInvocationHandler.invoke(CachingRelMetadataProvider.java:132)
              at com.sun.proxy.$Proxy22.getPredicates(Unknown Source)
              at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:484)
              at org.apache.calcite.rel.metadata.RelMdPredicates.getPredicates(RelMdPredicates.java:186)
      

        Issue Links

          Activity

          Hide
          jcamachorodriguez Jesus Camacho Rodriguez added a comment -

          Julian Hyde, any ideas? I didn't look into detail yet, but it seems we are hitting this problem in the Hive tests that contain plans with semijoin operators.

          Show
          jcamachorodriguez Jesus Camacho Rodriguez added a comment - Julian Hyde , any ideas? I didn't look into detail yet, but it seems we are hitting this problem in the Hive tests that contain plans with semijoin operators.
          Hide
          julianhyde Julian Hyde added a comment -

          Is the metadata call going into the same RelNode instances multiple times? In order words, is the graph cyclic? (We don't handle cyclic graphs very well - see CALCITE-794 - but usually you get away with it.)

          Try adding metadata handlers like 'public RelOptPredicateList getPredicates(HiveSemiJoin semiJoin)'. See RelMdPredicates (in Calcite) and HiveDefaultRelMetadataProvider (in Hive).

          Show
          julianhyde Julian Hyde added a comment - Is the metadata call going into the same RelNode instances multiple times? In order words, is the graph cyclic? (We don't handle cyclic graphs very well - see CALCITE-794 - but usually you get away with it.) Try adding metadata handlers like 'public RelOptPredicateList getPredicates(HiveSemiJoin semiJoin)'. See RelMdPredicates (in Calcite) and HiveDefaultRelMetadataProvider (in Hive).
          Hide
          jcamachorodriguez Jesus Camacho Rodriguez added a comment -

          This is not a problem on the Calcite side; we were hitting the infinite loop because of the design of the rule in Hive. Thus, I close this issue.

          Show
          jcamachorodriguez Jesus Camacho Rodriguez added a comment - This is not a problem on the Calcite side; we were hitting the infinite loop because of the design of the rule in Hive. Thus, I close this issue.
          Hide
          jcamachorodriguez Jesus Camacho Rodriguez added a comment -

          It seems my initial assessment was wrong; although I had found a workaround on Hive, the problem seems to be in Calcite and was introduced in CALCITE-390.

          I think we should infer predicates for SemiJoin the same way that we do it for Join operators i.e. through JoinConditionBasedPredicateInference. Thus, left inferred and right inferred will only contain inferred predicates. Otherwise, the problem that we are finding in Hive is that JoinPushTransitivePredicatesRule might enter in an infinite loop as it continuously infers the same filter predicates.

          Julian Hyde, I'm sure you have a clearer idea about this, what do you think? I have created a pull request with the modification of JoinConditionBasedPredicateInference that solves this issue https://github.com/apache/incubator-calcite/pull/121

          Thanks

          Show
          jcamachorodriguez Jesus Camacho Rodriguez added a comment - It seems my initial assessment was wrong; although I had found a workaround on Hive, the problem seems to be in Calcite and was introduced in CALCITE-390 . I think we should infer predicates for SemiJoin the same way that we do it for Join operators i.e. through JoinConditionBasedPredicateInference. Thus, left inferred and right inferred will only contain inferred predicates. Otherwise, the problem that we are finding in Hive is that JoinPushTransitivePredicatesRule might enter in an infinite loop as it continuously infers the same filter predicates. Julian Hyde , I'm sure you have a clearer idea about this, what do you think? I have created a pull request with the modification of JoinConditionBasedPredicateInference that solves this issue https://github.com/apache/incubator-calcite/pull/121 Thanks
          Hide
          julianhyde Julian Hyde added a comment -

          Harish Butani, Since you wrote RelOptPredicateList, can you explain the difference between "inferred predicates" and "pulled up predicates"? I think I screwed things up in CALCITE-390 because I didn't understand.

          Jesus Camacho Rodriguez, I'm testing your pull request and will check in if it improves things.

          Show
          julianhyde Julian Hyde added a comment - Harish Butani , Since you wrote RelOptPredicateList, can you explain the difference between "inferred predicates" and "pulled up predicates"? I think I screwed things up in CALCITE-390 because I didn't understand. Jesus Camacho Rodriguez , I'm testing your pull request and will check in if it improves things.
          Show
          julianhyde Julian Hyde added a comment - Fixed in http://git-wip-us.apache.org/repos/asf/incubator-calcite/commit/c9d7233d .
          Hide
          rhbutani Harish Butani added a comment -

          Julian Hyde I haven't worked on this code in a long time.
          It will take some effort to get back in the code to answer your question.
          Looks like you have a patch. Do you still want me to look at this?

          Show
          rhbutani Harish Butani added a comment - Julian Hyde I haven't worked on this code in a long time. It will take some effort to get back in the code to answer your question. Looks like you have a patch. Do you still want me to look at this?
          Hide
          julianhyde Julian Hyde added a comment -

          Harish Butani, It would be helpful if you could write a one-line description of each of RelOptPredicateList's 3 fields. I never understood what you meant by "inferred" and "pulled up" - I should have pushed back when you contributed the patch - and until I do I won't be able to maintain this code.

          All the context you need is in https://github.com/apache/incubator-calcite/commit/c33e602fa0c215e8a9f426781f9bb445fd9e6e94.

          Just write the descriptions in this case and I will add them to the code.

          Jesus Camacho Rodriguez, Can you please close that pull request. I forgot to.

          Show
          julianhyde Julian Hyde added a comment - Harish Butani , It would be helpful if you could write a one-line description of each of RelOptPredicateList's 3 fields. I never understood what you meant by "inferred" and "pulled up" - I should have pushed back when you contributed the patch - and until I do I won't be able to maintain this code. All the context you need is in https://github.com/apache/incubator-calcite/commit/c33e602fa0c215e8a9f426781f9bb445fd9e6e94 . Just write the descriptions in this case and I will add them to the code. Jesus Camacho Rodriguez , Can you please close that pull request. I forgot to.
          Hide
          jcamachorodriguez Jesus Camacho Rodriguez added a comment -

          Julian Hyde, I already closed the request, no problem. Thanks for pushing it to master.

          About "inferred" vs "pulled up", my understanding is that for joins:

          • leftInferred contains only the predicates that were inferred from the right input subplan
          • rightInferred contains only the predicates that were inferred from the left input subplan
          • pulledUp contains all predicates that can be pulled up from the join and its inputs

          Thus, leftInferred and rightInferred is only created for join operators, and it is used by JoinPushTransitivePredicatesRule to create new filters on the join inputs.

          Show
          jcamachorodriguez Jesus Camacho Rodriguez added a comment - Julian Hyde , I already closed the request, no problem. Thanks for pushing it to master. About "inferred" vs "pulled up", my understanding is that for joins: leftInferred contains only the predicates that were inferred from the right input subplan rightInferred contains only the predicates that were inferred from the left input subplan pulledUp contains all predicates that can be pulled up from the join and its inputs Thus, leftInferred and rightInferred is only created for join operators, and it is used by JoinPushTransitivePredicatesRule to create new filters on the join inputs.
          Hide
          jnadeau Jacques Nadeau added a comment -

          Resolved in release 1.4.0-incubating (2015-08-23)

          Show
          jnadeau Jacques Nadeau added a comment - Resolved in release 1.4.0-incubating (2015-08-23)

            People

            • Assignee:
              jcamachorodriguez Jesus Camacho Rodriguez
              Reporter:
              jcamachorodriguez Jesus Camacho Rodriguez
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development