Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-742

RelFieldTrimmer throws NoSuchElementException in some cases

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 1.4.0-incubating
    • Component/s: None
    • Labels:
      None

      Description

      RelFieldTrimmer runs into NoSuchElementException in some cases.

      Stack trace:

      Exception in thread "main" java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Sort,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
      	at org.apache.calcite.util.Util.newInternal(Util.java:743)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trim(RelFieldTrimmer.java:175)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:947)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:820)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:768)
      	at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:109)
      	at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:730)
      	at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:145)
      	at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:105)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:607)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:244)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10048)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:207)
      	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
      	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
      	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
      	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
      	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
      	... 32 more
      Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Sort,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
      	at org.apache.calcite.util.Util.newInternal(Util.java:743)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:210)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:499)
      	... 37 more
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
      	... 40 more
      Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Project,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
      	at org.apache.calcite.util.Util.newInternal(Util.java:743)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:210)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:499)
      	... 45 more
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
      	... 48 more
      Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Aggregate,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
      	at org.apache.calcite.util.Util.newInternal(Util.java:743)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:210)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:345)
      	... 53 more
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
      	... 56 more
      Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Project,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
      	at org.apache.calcite.util.Util.newInternal(Util.java:743)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:210)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:772)
      	... 61 more
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
      	... 64 more
      Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Project,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
      	at org.apache.calcite.util.Util.newInternal(Util.java:743)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:210)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:345)
      	... 69 more
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
      	... 72 more
      Caused by: java.util.NoSuchElementException
      	at java.util.AbstractList$Itr.next(AbstractList.java:364)
      	at java.util.AbstractList.hashCode(AbstractList.java:540)
      	at org.apache.calcite.util.Util.hash(Util.java:230)
      	at org.apache.calcite.util.Pair.hashCode(Pair.java:87)
      	at com.google.common.base.Equivalences$Equals.doHash(Equivalences.java:70)
      	at com.google.common.base.Equivalence.hash(Equivalence.java:105)
      	at com.google.common.cache.LocalCache.hash(LocalCache.java:1888)
      	at com.google.common.cache.LocalCache.getIfPresent(LocalCache.java:3953)
      	at com.google.common.cache.LocalCache$LocalManualCache.getIfPresent(LocalCache.java:4758)
      	at org.apache.calcite.rel.type.RelDataTypeFactoryImpl.canonize(RelDataTypeFactoryImpl.java:352)
      	at org.apache.calcite.rel.type.RelDataTypeFactoryImpl.createStructType(RelDataTypeFactoryImpl.java:148)
      	at org.apache.calcite.rel.type.RelDataTypeFactoryImpl.createStructType(RelDataTypeFactoryImpl.java:172)
      	at org.apache.calcite.plan.RelOptUtil.permute(RelOptUtil.java:2589)
      	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:383)
      	... 77 more
      

      The query executed is the following one:

      SELECT w_warehouse_name, 
             w_warehouse_sq_ft, 
             w_city, 
             w_county, 
             w_state, 
             w_country, 
             ship_carriers, 
             year, 
             Sum(jan_sales)                     AS jan_sales, 
             Sum(feb_sales)                     AS feb_sales, 
             Sum(jan_sales / w_warehouse_sq_ft) AS jan_sales_per_sq_foot, 
             Sum(feb_sales / w_warehouse_sq_ft) AS feb_sales_per_sq_foot, 
             Sum(jan_net)                       AS jan_net, 
             Sum(feb_net)                       AS feb_net 
      FROM   (SELECT w_warehouse_name, 
                     w_warehouse_sq_ft, 
                     w_city, 
                     w_county, 
                     w_state, 
                     w_country, 
                     Concat('DHL', ',', 'BARIAN') AS ship_carriers, 
                     d_year                       AS year, 
                     Sum(CASE 
                           WHEN d_moy = 1 THEN ws_ext_sales_price * ws_quantity 
                           ELSE Cast(0 AS DECIMAL(7, 2)) 
                         end)                     AS jan_sales, 
                     Sum(CASE 
                           WHEN d_moy = 2 THEN ws_ext_sales_price * ws_quantity 
                           ELSE Cast(0 AS DECIMAL(7, 2)) 
                         end)                     AS feb_sales, 
                     Sum(CASE 
                           WHEN d_moy = 1 THEN ws_net_paid * ws_quantity 
                           ELSE Cast(0 AS DECIMAL(7, 2)) 
                         end)                     AS jan_net, 
                     Sum(CASE 
                           WHEN d_moy = 2 THEN ws_net_paid * ws_quantity 
                           ELSE Cast(0 AS DECIMAL(7, 2)) 
                         end)                     AS feb_net 
              FROM   web_sales 
                     JOIN warehouse 
                       ON web_sales.ws_warehouse_sk = warehouse.w_warehouse_sk 
                     JOIN date_dim 
                       ON web_sales.ws_sold_date_sk = date_dim.d_date_sk 
                     JOIN time_dim 
                       ON web_sales.ws_sold_time_sk = time_dim.t_time_sk 
                     JOIN ship_mode 
                       ON web_sales.ws_ship_mode_sk = ship_mode.sm_ship_mode_sk 
              WHERE  d_year = 2001 
                     AND t_time BETWEEN 30838 AND 30838 + 28800 
                     AND sm_carrier IN ( 'DHL', 'BARIAN' ) 
              GROUP  BY w_warehouse_name, 
                        w_warehouse_sq_ft, 
                        w_city, 
                        w_county, 
                        w_state, 
                        w_country, 
                        d_year 
              UNION DISTINCT
              SELECT w_warehouse_name, 
                     w_warehouse_sq_ft, 
                     w_city, 
                     w_county, 
                     w_state, 
                     w_country, 
                     Concat('DHL', ',', 'BARIAN') AS ship_carriers, 
                     d_year                       AS year, 
                     Sum(CASE 
                           WHEN d_moy = 1 THEN cs_sales_price * cs_quantity 
                           ELSE Cast(0 AS DECIMAL(7, 2)) 
                         end)                     AS jan_sales, 
                     Sum(CASE 
                           WHEN d_moy = 2 THEN cs_sales_price * cs_quantity 
                           ELSE Cast(0 AS DECIMAL(7, 2)) 
                         end)                     AS feb_sales, 
                     Sum(CASE 
                           WHEN d_moy = 1 THEN cs_net_paid_inc_tax * cs_quantity 
                           ELSE Cast(0 AS DECIMAL(7, 2)) 
                         end)                     AS jan_net, 
                     Sum(CASE 
                           WHEN d_moy = 2 THEN cs_net_paid_inc_tax * cs_quantity 
                           ELSE Cast(0 AS DECIMAL(7, 2)) 
                         end)                     AS feb_net 
              FROM   catalog_sales 
                     JOIN warehouse 
                       ON catalog_sales.cs_warehouse_sk = warehouse.w_warehouse_sk 
                     JOIN date_dim 
                       ON catalog_sales.cs_sold_date_sk = date_dim.d_date_sk 
                     JOIN time_dim 
                       ON catalog_sales.cs_sold_time_sk = time_dim.t_time_sk 
                     JOIN ship_mode 
                       ON catalog_sales.cs_ship_mode_sk = ship_mode.sm_ship_mode_sk 
              WHERE  d_year = 2001 
                     AND t_time BETWEEN 30838 AND 30838 + 28800 
                     AND sm_carrier IN ( 'DHL', 'BARIAN' ) 
              GROUP  BY w_warehouse_name, 
                        w_warehouse_sq_ft, 
                        w_city, 
                        w_county, 
                        w_state, 
                        w_country, 
                        d_year) x 
      GROUP  BY w_warehouse_name, 
                w_warehouse_sq_ft, 
                w_city, 
                w_county, 
                w_state, 
                w_country, 
                ship_carriers, 
                year 
      ORDER  BY w_warehouse_name 
      LIMIT  100; 
      

      The following smaller query can be also used to reproduce the problem:

      EXPLAIN
      SELECT w_warehouse_sq_ft, 
             Sum(jan_sales)                     AS jan_sales, 
             Sum(jan_sales / w_warehouse_sq_ft) AS jan_sales_per_sq_foot, 
             Sum(jan_net)                       AS jan_net
      FROM   (SELECT w_warehouse_sq_ft, 
                     0 AS jan_sales,
                     0 AS jan_net
              FROM   web_sales 
                     JOIN warehouse 
                       ON web_sales.ws_warehouse_sk = warehouse.w_warehouse_sk 
              UNION
              SELECT w_warehouse_sq_ft, 
                     0 AS jan_sales,
                     0 AS jan_net
              FROM   web_sales 
                     JOIN warehouse 
                       ON web_sales.ws_warehouse_sk = warehouse.w_warehouse_sk) x 
      GROUP  BY w_warehouse_sq_ft; 
      

      The problem is in RelFieldTrimmer. The problem seems to be in trimChild method (line 191), and in particular, in the following lines:

          final ImmutableList<RelCollation> collations =
              RelMetadataQuery.collations(rel);
      

      trimChild passes down the columns that we need to keep, including the columns on which collations exist.
      Currently we take the collations from the parent relation, which does not seem correct, as we end up pointing to positions that do not exist in the child relation; it seems the collations should be taken from the child RelNode. Thus, the call would be:

          final ImmutableList<RelCollation> collations =
              RelMetadataQuery.collations(input);
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jcamachorodriguez Jesus Camacho Rodriguez
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: