Apache Drill
  1. Apache Drill
  2. DRILL-1132

CTAS statement gives IndexOutOfBoundsException during planning

    Details

      Description

      Using TPCH schema, the following CTAS statement fails with IOBE during planning:

      CREATE TABLE T6 AS SELECT n_regionkey, count, n_nationkey
      FROM nation
      GROUP BY n_regionkey, n_nationkey;

      message: "Failure while setting up Foreman. < AssertionError:[ Internal error: Error while applying rule ExpandConversionRule, args [rel#18181:AbstractConverter.PHYSICAL.HASH_DISTRIBUTED([[$0], [$2]]).[0, 2](child=rel#17954:Subset#9.PHYSICAL.HASH_DISTRIBUTED([[$0]]).[],convention=PHYSICAL,DrillDistributionTraitDef=HASH_DISTRIBUTED([[$0], [$2]]),sort=[0, 2])] ] < IndexOutOfBoundsException:[ index (2) must be less than size (2) ]"
      ]

      NOTE: running the SELECT portion of the CTAS succeeds.

        Activity

        Hide
        Aman Sinha added a comment -

        The same problem occurs for the following query with 3 GROUP-BY columns
        create table tt9 as select n_nationkey, n_regionkey, n_name from nation group by n_nationkey, n_regionkey, n_name;

        The problem is that when we project the output of the WriterPrel, it has a RowType of 2 columns only:
        RecordType(VARCHAR(255) Fragment, BIGINT Number of records written)

        Whereas we are trying to create a converted Rel with collation and distribution on 3 columns (see below). Will discuss the desired behavior with relevant folks and provide a fix.

        18:22:03.239 [2b29914b-7b51-4f6e-948e-5986a447bea5:foreman] ERROR o.a.drill.exec.work.foreman.Foreman - Error 7f0c2271-ef9a-49ee-bc22-3feb6666b25b: Failure while setting up Foreman.
        java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) ~[guava-14.0.1.jar:na]
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) ~[guava-14.0.1.jar:na]
        at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81) ~[guava-14.0.1.jar:na]
        at org.eigenbase.rex.RexBuilder.makeInputRef(RexBuilder.java:731) ~[optiq-core-0.7-20140710.204128-10.jar:na]
        at org.eigenbase.rel.SortRel.<init>(SortRel.java:94) ~[optiq-core-0.7-20140710.204128-10.jar:na]
        at org.eigenbase.rel.SortRel.<init>(SortRel.java:59) ~[optiq-core-0.7-20140710.204128-10.jar:na]
        at org.eigenbase.rel.RelCollationTraitDef.convert(RelCollationTraitDef.java:77) ~[optiq-core-0.7-20140710.204128-10.jar:na]
        at org.eigenbase.rel.RelCollationTraitDef.convert(RelCollationTraitDef.java:36) ~[optiq-core-0.7-20140710.204128-10.jar:na]
        at org.eigenbase.relopt.volcano.VolcanoPlanner.changeTraitsUsingConverters(VolcanoPlanner.java:1002) ~[optiq-core-0.7-20140710.204128-10.jar:na]
        at org.eigenbase.relopt.volcano.VolcanoPlanner.changeTraitsUsingConverters(VolcanoPlanner.java:1094) ~[optiq-core-0.7-20140710.204128-10.jar:na]
        at org.eigenbase.relopt.volcano.AbstractConverter$ExpandConversionRule.onMatch(AbstractConverter.java:107) ~[optiq-core-0.7-20140710.204128-10.jar:na]
        at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:221) ~[optiq-core-0.7-20140710.204128-10.jar:na]
        at org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:653) ~[optiq-core-0.7-20140710.204128-10.jar:na]
        at net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:271) ~[optiq-core-0.7-20140710.204128-10.jar:na]

        Show
        Aman Sinha added a comment - The same problem occurs for the following query with 3 GROUP-BY columns create table tt9 as select n_nationkey, n_regionkey, n_name from nation group by n_nationkey, n_regionkey, n_name; The problem is that when we project the output of the WriterPrel, it has a RowType of 2 columns only: RecordType(VARCHAR(255) Fragment, BIGINT Number of records written) Whereas we are trying to create a converted Rel with collation and distribution on 3 columns (see below). Will discuss the desired behavior with relevant folks and provide a fix. 18:22:03.239 [2b29914b-7b51-4f6e-948e-5986a447bea5:foreman] ERROR o.a.drill.exec.work.foreman.Foreman - Error 7f0c2271-ef9a-49ee-bc22-3feb6666b25b: Failure while setting up Foreman. java.lang.IndexOutOfBoundsException: index (2) must be less than size (2) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) ~ [guava-14.0.1.jar:na] at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) ~ [guava-14.0.1.jar:na] at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81) ~ [guava-14.0.1.jar:na] at org.eigenbase.rex.RexBuilder.makeInputRef(RexBuilder.java:731) ~ [optiq-core-0.7-20140710.204128-10.jar:na] at org.eigenbase.rel.SortRel.<init>(SortRel.java:94) ~ [optiq-core-0.7-20140710.204128-10.jar:na] at org.eigenbase.rel.SortRel.<init>(SortRel.java:59) ~ [optiq-core-0.7-20140710.204128-10.jar:na] at org.eigenbase.rel.RelCollationTraitDef.convert(RelCollationTraitDef.java:77) ~ [optiq-core-0.7-20140710.204128-10.jar:na] at org.eigenbase.rel.RelCollationTraitDef.convert(RelCollationTraitDef.java:36) ~ [optiq-core-0.7-20140710.204128-10.jar:na] at org.eigenbase.relopt.volcano.VolcanoPlanner.changeTraitsUsingConverters(VolcanoPlanner.java:1002) ~ [optiq-core-0.7-20140710.204128-10.jar:na] at org.eigenbase.relopt.volcano.VolcanoPlanner.changeTraitsUsingConverters(VolcanoPlanner.java:1094) ~ [optiq-core-0.7-20140710.204128-10.jar:na] at org.eigenbase.relopt.volcano.AbstractConverter$ExpandConversionRule.onMatch(AbstractConverter.java:107) ~ [optiq-core-0.7-20140710.204128-10.jar:na] at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:221) ~ [optiq-core-0.7-20140710.204128-10.jar:na] at org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:653) ~ [optiq-core-0.7-20140710.204128-10.jar:na] at net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:271) ~ [optiq-core-0.7-20140710.204128-10.jar:na]
        Hide
        Aman Sinha added a comment -

        Uploaded patch. Create the Writer with the child's distribution because the degree of parallelism for the writer should correspond to the number of child minor fragments. However, the Writer itself is not concerned with the collation of the child. Note that the Writer's output RowType consists of

        {fragment_id varchar(255), number_of_records_written bigint}

        which are very different from the child's output RowType.

        Show
        Aman Sinha added a comment - Uploaded patch. Create the Writer with the child's distribution because the degree of parallelism for the writer should correspond to the number of child minor fragments. However, the Writer itself is not concerned with the collation of the child. Note that the Writer's output RowType consists of {fragment_id varchar(255), number_of_records_written bigint} which are very different from the child's output RowType.

          People

          • Assignee:
            DrillCommitter
            Reporter:
            Aman Sinha
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Resolved:

              Development