[CALCITE-2000] UNNEST a collection that has a field with nested data generates an Exception

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.27.0
• Labels: pull-request-available
• Assignee: Ruben Q L
• Reporter: Luis Fernando Kauer
• Created: 05/Oct/17 14:56
• Updated: 03/Jun/21 22:31
• Resolved: 01/Mar/21 08:52

Description

When unnesting a column that has another level of nesting it generates the following exception:

Caused by: java.lang.ArrayStoreException
	at java.lang.System.arraycopy(Native Method)
	at org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
	at org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
	at org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
	at org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
	at Baz$4$1.moveNext(Unknown Source)
	at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(Linq4j.java:680)
	at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
	at org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
	at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
	at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
	at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
	at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:1)
	at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
	at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:607)
	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
	at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
	... 29 more

The problem is that Calcite uses FlatLists internally that expect items to implement Comparable.
However, a column with a nested collection will have an array or a List as the column value, which do not implement Comparable, so it generates the Exception.
To test this in JdbcTest, I created a field in Employee with a collection.
For simplicity, I created a field called dependents in Employee class:

public static class HrSchema {
    @Override public String toString() {
      return "HrSchema";
    }

    public final Dependent[] dependents = {
        new Dependent(10, "Michael"),
        new Dependent(10, "Jane"),
    };
    public final Employee[] emps = {
      new Employee(100, 10, "Bill", 10000, 1000, Arrays.asList(dependents)),
      new Employee(200, 20, "Eric", 8000, 500,  Collections.<Dependent>emptyList()),
      new Employee(150, 10, "Sebastian", 7000, null,  Collections.<Dependent>emptyList()),
      new Employee(110, 10, "Theodore", 11500, 250,  Collections.<Dependent>emptyList()),
    };
    public final Department[] depts = {
      new Department(10, "Sales", Arrays.asList(emps[0], emps[2]),
          new Location(-122, 38)),
      new Department(30, "Marketing", Collections.<Employee>emptyList(),
          new Location(0, 52)),
      new Department(40, "HR", Collections.singletonList(emps[1]), null),
    };
    public final Dependent[] locations = {
      new Dependent(10, "San Francisco"),
      new Dependent(20, "San Diego"),
    };

    public QueryableTable foo(int count) {
      return Smalls.generateStrings(count);
    }

    public TranslatableTable view(String s) {
      return Smalls.view(s);
    }
  }

  public static class Employee {
    public final int empid;
    public final int deptno;
    public final String name;
    public final float salary;
    public final Integer commission;
    public final List<Dependent> dependents;

    public Employee(int empid, int deptno, String name, float salary,
        Integer commission) {
      this(empid, deptno, name, salary, commission, Collections.<Dependent>emptyList());
    }

    public Employee(int empid, int deptno, String name, float salary,
        Integer commission, List<Dependent> dependents) {
      this.empid = empid;
      this.deptno = deptno;
      this.name = name;
      this.salary = salary;
      this.commission = commission;
      this.dependents = dependents;
    }

    @Override public String toString() {
      return "Employee [empid: " + empid + ", deptno: " + deptno
          + ", name: " + name + "]";
    }

    @Override public boolean equals(Object obj) {
      return obj == this
          || obj instanceof Employee
          && empid == ((Employee) obj).empid;
    }
  }

Just running the test case JdbcTest.testUnnestArrayColumn generates the exception.

Issue Links

• is duplicated by: CALCITE-4064 Unnest on array with nested structs causes ArrayStoreException
• is related to: CALCITE-4063 Unnest an array of single-item structs causes ClassCastException
• links to: GitHub Pull Request #2353