[BEAM-14026] Using unnest on a array with rows having nested rows throws IllegalArgumentException

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.36.0
• Fix Version/s: 2.38.0
• Component/s: dsl-sql, sdk-java-core
• Flags: Patch, Important

Language

java

• Assignee: Unassigned
• Reporter: Abhijeet
• Created: 02/Mar/22 16:10
• Updated: 04/Mar/22 23:04
• Resolved: 04/Mar/22 23:04

Description

Using unnest with example sql statement like

"select t.a1, t.a2, t.a3, d.b1, d.b2, d.b4, d.b3.c1, d.b3.c2, d.b3.c3 from test t cross join unnest(t.a4) d"

on the following dataset throws Illegal argument exception

Java code snippet

---

// define the input row format level3
Schema level3Type =
Schema.builder().addInt32Field("c1").addStringField("c2").addDoubleField("c3").build();

Row level3Row1 = Row.withSchema(level3Type).addValues(1, "row", 1.0).build();
Row level3Row2 = Row.withSchema(level3Type).addValues(2, "row", 2.0).build();
Row level3Row3 = Row.withSchema(level3Type).addValues(3, "row", 3.0).build();

// define the input row format level3
Schema level2Type =
Schema.builder().addInt32Field("b1")
.addStringField("b2")
.addRowField("b3", level3Type)
.addDoubleField("b4").build();

Row level2Row1 = Row.withSchema(level2Type).addValues(1, "row", level3Row1, 1.0).build();
Row level2Row2 = Row.withSchema(level2Type).addValues(2, "row", level3Row2, 2.0).build();
Row level2Row3 = Row.withSchema(level2Type).addValues(3, "row", level3Row3, 3.0).build();

// define the input row format level3
Schema level1Type =
Schema.builder().addInt32Field("a1")
.addStringField("a2")
.addDoubleField("a3")
.addArrayField("a4", Schema.FieldType.row(level2Type))
.build();
Row level1Row1 = Row.withSchema(level1Type).addValues(1, "row", 1.0,
Arrays.asList(level2Row1, level2Row2, level2Row3)).build();
Row level1Row2 = Row.withSchema(level1Type).addValues(2, "row", 2.0,
Arrays.asList(level2Row1, level2Row2, level2Row3)).build();
Row level1Row3 = Row.withSchema(level1Type).addValues(3, "row", 3.0,
Arrays.asList(level2Row1, level2Row2, level2Row3)).build();

// create a source PCollection with Create.of();
PCollection<Row> inputTable =
PBegin.in(p).apply(Create.of(level1Row1, level1Row2, level1Row3).withRowSchema(level1Type));

---

Throws following exception

java.lang.IllegalArgumentException: Row expected 10 fields. initialized with 8 fields

Issue Links

• links to: GitHub Pull Request #16988