[BEAM-13294] innerBroadcastJoin fails when batch column Nullable, streaming column not

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Affects Version/s: 2.21.0, 2.22.0, 2.23.0, 2.24.0, 2.25.0, 2.26.0, 2.27.0, 2.28.0, 2.29.0, 2.30.0, 2.31.0, 2.32.0, 2.33.0, 2.34.0, 2.35.0
• Fix Version/s: 2.36.0
• Component/s: dsl-sql, dsl-sql-zetasql, sdk-java-core
• Assignee: Andrew Pilloud
• Reporter: Andrew Pilloud
• Created: 19/Nov/21 20:09
• Updated: 09/Dec/21 20:44
• Resolved: 09/Dec/21 20:44

Description

The join always fail when given a batch column that is Nullable and a streaming column that is Not Nullable. This was discovered in BeamSideInputJoinRel but is likely an issue in the Beam core join library.

Trivial reproduction with a small test diff:
./gradlew :runners:direct-java:needsRunnerTests --tests org.apache.beam.sdk.schemas.transforms.JoinTest.testInnerJoinDifferentKeys

--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
@@ -51,7 +51,7 @@ public class JoinTest {
           .build();
   private static final Schema CG_SCHEMA_2 =
       Schema.builder()
- .addStringField("user2")
+ .addNullableField("user2", Schema.FieldType.STRING)
           .addInt32Field("count2")
           .addStringField("country2")
           .build();

At a lower level:
./gradlew :runners:direct-java:needsRunnerTests --tests org.apache.beam.sdk.schemas.transforms.CoGroupTest.testCoGroupByDifferentFields

--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CoGroupTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CoGroupTest.java
@@ -237,7 +237,7 @@ public class CoGroupTest {
 
   private static final Schema CG_SCHEMA_2 =
       Schema.builder()
-          .addStringField("user2")
+          .addNullableField("user2", Schema.FieldType.STRING)
           .addInt32Field("count2")
           .addStringField("country2")
           .build();

Issue Links

• relates to: BEAM-9670 CoGroup transform should allow widening nullability in key schemas
• links to: GitHub Pull Request #16158