[KAFKA-16858] Flatten SMT throws NPE

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6.0
• Fix Version/s: 3.8.0, 3.9.0
• Component/s: connect
• Environment: Kafka 3.6 by way of CP 7.6.0
• Assignee: Unassigned
• Reporter: Adam Strickland
• Created: 28/May/24 14:03
• Updated: 12/Jun/24 19:13
• Resolved: 12/Jun/24 19:13

Description

ConnectSchema.expectedClassesFor sometimes will throw an NPE as part of a call to an SMT chain.  Stack trace snippet:

at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.apply(MomentFlatten.java:84)
at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.applyWithSchema(MomentFlatten.java:173)
at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)
at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)
at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)
at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:274)
at org.apache.kafka.connect.data.Struct.put(Struct.java:203)
at org.apache.kafka.connect.data.Struct.put(Struct.java:216)
at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:255)
at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:213)
at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:224)
at org.apache.kafka.connect.data.ConnectSchema.expectedClassesFor(ConnectSchema.java:268)

(the above transform is a sub-class of o.a.k.connect.transforms.Flatten; have confirmed that the error occurs regardless).

The field being transformed is an array of structs. If the call to Schema#valueSchema() (o.a.k.connect.data.ConnectSchema.java:255) returns null, the subsequent call to Schema#name() at o.a.k.connect.data.ConnectSchema:268 throws an NPE.

The strange thing that we have observed is that this doesn't always happen; sometimes the struct's schema is found and sometimes it is not. We have been unable to determine the root cause, but have constructed a test that replicates the problem as observed (see attachment).

In our case we have worked around the issue with the aforementioned sub-class of Flatten, catching and logging the NPE on that specific use-case.

Issue Links

• is related to: KAFKA-16870 Values.parseString returns objects which fail ConnectSchema.validateValue
• links to: GitHub Pull Request #16161