[SPARK-46275] Protobuf: Permissive mode should return null rather than struct with null fields

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.2, 3.5.0, 4.0.0
• Fix Version/s: 4.0.0, 3.5.1, 3.4.3
• Component/s: Protobuf, Structured Streaming
• Labels: pull-request-available
• Epic Link: Protobuf Support
• Assignee: Raghu Angadi
• Reporter: Raghu Angadi
• Created: 05/Dec/23 22:54
• Updated: 08/Dec/23 22:42
• Resolved: 08/Dec/23 07:44

Description

Consider a protobuf with two fields message Person { string name = 1; int id = 2; 

• The struct returned by from_protobuf("Person") like this:
  • STRUCT<name STRING, id INT>

• If the underlying binary record fails to deserialize, it results in a exception and query fails.

• Buf if the option mode is set to PERMISSIVE , malformed records are tolerated null is returned.
  • BUT: The retuned struct looks like this {"name: null, id: "null"}

• • • This is not convenient to the user.
    • Ideally, from_protobuf() should return null .
  • from_protobuf() borrowed the current behavior from from_avro() implementation. It is not clear what the motivation was.

I think we should update the implementation to return null rather than a struct with null-fields inside.

Issue Links

• links to: GitHub Pull Request #44214
• links to: GitHub Pull Request #44265