[HUDI-1743] Add support for Spark SQL File based transformer for deltastreamer

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Component/s: deltastreamer
• Labels: features pull-request-available sev:normal
• Assignee: Vinoth Govindarajan
• Reporter: Vinoth Govindarajan
• Created: 31/Mar/21 05:08
• Updated: 25/Apr/22 04:03
• Resolved: 25/Apr/22 04:03

Description

The current SQLQuery based transformer is limited in functionality, you can't pass multiple Spark SQL statements separated by a semicolon which is necessary if your transformation is complex.

 

The ask is to add a new SQLFileBasedTransformer which takes a Spark SQL file as input with multiple Spark SQL statements and applies the transformation to the delta streamer payload.

Issue Links

• links to: GitHub Pull Request #2747