[HUDI-60] [UMBRELLA] Support Apache Beam / Hudi IO

• Type: Epic
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.1.0, 0.15.0
• Component/s: spark, Utilities
• Labels: gsoc gsoc2021 hudi-umbrellas mentor
• Epic Name: Support Apache Beam for incremental tailing
• Assignee: xy
• Reporter: Vinoth Chandar
• Created: 06/Mar/19 01:03
• Updated: 25/Mar/24 11:36

Description

We would like to add a HudiIO for Beam, along the lines of https://github.com/apache/beam/blob/master/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java 

For the initial cut : we can leave the table services turned off on the writer and advise users to run them independently?

During this work - we can also look into anything need to be fixed on the java-client module, which works with GenericRecords as well (used by the Kafka Connect Sink). So if thats in shape, this can be much easier.

Issue Links

• links to: GitHub issue