[HUDI-1765] disabling auto commit does not work with spark ds

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Fix Version/s: 0.11.0
• Component/s: spark
• Labels: sev:high
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 06/Apr/21 03:50
• Updated: 13/Dec/21 14:28
• Resolved: 13/Dec/21 14:28

Description

commitAndPerformPostOperations in HoodieSparkSqlWriter commits all operations w/o checking for autoCommitProp. So, this prop is not being honored at spark ds layer. 

 

Added below test to TestCOWDataSource and it fails. 

https://gist.github.com/nsivabalan/479320351bec95ba3e0c0dfa5abb2b06