[HUDI-1873] collect() call causing issues with very large upserts

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0, 0.8.0
• Fix Version/s: 0.9.0
• Component/s: spark
• Labels: newbie pull-request-available sev:high
• Environment: EMR 5.28 Spark 11
• Assignee: Unassigned
• Reporter: Matt Pouttu
• Created: 01/May/21 00:15
• Updated: 04/Nov/21 01:53
• Resolved: 27/May/21 14:44

Description

A collect call causes resource issues with very large upserts, and is only used for reporting error messages that are already in the spark task logs. I replaced it with a .isEmpty() call and amended the error message to direct the user to the task logs.

Issue Links

• links to: GitHub Pull Request #2907