[PIG-5319] Investigate why TestStoreInstances fails with Spark 2.2

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: spark
• Assignee: Unassigned
• Reporter: Nándor Kollár
• Created: 07/Dec/17 11:00
• Updated: 15/Jun/21 19:30

Description

TestStoreInstances unit test fails with Spark 2.2.x. It seems in job and task commit logic changed a lot since Spark 2.1.x, now it looks like Spark uses a different PigOutputFormat when writing to files, and a different one when getting the OutputCommitters