[PIG-5439] Support Spark 3 and drop SparkShim

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.19.0
• Component/s: spark
• Assignee: Koji Noguchi
• Reporter: Koji Noguchi
• Created: 11/May/23 20:22
• Updated: 15/Jul/23 20:33

Description

Support Pig-on-Spark to run on spark3.

Initial version would only run up to Spark 3.2.4 and not on 3.3 or 3.4.
This is due to log4j mismatch.

After moving to log4j2 (PIG-5426), we can move Spark to 3.3 or higher.

So far, not all unit/e2e tests pass with the proposed patch but at least compilation goes through.