[SPARK-33929] Spark-submit with --package deequ doesn't pull all jars

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.7
• Component/s: Spark Submit
• Assignee: Unassigned
• Reporter: Dustin Smith
• Created: 29/Dec/20 04:48
• Updated: 16/Apr/21 06:51

Description

This issue was marked as solved SPARK-24074; however, another hyukjin.kwon pointed out in the comments that version 2.4x was experiencing this same problem when using Amazon Deequ.

This problem exist in 2.3.x ecosystem as well for Deequ.

Issue Links

• relates to: SPARK-35084 [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext