[SPARK-40092] is breeze required if not using ML?

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.3.0
• Component/s: Deploy
• Assignee: Unassigned
• Reporter: t oo
• Created: 16/Aug/22 01:58
• Updated: 16/Aug/22 06:16
• Resolved: 16/Aug/22 06:16

Description

my docker file is very big with pyspark

can i remove dis files below if i don't use 'ML'?

14M     /usr/local/lib/python3.10/site-packages/pyspark/jars/breeze_2.12-1.2.jar

5.9M    /usr/local/lib/python3.10/site-packages/pyspark/jars/spark-mllib_2.12-3.3.0.jar

Issue Links

• duplicates: SPARK-40093 is kubernetes jar required if not using that executor?
• is a clone of: SPARK-40091 is rocksdbjni required if not using streaming?
• is cloned by: SPARK-40093 is kubernetes jar required if not using that executor?