[SPARK-40093] is kubernetes jar required if not using that executor?

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.3.0
• Component/s: Deploy
• Assignee: Unassigned
• Reporter: t oo
• Created: 16/Aug/22 02:00
• Updated: 12/Dec/22 18:10
• Resolved: 16/Aug/22 06:16

Description

my docker file is very big with pyspark

can i remove dis files below if i don't use 'kubernetes executor'?

11M     total
4.0M    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-core-5.12.2.jar
840K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-client-5.12.2.jar
760K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-admissionregistration-5.12.2.jar
704K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-apiextensions-5.12.2.jar
640K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-autoscaling-5.12.2.jar
528K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-extensions-5.12.2.jar
516K    /usr/local/lib/python3.10/site-packages/pyspark/jars/spark-kubernetes_2.12-3.3.0.jar
456K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-networking-5.12.2.jar
436K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-apps-5.12.2.jar
364K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-storageclass-5.12.2.jar
336K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-policy-5.12.2.jar
264K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-flowcontrol-5.12.2.jar
244K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-batch-5.12.2.jar
192K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-discovery-5.12.2.jar
176K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-rbac-5.12.2.jar
160K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-node-5.12.2.jar
144K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-certificates-5.12.2.jar
104K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-events-5.12.2.jar
80K     /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-metrics-5.12.2.jar
68K     /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-scheduling-5.12.2.jar
48K     /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-coordination-5.12.2.jar
20K     /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-common-5.12.2.jar

Issue Links

• is a clone of: SPARK-40092 is breeze required if not using ML?
• is duplicated by: SPARK-40091 is rocksdbjni required if not using streaming?
• is duplicated by: SPARK-40092 is breeze required if not using ML?