Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Right now we use spark-app-selector as the label for the applicationID for Spark pods: https://github.com/apache/incubator-yunikorn-k8shim/blob/master/pkg/common/utils/utils.go#L87
When linking the Spark pod group to the CRD we use the Application ID, but for the CRD we use namespace-name convention as ApplicationID and for the spark pod group we use the spark-app-selector what will result in having two different applications internally: one for the CRD and one for the Spark POD group.
I think we should change this label to something else, what we can modify easily without any side effects.
Attachments
Attachments
Issue Links
- links to
hi kmarton
Looks like we can solve this by adding an annotation to spark driver/executor pods. There are 2 cases.
1. for spark jobs submitted by spark-submit
We need to add CLI options spark.kubernetes.driver.label.[LabelName], something like spark.kubernetes.driver.label.applicationId (same for executor). And in our code. We can identify the appID first by the label, then by spark-app-selector.
2. for spark jobs submitted by spark-k8s-operator
We can do something like this. I believe options set here will be picked up by spark pods, see the logic: here
Huang Ting Yao, kmarton Please let me know your thoughts