Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.10
    • None

    Description

      Right now we use spark-app-selector as the label for the applicationID for Spark pods: https://github.com/apache/incubator-yunikorn-k8shim/blob/master/pkg/common/utils/utils.go#L87

      When linking the Spark pod group to the CRD we use the Application ID, but for the CRD we use namespace-name convention as ApplicationID and for the spark pod group we use the spark-app-selector what will result in having two different applications internally: one for the CRD and one for the Spark POD group.

      I think we should change this label to something else, what we can modify easily without any side effects.

      Attachments

        1. spark-driver.PNG
          78 kB
          huang ting yao

        Issue Links

          Activity

            wwei Weiwei Yang added a comment - - edited

            hi kmarton

            Looks like we can solve this by adding an annotation to spark driver/executor pods. There are 2 cases.
            1. for spark jobs submitted by spark-submit
            We need to add CLI options spark.kubernetes.driver.label.[LabelName], something like spark.kubernetes.driver.label.applicationId (same for executor). And in our code. We can identify the appID first by the label, then by spark-app-selector.

            2. for spark jobs submitted by spark-k8s-operator
            We can do something like this. I believe options set here will be picked up by spark pods, see the logic: here

            Huang Ting Yao, kmarton Please let me know your thoughts

            wwei Weiwei Yang added a comment - - edited hi kmarton Looks like we can solve this by adding an annotation to spark driver/executor pods. There are 2 cases. 1. for spark jobs submitted by spark-submit We need to add CLI options spark.kubernetes.driver.label. [LabelName] , something like spark.kubernetes.driver.label.applicationId (same for executor). And in our code . We can identify the appID first by the label, then by spark-app-selector . 2. for spark jobs submitted by spark-k8s-operator We can do something like this . I believe options set here will be picked up by spark pods, see the logic: here Huang Ting Yao , kmarton Please let me know your thoughts
            tingyao TingYao Huang added a comment - - edited

            wwei Agree with case 2. The driver pod can already get application name in Annotations part(not in labels).

            In our code, we can move annotation part before labels part, and add constant YunikoenApplication/Name in contstants.go.

            Let it check the pod is belong to yunikorn CRD or not first, if the annotation isn't match then check label part.

            And return appid in namespace + name format.

             

            tingyao TingYao Huang added a comment - - edited wwei  Agree with case 2. The driver pod can already get application name in Annotations part(not in labels). In our code , we can move annotation part before labels part, and add constant  YunikoenApplication/Name  in contstants.go. Let it check the pod is belong to yunikorn CRD or not first, if the annotation isn't match then check label part. And return appid in namespace + name format.  
            wwei Weiwei Yang added a comment -

            hi Huang Ting Yao

            OK. Can we use "yunikorn.apache.org/app-id" as the annotation name? We need to comply with the naming convention of K8s annotation fields.

            wwei Weiwei Yang added a comment - hi Huang Ting Yao OK. Can we use "yunikorn.apache.org/app-id" as the annotation name? We need to comply with the naming convention of K8s annotation fields.
            tingyao TingYao Huang added a comment - - edited

            hi wwei 

            sure. Just need to modify the yunikorn part in spark operator.

            tingyao TingYao Huang added a comment - - edited hi wwei   sure. Just need to modify the yunikorn part in spark operator.
            wwei Weiwei Yang added a comment -

            Sounds great. Please try this approach and let me know if this can work or not.
            Thank you Huang Ting Yao, kmarton.

            wwei Weiwei Yang added a comment - Sounds great. Please try this approach and let me know if this can work or not. Thank you Huang Ting Yao , kmarton .
            kmarton Kinga Marton added a comment -

            I think it cam work. I will implement today the YK part.

            kmarton Kinga Marton added a comment - I think it cam work. I will implement today the YK part.
            kmarton Kinga Marton added a comment -

            I have one note for the part when we populate the application ID:

            • The application ID should be in the form of namespace-name, where both the namespace and the name must be the values of the app CRD, not the name of the spark pods.
            • when setting the name for the CRD, we have to make sure that the name does not have ''-", since we use it as separator.
            kmarton Kinga Marton added a comment - I have one note for the part when we populate the application ID: The application ID should be in the form of namespace-name , where both the namespace and the name must be the values of the app CRD, not the name of the spark pods. when setting the name for the CRD, we have to make sure that the name does not have ''-", since we use it as separator.
            wwei Weiwei Yang added a comment -

            hi kmarton

            I just realize there might be a problem. Spark operator supports 2 types of jobs, SparkApplication and ScheduledSparkApplication.
            The latter one can run job in a certain schedule, e.g @10minutes. This means, the ScheduledSparkApplication CRD will have a name, and every internal, it will create a Spark job with a different spark-app-id. If we generate the app-CRD based on ScheduledSparkApplication CRD, that will mean these jobs will be considered as 1 same app inside of YuniKorn. I guess we can live with that probably. Just bring this up in case we are missing anything in the design.

            wwei Weiwei Yang added a comment - hi kmarton I just realize there might be a problem. Spark operator supports 2 types of jobs, SparkApplication and ScheduledSparkApplication . The latter one can run job in a certain schedule, e.g @10minutes. This means, the ScheduledSparkApplication CRD will have a name, and every internal, it will create a Spark job with a different spark-app-id. If we generate the app-CRD based on ScheduledSparkApplication CRD, that will mean these jobs will be considered as 1 same app inside of YuniKorn. I guess we can live with that probably. Just bring this up in case we are missing anything in the design.
            kmarton Kinga Marton added a comment -

            Hi wwei from YuniKorn point of view, can it cause troubles while we are doing the scheduling of that jobs, if they will all belong to the same application?

            I think that since we are talking about one single spark application launching a bunch of jobs, it would be logic to handle it as one single YuniKorn application as well.

            kmarton Kinga Marton added a comment - Hi wwei  from YuniKorn point of view, can it cause troubles while we are doing the scheduling of that jobs, if they will all belong to the same application? I think that since we are talking about one single spark application launching a bunch of jobs, it would be logic to handle it as one single YuniKorn application as well.
            wwei Weiwei Yang added a comment -

            PR committed, thanks kmarton

            wwei Weiwei Yang added a comment - PR committed, thanks kmarton

            People

              kmarton Kinga Marton
              kmarton Kinga Marton
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: