Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22839

Refactor Kubernetes code for configuring driver/executor pods to use consistent and cleaner abstraction

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 3.0.0
    • Component/s: Kubernetes
    • Labels:
      None

      Description

      As discussed in https://github.com/apache/spark/pull/19954, the current code for configuring the driver pod vs the code for configuring the executor pods are not using the same abstraction. Besides that, the current code leaves a lot to be desired in terms of the level and cleaness of abstraction. For example, the current code is passing around many pieces of information around different class hierarchies, which makes code review and maintenance challenging. We need some thorough refactoring of the current code to achieve better, cleaner, and consistent abstraction.

        Attachments

          Activity

            People

            • Assignee:
              mcheah Matt Cheah
              Reporter:
              liyinan926 Yinan Li
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: