Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.3.0
-
None
Description
As discussed in https://github.com/apache/spark/pull/19954, the current code for configuring the driver pod vs the code for configuring the executor pods are not using the same abstraction. Besides that, the current code leaves a lot to be desired in terms of the level and cleaness of abstraction. For example, the current code is passing around many pieces of information around different class hierarchies, which makes code review and maintenance challenging. We need some thorough refactoring of the current code to achieve better, cleaner, and consistent abstraction.
