This is an umbrella issue for enabling fine grained resource management in Flink.
Fine grained resource management is a big topic that requires long term efforts. There are many issues to be addressed and designing decisions to be made, some of which may not be resolved in short time. Here we propose our design and implementation plan for the upcoming release 1.9, as well as our thoughts and ideas on the long term road map on this topic.
A practical short term target is to enable fine grained resource management for batch sql jobs only in the upcoming Flink 1.9. This is necessary for batch operators added from blink to achieve good performance.
Please find detailed design and implementation plan in attached docs. Any comment and feedback are welcomed and appreciated.