Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.10.0
Description
In runtime, a null slot sharing group means the vertex should be run in an individual slot which is not shared with other vertices.
However, at the moment, there is no way to set the slot sharing group of a vertex to null in production. This is because StreamGraphGenerator would always assign a non-null slot sharing group to an operator(StreamNode), and that would be the group of the generated JobVertex. (In the case of DataSet, all vertices are in the "default" group.)
Moreover, currently users are already able to force an operator to run in an individual slot by specify a different slot sharing group key to that operator.
So looks that supporting null slot sharing groups in runtime does not make much sense. And it's making things more complex, with codes like @Nullable annotations for slot sharing group, task required resources calculations, and single task slot allocation.
To make runtime code base and future development simpler, I'd propose to introduce the assumption that task slot sharing group should not be null.
The detailed proposal is:
1. In StreamingJobGraphGenerator, we check to ensure the slot sharing group key of a StreamNode is non-null, so that we always set a non-null SlotSharingGroup to a JobVertex
2. Remove all the @Nullable annotations for SlotSharingGroup and SlotSharingGroupID in several classes
3. Always use the slot sharing group resources as the physical slot resources for a task
4. Remove SchedulerImp#allocateSingleSlot and the related code paths
Attachments
Issue Links
- links to