[FLINK-26351] After scaling a flink task running on k8s, the flink web ui graph always shows the parallelism of the first deployment.

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.15.0
• Component/s: API / Core
• Labels: auto-deprioritized-major pull-request-available
• Assignee: Unassigned
• Reporter: qiunan
• Created: 24/Feb/22 12:56
• Updated: 22/Aug/23 10:35

Description

In the code，flink web ui graph data from under method.

AdaptiveScheduler.requestJob()

 @Override
    public ExecutionGraphInfo requestJob() {
        return new ExecutionGraphInfo(state.getJob(), exceptionHistory.toArrayList());
   } 

This executionGraphInfo is task restart build and restore to state.

You can see the code, the parallelism recalculate and copy jobGraph to reset.

AdaptiveScheduler.createExecutionGraphWithAvailableResourcesAsync().

vertexParallelism = determineParallelism(slotAllocator);
JobGraph adjustedJobGraph = jobInformation.copyJobGraph();

for (JobVertex vertex : adjustedJobGraph.getVertices()) {
    JobVertexID id = vertex.getID();

    // use the determined "available parallelism" to use
    // the resources we have access to
    vertex.setParallelism(vertexParallelism.getParallelism(id));
}

But in the restoreState copy jobGraph again, so the jobGraph parallelism always deployed for the first time.

AdaptiveScheduler.createExecutionGraphAndRestoreState(VertexParallelismStore adjustedParallelismStore)  

private ExecutionGraph createExecutionGraphAndRestoreState(
        VertexParallelismStore adjustedParallelismStore) throws Exception {
    return executionGraphFactory.createAndRestoreExecutionGraph(
            jobInformation.copyJobGraph(),
            completedCheckpointStore,
            checkpointsCleaner,
            checkpointIdCounter,
            TaskDeploymentDescriptorFactory.PartitionLocationConstraint.MUST_BE_KNOWN,
            initializationTimestamp,
            vertexAttemptNumberStore,
            adjustedParallelismStore,
            deploymentTimeMetrics,
            LOG);
} 

Issue Links

• links to: GitHub Pull Request #18915