[FLINK-12296] Data loss silently in RocksDBStateBackend when more than one operator(has states) chained in a single task

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.6.3, 1.6.4, 1.7.2, 1.8.0
• Fix Version/s: 1.7.3, 1.8.1, 1.9.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: Congxian Qiu
• Reporter: Congxian Qiu
• Created: 23/Apr/19 06:53
• Updated: 29/May/19 04:31
• Resolved: 29/May/19 02:42

Description

As the mail list said[1], there may be a problem when more than one operator chained in a single task, and all the operators have states, we'll encounter data loss silently problem.

Currently, the local directory we used is like below

../local_state_root_1/allocation_id/job_id/vertex_id_subtask_idx/chk_1/(state),

 

if more than one operator chained in a single task, and all the operators have states, then all the operators will share the same local directory(because the vertext_id is the same), this will lead a data loss problem. 

 

The path generation logic is below:

// LocalRecoveryDirectoryProviderImpl.java

@Override
public File subtaskSpecificCheckpointDirectory(long checkpointId) {
   return new File(subtaskBaseDirectory(checkpointId), checkpointDirString(checkpointId));
}


@VisibleForTesting
String subtaskDirString() {
   return Paths.get("jid_" + jobID, "vtx_" + jobVertexID + "_sti_" + subtaskIndex).toString();
}

@VisibleForTesting
String checkpointDirString(long checkpointId) {
   return "chk_" + checkpointId;
}

[1] http://mail-archives.apache.org/mod_mbox/flink-user/201904.mbox/[e-mail]%3E

Issue Links

• links to: GitHub Pull Request #8263