Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.16.1
Description
We have a limitation that if we create savepoints with an injected entropy, they are not relocatable (https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/#triggering-savepoints).
FLINK-25952 improves the check by inspecting both the FileSystem extending EntropyInjectingFileSystem and FlinkS3FileSystem#getEntropyInjectionKey not returning null. We can improve this further by checking the checkpoint path is indeed using the entropy injection key. Without that, the savepoint is not relocatable even if the state.savepoints.dir does not contain the entropy.
In our setting, we enable entropy injection by setting s3.entropy.key to
_ENTROPY_KEY_ and use the entropy key in the checkpoint path (for e.g. s3://mybuket/checkpoints/_ENTROPY_KEY_/myapp). However, in the savepoint path, we don't use the entropy key (for e.g. s3://mybuket/savepoints/myapp) because we want the savepoint to be relocatable. But the current logic still generates non-relocatable savepoint path just because the entropy injection key is non-null.