[FLINK-22081] Entropy key not resolved if flink-s3-fs-hadoop is added as a plugin

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10.3, 1.11.3, 1.12.2, 1.13.0
• Fix Version/s: 1.11.4, 1.13.0, 1.12.3
• Component/s: FileSystems
• Labels: pull-request-available
• Assignee: Chen Qin
• Reporter: Chen Qin
• Created: 01/Apr/21 04:48
• Updated: 22/Jun/21 13:55
• Resolved: 08/Apr/21 07:33

Description

Using flink 1.11.2

I added the flink-s3-fs-hadoop jar in plugins dir but I am seeing the checkpoints paths like s3://my_app/_ENTROPY_/app_name-staging/flink/checkpoints/e10f47968ae74934bd833108d2272419/chk-3071 which means the entropy injection key is not being resolved. After some debugging I found that in the EntropyInjector we check if the given fileSystem is of type ClassLoaderFixingFileSystem and if so we check if the filesysystem is of type SafetyNetWrapperFileSystem as well as it's delegate }}but don't check for {{ClassLoaderFixingFileSystem directly in getEntorpyFs method which would be the type if S3 file system dependencies are added as a plugin.

 

Repro steps: 

Flink 1.11.2 with flink-s3-fs-hadoop as plugin and turn on entropy injection key entropy
observe checkpoint dir with entropy marker not removed.
s3a://xxx/dev/checkpoints/entropy/xenon/event-stream-splitter/jobid/chk-5/  
compare to removed when running Flink 1.9.1
s3a://xxx/dev/checkpoints/xenon/event-stream-splitter/jobid/chk-5/  

Add some logging to getEntropyFs, observe it return null because passed in parameter is not SafetyNetWrapperFileSystem but ClassLoaderFixingFileSystem

Apply patch, build release and run same job, resolved issue as attachment shows

 

 

Issue Links

• is a clone of: FLINK-17359 Entropy key is not resolved if flink-s3-fs-hadoop is added as a plugin
• links to: GitHub Pull Request #15442