Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-22081

Entropy key not resolved if flink-s3-fs-hadoop is added as a plugin

    XMLWordPrintableJSON

    Details

      Description

      Using flink 1.11.2

      I added the flink-s3-fs-hadoop jar in plugins dir but I am seeing the checkpoints paths like s3://my_app/_ENTROPY_/app_name-staging/flink/checkpoints/e10f47968ae74934bd833108d2272419/chk-3071 which means the entropy injection key is not being resolved. After some debugging I found that in the EntropyInjector we check if the given fileSystem is of type ClassLoaderFixingFileSystem and if so we check if the filesysystem is of type SafetyNetWrapperFileSystem as well as it's delegate }}but don't check for {{ClassLoaderFixingFileSystem directly in getEntorpyFs method which would be the type if S3 file system dependencies are added as a plugin.

       

      Repro steps: 

      Flink 1.11.2 with flink-s3-fs-hadoop as plugin and turn on entropy injection key entropy
      observe checkpoint dir with entropy marker not removed.
      s3a://xxx/dev/checkpoints/entropy/xenon/event-stream-splitter/jobid/chk-5/  
      compare to removed when running Flink 1.9.1
      s3a://xxx/dev/checkpoints/xenon/event-stream-splitter/jobid/chk-5/  

      Add some logging to getEntropyFs, observe it return null because passed in parameter is not SafetyNetWrapperFileSystem but ClassLoaderFixingFileSystem

      Apply patch, build release and run same job, resolved issue as attachment shows

       

       

        Attachments

        1. image (13).png
          194 kB
          Chen Qin

          Issue Links

            Activity

              People

              • Assignee:
                foxss Chen Qin
                Reporter:
                foxss Chen Qin
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: