Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8406

BucketingSink does not detect hadoop file systems

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.5.0
    • Fix Version/s: 1.4.1, 1.5.0
    • Component/s: FileSystems
    • Labels:
      None

      Description

      In BucketingSink#createHadoopFileSystem one can find this piece of code:

      
      final org.apache.flink.core.fs.FileSystem flinkFs =
       	org.apache.flink.core.fs.FileSystem.get(path.toUri());
      final FileSystem hadoopFs = (flinkFs instanceof HadoopFileSystem)
      	? ((HadoopFileSystem) flinkFs).getHadoopFileSystem()
      	: null;
      

      FileSystem#get() wraps the created FileSystem in a SafetyNetWrapperFileSystem, resulting in the instanceof check to categorically fail.

      We may want to replace the get() call with getUnguardedFileSystem(). We should also look for other occurrences of similar instanceof checks.

      According to a thread on the mailing list this causes the BucketingSink to be unusable. http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/BucketingSink-broken-in-flink-1-4-0-td17710.html

        Attachments

          Activity

            People

            • Assignee:
              sewen Stephan Ewen
              Reporter:
              chesnay Chesnay Schepler

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment