Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20197

FileSink fails with UnsupportedOperationException when using default values

    XMLWordPrintableJSON

    Details

      Description

      When using the default values for the new FileSink, it fails with

      Caused by: java.lang.UnsupportedOperationException: not supported
      	at org.apache.flink.connector.file.sink.writer.FileWriter$BucketerContext.currentProcessingTime(FileWriter.java:296) ~[flink-connector-files-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner.getBucketId(DateTimeBucketAssigner.java:111) ~[flink-file-sink-common-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner.getBucketId(DateTimeBucketAssigner.java:55) ~[flink-file-sink-common-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.connector.file.sink.writer.FileWriter.write(FileWriter.java:189) ~[flink-connector-files-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.operators.sink.AbstractSinkWriterOperator.processElement(AbstractSinkWriterOperator.java:80) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.api.operators.StreamGroupedReduceOperator.processElement(StreamGroupedReduceOperator.java:68) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:193) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:372) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:574) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:538) ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722) ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547) ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      	at java.lang.Thread.run(Thread.java:834) ~[?:?]
      

      The problem seems that the FileWriter.BucketerContext.currentProcessingTime throws an UnsupportedOperationException. This feels not correct. A job for reproducing the problem can be found here.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gaoyunhaii Yun Gao
                Reporter:
                trohrmann Till Rohrmann
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: