[FLINK-16684] StreamingFileSink builder does not work with Scala

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.10.0
• Fix Version/s: 1.10.1, 1.11.0
• Component/s: API / Scala, Connectors / FileSystem
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 19/Mar/20 13:51
• Updated: 19/Mar/20 18:55
• Resolved: 19/Mar/20 16:39

Description

The StreamingFileSink builders don't work with Scala as they lose with every with* method more type information. For example, the following example does not compile:

val sink = StreamingFileSink.forRowFormat(new Path("s3a://123"), new Encoder[String] {
      override def encode(element: String, stream: OutputStream): Unit = ???
    }).withRollingPolicy(
      DefaultRollingPolicy.builder()
        .withRolloverInterval(TimeUnit.MINUTES.toMinutes(5))
        .withInactivityInterval(TimeUnit.MINUTES.toMinutes(5))
        .withMaxPartSize(128 * 1024 * 1024)
        .build()
    ).withBucketAssigner(
      new BucketAssigner[String, String] {
        override def getBucketId(element: String, context: BucketAssigner.Context): String = ???
        override def getSerializer: SimpleVersionedSerializer[String] = ???
      }
    ).build();

The problem seems to be that Scala does type inference slightly differently than Java. I believe that the unspecified RowFormatBuilder type cannot be properly resolved.

Issue Links

• Dependent: FLINK-16616 Drop BucketingSink
• links to: GitHub Pull Request #11454