Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8814

Control over the extension of part files created by BucketingSink

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.5.0
    • Component/s: Connectors / Common
    • Labels:
      None

      Description

      BucketingSink creates files with the following pattern

      partPrefix + "-" + subtaskIndex + "-" + bucketState.partCounter

      When using checkpointing you have no control over the extension of the final files generated. This is incovenient when you are for instance writing files in the avro format because

      1. Hue will not be able to render the files as avro See this file
      2. Spark avro will not be able to read the files unless you set a special property. See this ticket

      It would be good if we had the ability to customize the extension of the files created

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jelmer1 Jelmer Kuperus
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: