Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6427

BucketingSink does not sync file length in case of cancel

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.2.1, 1.3.0
    • Fix Version/s: 1.3.0
    • Component/s: Streaming Connectors
    • Labels:
      None

      Description

      This is from a discussion on the user mailing lists: https://lists.apache.org/thread.html/917dbbbd7c8f48b33e7b470fa9ed382df61cf43cb5deb46becebf4b5@%3Cuser.flink.apache.org%3E

      A possible solution is to add this to StreamWriterBase.hflushOrSync():

      if (os.getWrappedStream() instanceof DFSOutputStream) {
          ((DFSOutputStream) os.getWrappedStream()).hsync(EnumSet.of(HdfsDataOutputStream.SyncFlag.UPDATE_LENGTH));
      }
      

        Attachments

          Activity

            People

            • Assignee:
              aljoscha Aljoscha Krettek
              Reporter:
              aljoscha Aljoscha Krettek
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: