Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-4329

Fix Streaming File Source Timestamps/Watermarks Handling

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0, 1.1.3
    • Component/s: Streaming Connectors
    • Labels:
      None

      Description

      The ContinuousFileReaderOperator does not correctly deal with watermarks, i.e. they are just passed through. This means that when the ContinuousFileMonitoringFunction closes and emits a Long.MAX_VALUE that watermark can "overtake" the records that are to be emitted in the ContinuousFileReaderOperator. Together with the new "allowed lateness" setting in window operator this can lead to elements being dropped as late.

      Also, ContinuousFileReaderOperator does not correctly assign ingestion timestamps since it is not technically a source but looks like one to the user.

        Attachments

          Activity

            People

            • Assignee:
              kkl0u Kostas Kloudas
              Reporter:
              aljoscha Aljoscha Krettek
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: