Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20573

Add a configuration item to flinksql's 'with' configuration syntax to filter on exception data

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.11.2
    • Fix Version/s: None
    • Component/s: Table SQL / API
    • Labels:
      None

      Description

      Currently, after defining the event time and watermark in flink sql, if abnormal data is received, such as test data, and it happens that the event time of that test data is very large, it will cause the watermark of the whole stream to become large, so that any normal data coming in afterwards will not be processed by the stream.
      Is it possible to set a configuration item in 'with' to identify what kind of data is abnormal data that can be automatically filtered, such as event time more than half an hour or more than 2 hours from the current time, will be automatically filtered by the stream; or the event time of an event with a certain value range in the event stream id will not affect the stream's watermark generation.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cxiiiiiii Xiao Cai

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment