[FLINK-7666] ContinuousFileReaderOperator swallows chained watermarks

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.3.2
• Fix Version/s: 1.4.0
• Component/s: Connectors / Common
• Assignee: Kostas Kloudas
• Reporter: Ufuk Celebi
• Created: 21/Sep/17 19:59
• Updated: 26/Oct/17 07:50
• Resolved: 26/Oct/17 07:50

Description

I use event time and read from a (finite) file. I assign watermarks right after the ContinuousFileReaderOperator with parallelism 1.

env
  .readFile(new TextInputFormat(...), ...)
  .setParallelism(1)
  .assignTimestampsAndWatermarks(...)
  .setParallelism(1)
  .map()...

The watermarks I assign never progress through the pipeline.

I can work around this by inserting a shuffle() after the file reader or starting a new chain at the assigner:

env
  .readFile(new TextInputFormat(...), ...)
  .setParallelism(1)
  .shuffle() 
  .assignTimestampsAndWatermarks(...)
  .setParallelism(1)
  .map()...

Issue Links

• links to: GitHub Pull Request #4900