Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12459

Watch does not properly advance the watermark by default

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.32.0
    • Component/s: sdk-java-core
    • Labels:
      None

      Description

      Assigning to Luke who has made substantial changes to this class most recently.

       

      It appears after investigation that when using Watch in the default configuration, the global watermark is not advanced properly, even though Watch documentation claims it should be (https://github.com/apache/beam/blob/8922c1cf23c093262af9e4570d69947a9a749506/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java#L118). The below is example code using Watch that will not advance the watermark, but should:

      ```
      Watch.growthOf(
      new PollFn<Integer, Integer>() {
          @Override
          public PollResult<Partition> apply(TopicPath element, Context c)

      {         return PollResult.incomplete(Instant.now(), List.of(0));     }

      })
      .withPollInterval(...)
      .withTerminationPerInput(Watch.Growth.never());
      ```

      I've been advised that changing the return statement to `return PollResult.incomplete(Instant.now(), List.of(0)).withWatermark(Instant.now());` will resolve this issue, but the `withWatermark` function is commented as "By default, the watermark for a particular input is computed from a poll result as "earliest timestamp of new elements in this poll result". It can also be set explicitly via {@link Growth.PollResult#withWatermark} if the {@link Growth.PollFn} can provide a more optimistic estimate.". The goal is not to provide a more optimistic estimate, but to allow any advancement at all. If withWatermark is needed to close windows, this function should be required (or at least more prominent in all example code).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lcwik Luke Cwik
                Reporter:
                dpcollins-google Daniel Collins
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 10m
                  2h 10m