Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-27399

Pulsar connector didn't set start consuming position correctly

    XMLWordPrintableJSON

Details

    • Hide
      A number of breaking changes were made to the Pulsar Connector cursor APIs:

      - CursorPosition#seekPosition() has been removed.
      - StartCursor#seekPosition() has been removed.
      - StopCursor#shouldStop now returns a StopCondition instead of a boolean.
      Show
      A number of breaking changes were made to the Pulsar Connector cursor APIs: - CursorPosition#seekPosition() has been removed. - StartCursor#seekPosition() has been removed. - StopCursor#shouldStop now returns a StopCondition instead of a boolean.

    Description

      The Pulsar connector didn't use the consuming position from the checkpoint. They just commit the position to Pulsar after the checkpoint is complete. And the connector starts to consume messages from Pulsar directly by the offset stored on the Pulsar subscription.

      This causes the test could be failed in some situations. The start cursor (position on Pulsar) would be reset to the wrong position, which caused the results didn't match the desired records.

      How to fix this issue

      Change the start position seeking mechanism from Pulsar consumer API to Pulsar admin API. Don't reset the start position when the topic has a subscription.

      This issue fixes

      1. FLINK-23944
      2. FLINK-24872
      3. FLINK-25815
      4. FLINK-25884
      5. FLINK-26177
      6. FLINK-26721
      7. FLINK-27833

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue FLINK-26721 PulsarSourceITCase.testSavepoint failed on azure pipeline

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Sub-task - The sub-task of the issue FLINK-23944 PulsarSourceITCase.testTaskManagerFailure is instable

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Sub-task - The sub-task of the issue FLINK-24872 PulsarSourceITCase.testMultipleSplits failed on AZP

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Sub-task - The sub-task of the issue FLINK-25815 PulsarSourceITCase.testTaskManagerFailure failed on azure due to timeout

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-26177 PulsarSourceITCase.testScaleDown fails with timeout

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-27833 PulsarSourceITCase.testTaskManagerFailure failed with AssertionError

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-25884 PulsarSourceITCase.testTaskManagerFailure failed on azure

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-28972 Add methods of StartCursor and StopCursor to align the Java

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #19972

          Web Link PR for 1.14 branch

          Web Link PR for 1.15 branch

          Activity

            People

              syhily Yufan Sheng
              syhily Yufan Sheng
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: