Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-26938

HybridSource recovery from savepoint fails When flink parallelism is greater than the number of Kafka partitions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.14.0
    • None
    • API / DataStream
    • None
    • Flink 1.14.0

    Description

      HybridSource recovery from savepoint fails When flink parallelism is greater than the number of Kafka partitions

      First test

      Flink job before savePoint
          flink parallelism =16
          kafka partition=3
      Flink after savePoint
      case 1:
          flink parallelism =16
          kafka partition=3

      HybridSource recovery from savepoint fails 

      case 2:
          flink parallelism =3
          kafka partition=3
      HybridSource recovery from savepoint  successful

      case 3:
          flink parallelism =8
          kafka partition=3
      HybridSource recovery from savepoint fails  the same NullPointerException: Source for index=0 not available

      case 4:
          flink parallelism =4
          kafka partition=3
      HybridSource recovery from savepoint fails  the same NullPointerException: Source for index=0 not available

      case 5:
          flink parallelism =1
          kafka partition=3
      HybridSource recovery from savepoint  successful

      Second test

      Flink job before savePoint
          flink parallelism =3
          kafka partition=3
      Flink after savePoint
      case 1:
          flink parallelism =3
          kafka partition=3
      HybridSource recovery from savepoint  successful

      case 2:
          flink parallelism =1
          kafka partition=3
      HybridSource recovery from savepoint  successful

      case 3:
          flink parallelism =4
          kafka partition=3
      HybridSource recovery from savepoint fails  the same NullPointerException: Source for index=0 not available

      Specific code see the attached test code HybridSourceTest

       

      Attachments

        1. image-2022-03-31-13-36-45-686.png
          216 kB
          wenbao
        2. HybridSourceTest.java
          5 kB
          wenbao

        Activity

          People

            nicholasjiang Nicholas Jiang
            wenbao wenbao
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated: