Uploaded image for project: 'Apache Apex Core'
  1. Apache Apex Core
  2. APEXCORE-494

Window id of downstream operator is not moving after dynamic partition of upstream operator.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.5.0
    • None
    • None

    Description

      Created the following application to reproduce this issue.
      Application has the following DAG:
      KafkaSinglePortStringInputOperator(Input) -> ConsoleOutputOperator(Output)

      I launched the application with below configuration:
      Kafka topic created with single partition and replication factor as 1.
      Partition Strategy: ONE_TO_ONE

      Launched the application successfully. After some time, I increased the topic partitions to 2. After re-partition, the window of down stream operator is not moving. By looking into the app Physical DAG, it looks like there is an issue in construction of Physical DAG after re-partition.

      Attachments

        1. PhysicalOperatorAfterKafkaRepartition.png
          47 kB
          Chaitanya Chebolu
        2. PhysicalDAGAfterKafkaRepartition.png
          14 kB
          Chaitanya Chebolu
        3. PhysicalDAGBeforeKafkaRepartition.png
          5 kB
          Chaitanya Chebolu

        Activity

          People

            chaithu Chaitanya Chebolu
            chaithu Chaitanya Chebolu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: