Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-18209

Reduce binary metadata synchronization time for CDC through Kafka

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • None
    • extensions
    • Docs Required, Release Notes Required

    Description

      Currently, when CDC through Kafka is used there are possible delays in replication between clusters when KafkaToIgniteCdcStreamerApplier tries to update binary metadata and marshaller mappings.

      Delays caused by calls of KafkaConsumer#poll in KafkaToIgniteMetadataUpdater#updateMetadata , when meta topic is empty:

      1. When first KafkaToIgniteCdcStreamerApplier meets META_UPDATE_MARKER it calls KafkaToIgniteMetadataUpdater#updateMetadata which in turn calls KafkaConsumer#poll, which returns immediately [1] when data is present in metadata topic. If there are few binary types and mappings to update, first KafkaToIgniteCdcStreamerApplier will consume all entries from metadata topic.
      2. KafkaToIgniteCdcStreamerApplier consequently call KafkaToIgniteMetadataUpdater#updateMetadata for each partition with meta update marker. All further consequent calls will wait for kafkaReqTimeout.
      3. Also there is a bottleneck, when multiple applier threads tries to update metadata and call synchronized method KafkaToIgniteMetadataUpdater#updateMetadata, because KafkaToIgniteMetadataUpdater is shared between applier threads.
      4. Because META_UPDATE_MARKER is sent twice to each Kafka partition of event topic from every node: firstly, in case of type mappings updates, secondly, in case of binary types update there are possible delays up to (clusterSize x topicPartitions x 2 - 1) x kafkaReqTimeout.
      5. Data updates are blocked for Kafka partitions with unprocessed update markers.
      6. For example for default timeout and 16 Kafka partitions last partition will be consumed after 1.5 minutes in case of two one-node clusters.

      Links:

      1. https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll-java.time.Duration-

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue IGNITE-18267 Set up parallel feature for CDC build configuration

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-19369 Metadata topic offset must be stored only after commit

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is part of

          Improvement - An improvement or enhancement to an existing feature or task. IGNITE-14449 Add binary meta change event to CDCCosumer

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue IGNITE-18216 Duplicated CDC test runs on TeamCity

          • Major - Major loss of function.
          • Resolved

          Test - A new unit, integration or system test. IGNITE-18549 Fix timeouts of CdcKafkaReplicationTest#afterTest

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Test - A new unit, integration or system test. IGNITE-18552 Fix TopicExistsException in CdcKafkaReplicationTest

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Test - A new unit, integration or system test. IGNITE-19063 Fix incorrect behaviour of MemorizingAppender with MutableLogEvent

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Test - A new unit, integration or system test. IGNITE-19067 MemorizingAppender improvements

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Test - A new unit, integration or system test. IGNITE-18512 Add ACL to CdcKafkaReplicationTest

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Test - A new unit, integration or system test. IGNITE-18548 Fix CdcIgniteToIgniteReplicationTest#testActivePassiveReplication flakiness

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Test - A new unit, integration or system test. IGNITE-18995 CDC test failures on BinaryMetadataFileStore.restoreMetadata

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. IGNITE-19102 CDC: Use manual assignment for consumer in KafkatoIgniteMetadataUpdater

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. IGNITE-18515 CDC: add documentation about metadata replication

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. IGNITE-18992 CDC: Use single partition topic for metadata updates

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. IGNITE-18574 CDC: add documentation about Kafka request timeouts

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Task - A task that needs to be done. IGNITE-18251 Refactor testing of ignite-extensions

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub pull request #202

          Activity

            People

              shishkovilja Ilya Shishkov
              shishkovilja Ilya Shishkov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: