Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5562

Do streams state directory cleanup on a single thread

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.11.0.1
    • None
    • None

    Description

      Currently in streams we clean up old state directories every so often (as defined by state.cleanup.delay.ms). However, every StreamThread runs the cleanup, which is both unnecessary and can potentially lead to race conditions.

      It would be better to perform the state cleanup on a single thread and only when the KafkaStreams instance is in a running state.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-4890 State directory being deleted when another thread holds the lock

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-5070 org.apache.kafka.streams.errors.LockException: task [0_18] Failed to lock the state directory: /opt/rocksdb/pulse10/0_18

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #3516

          Web Link GitHub Pull Request #3654

          Activity

            People

              damianguy Damian Guy
              damianguy Damian Guy
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: