Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9727

Flaky system test StreamsEOSTest.test_failure_and_recovery

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: streams, system tests
    • Labels:
      None

      Description

      Hits no lock available exceptions sometime after task revive:

       

      [2020-03-13 05:40:50,224] ERROR stream-thread [EosTest-46de8ee5-82a1-4bd4-af23-f4acd8515f0f-StreamThread-1] Encountered the following exception during processing and the thread is going to shut down:  (org.apache.kafka.streams.processor.internals.StreamThread)

      org.apache.kafka.streams.errors.ProcessorStateException: Error opening store KSTREAM-AGGREGATE-STATE-STORE-0000000003 at location /mnt/streams/EosTest/0_0/rocksdb/KSTREAM-AGGREGATE-STATE-STORE-0000000003

              at org.apache.kafka.streams.state.internals.RocksDBTimestampedStore.openRocksDB(RocksDBTimestampedStore.java:87)

              at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:191)

              at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:230)

              at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)

              at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:44)

              at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)

              at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$init$0(MeteredKeyValueStore.java:101)

              at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:806)

              at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:101)

              at org.apache.kafka.streams.processor.internals.StateManagerUtil.registerStateStores(StateManagerUtil.java:81)

              at org.apache.kafka.streams.processor.internals.StandbyTask.initializeIfNeeded(StandbyTask.java:86)

              at org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:275)

              at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:583)

              at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:498)

              at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:472)

      Caused by: org.rocksdb.RocksDBException: lock : /mnt/streams/EosTest/0_0/rocksdb/KSTREAM-AGGREGATE-STATE-STORE-0000000003/LOCK: No locks available

              at org.rocksdb.RocksDB.open(Native Method)

              at org.rocksdb.RocksDB.open(RocksDB.java:286)

              at org.apache.kafka.streams.state.internals.RocksDBTimestampedStore.openRocksDB(RocksDBTimestampedStore.java:75)

              ... 14 more

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bchen225242 Boyang Chen
                Reporter:
                bchen225242 Boyang Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: