[KAFKA-5070] org.apache.kafka.streams.errors.LockException: task [0_18] Failed to lock the state directory: /opt/rocksdb/pulse10/0_18

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.10.2.0
• Component/s: streams
• Environment: Linux Version
• Assignee: Matthias J. Sax
• Reporter: Dhana
• Created: 14/Apr/17 07:27
• Updated: 21/Aug/17 00:32
• Resolved: 21/Aug/17 00:32

Description

Notes: we run two instance of consumer in two difference machines/nodes.
we have 400 partitions. 200 stream threads/consumer, with 2 consumer.
We perform HA test(on rebalance - shutdown of one of the consumer/broker), we see this happening

Error:
2017-04-05 11:36:09.352 WARN StreamThread:1184 StreamThread-66 - Could not create task 0_115. Will retry.
org.apache.kafka.streams.errors.LockException: task [0_115] Failed to lock the state directory: /opt/rocksdb/pulse10/0_115
at org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:102)
at org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:73)
at org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:108)
at org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
at org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
at org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
at org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
at org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
at org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)

Issue Links

• duplicates: KAFKA-5167 streams task gets stuck after re-balance due to LockException
• is related to: KAFKA-5167 streams task gets stuck after re-balance due to LockException
• relates to: KAFKA-5562 Do streams state directory cleanup on a single thread