[FLINK-15751] RocksDB Memory Management end-to-end test fails

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.10.0
• Fix Version/s: 1.10.0
• Component/s: Tests
• Labels: pull-request-available
• Assignee: Piotr Nowojski
• Reporter: Gary Yao
• Created: 24/Jan/20 14:18
• Updated: 27/Apr/21 18:06
• Resolved: 27/Jan/20 14:15

Description

RocksDB Memory Management end-to-end test fails due to non-whitelisted exceptions in the TaskManager log.

2020-01-24 15:02:50,957 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task Source: EventSource -> Timestamps/Watermarks (1/2) (c39cb61c33b9ddc6470432536d031df5).
2020-01-24 15:02:50,999 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: EventSource -> Timestamps/Watermarks (1/2) (c39cb61c33b9ddc6470432536d031df5) switched from RUNNING to CANCELING.
2020-01-24 15:02:50,999 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Source: EventSource -> Timestamps/Watermarks (1/2) (c39cb61c33b9ddc6470432536d031df5).
2020-01-24 15:02:51,005 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task Source: EventSource -> Timestamps/Watermarks (2/2) (b9cbd0aed214bb91ce4c4fb0fa6473a6).
2020-01-24 15:02:51,006 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: EventSource -> Timestamps/Watermarks (2/2) (b9cbd0aed214bb91ce4c4fb0fa6473a6) switched from RUNNING to CANCELING.
2020-01-24 15:02:51,006 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Source: EventSource -> Timestamps/Watermarks (2/2) (b9cbd0aed214bb91ce4c4fb0fa6473a6).
2020-01-24 15:02:51,035 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: EventSource -> Timestamps/Watermarks (1/2) (c39cb61c33b9ddc6470432536d031df5) switched from CANCELING to CANCELED.
2020-01-24 15:02:51,035 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Source: EventSource -> Timestamps/Watermarks (1/2) (c39cb61c33b9ddc6470432536d031df5).
2020-01-24 15:02:51,048 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: EventSource -> Timestamps/Watermarks (2/2) (b9cbd0aed214bb91ce4c4fb0fa6473a6) switched from CANCELING to CANCELED.
2020-01-24 15:02:51,048 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Source: EventSource -> Timestamps/Watermarks (2/2) (b9cbd0aed214bb91ce4c4fb0fa6473a6).
2020-01-24 15:02:51,048 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task ValueStateMapper (1/2) (1e835500a59643555fafad3a353c5577).
2020-01-24 15:02:51,048 INFO  org.apache.flink.runtime.taskmanager.Task                     - ValueStateMapper (1/2) (1e835500a59643555fafad3a353c5577) switched from RUNNING to CANCELING.
2020-01-24 15:02:51,048 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code ValueStateMapper (1/2) (1e835500a59643555fafad3a353c5577).
2020-01-24 15:02:51,050 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Source: EventSource -> Timestamps/Watermarks (2/2) (b9cbd0aed214bb91ce4c4fb0fa6473a6) [CANCELED]
2020-01-24 15:02:51,050 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Source: EventSource -> Timestamps/Watermarks (1/2) (c39cb61c33b9ddc6470432536d031df5) [CANCELED]
2020-01-24 15:02:51,057 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task ValueStateMapper (2/2) (5898050cf4efb76df5163a2c10770c16).
2020-01-24 15:02:51,057 INFO  org.apache.flink.runtime.taskmanager.Task                     - ValueStateMapper (2/2) (5898050cf4efb76df5163a2c10770c16) switched from RUNNING to CANCELING.
2020-01-24 15:02:51,057 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code ValueStateMapper (2/2) (5898050cf4efb76df5163a2c10770c16).
2020-01-24 15:02:51,058 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task ListStateMapper (1/2) (ebf0ef0b803fde1fa64a86b647542130).
2020-01-24 15:02:51,059 INFO  org.apache.flink.runtime.taskmanager.Task                     - ListStateMapper (1/2) (ebf0ef0b803fde1fa64a86b647542130) switched from RUNNING to CANCELING.
2020-01-24 15:02:51,059 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code ListStateMapper (1/2) (ebf0ef0b803fde1fa64a86b647542130).
2020-01-24 15:02:51,061 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task ListStateMapper (2/2) (af48a33505745e0596e3586b20486b7c).
2020-01-24 15:02:51,061 INFO  org.apache.flink.runtime.taskmanager.Task                     - ListStateMapper (2/2) (af48a33505745e0596e3586b20486b7c) switched from RUNNING to CANCELING.
2020-01-24 15:02:51,061 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code ListStateMapper (2/2) (af48a33505745e0596e3586b20486b7c).
2020-01-24 15:02:51,062 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task MapStateMapper (1/2) (46e94e505ab20710adf458ec861e2967).
2020-01-24 15:02:51,062 INFO  org.apache.flink.runtime.taskmanager.Task                     - MapStateMapper (1/2) (46e94e505ab20710adf458ec861e2967) switched from RUNNING to CANCELING.
2020-01-24 15:02:51,062 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code MapStateMapper (1/2) (46e94e505ab20710adf458ec861e2967).
2020-01-24 15:02:51,063 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task MapStateMapper (2/2) (767769894b0a27d10c8e7be76d29a1f2).
2020-01-24 15:02:51,065 INFO  org.apache.flink.runtime.taskmanager.Task                     - MapStateMapper (2/2) (767769894b0a27d10c8e7be76d29a1f2) switched from RUNNING to CANCELING.
2020-01-24 15:02:51,065 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code MapStateMapper (2/2) (767769894b0a27d10c8e7be76d29a1f2).
2020-01-24 15:02:51,066 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task TumblingWindowOperator (1/2) (8b171385ccd972f3840c503fb6ac6fdb).
2020-01-24 15:02:51,066 INFO  org.apache.flink.runtime.taskmanager.Task                     - TumblingWindowOperator (1/2) (8b171385ccd972f3840c503fb6ac6fdb) switched from RUNNING to CANCELING.
2020-01-24 15:02:51,066 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code TumblingWindowOperator (1/2) (8b171385ccd972f3840c503fb6ac6fdb).
2020-01-24 15:02:51,068 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task TumblingWindowOperator (2/2) (df6566baa3f1390be435b08bc97fdafa).
2020-01-24 15:02:51,068 INFO  org.apache.flink.runtime.taskmanager.Task                     - TumblingWindowOperator (2/2) (df6566baa3f1390be435b08bc97fdafa) switched from RUNNING to CANCELING.
2020-01-24 15:02:51,068 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code TumblingWindowOperator (2/2) (df6566baa3f1390be435b08bc97fdafa).
2020-01-24 15:02:51,071 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state CANCELED to JobManager for task Source: EventSource -> Timestamps/Watermarks c39cb61c33b9ddc6470432536d031df5.
2020-01-24 15:02:51,072 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state CANCELED to JobManager for task Source: EventSource -> Timestamps/Watermarks b9cbd0aed214bb91ce4c4fb0fa6473a6.
2020-01-24 15:02:51,101 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - Error while canceling task.
org.apache.flink.runtime.execution.CancelTaskException: Consumed partition PipelinedSubpartitionView(index: 1) of ResultPartition 30f8f32ccd788c5115aa536e557ca523@c39cb61c33b9ddc6470432536d031df5 has been released.
	at org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.getNextBuffer(LocalInputChannel.java:190)
	at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.waitAndGetNextData(SingleInputGate.java:509)
	at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:487)
	at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.pollNext(SingleInputGate.java:475)
	at org.apache.flink.runtime.taskmanager.InputGateWithMetrics.pollNext(InputGateWithMetrics.java:75)
	at org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:125)
	at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:133)
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:69)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:311)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:187)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:487)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:470)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:702)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:527)
	at java.lang.Thread.run(Thread.java:748)

Issue Links

• links to: GitHub Pull Request #10943