Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-23668 Check Log Pollution for 1.14 release
  3. FLINK-23908

Hint user about possibly non-deterministic key

    XMLWordPrintableJSON

    Details

      Description

      If user has non-deterministic hashCode and equals implementation, he ends up with "confusing" (if you don't now details about Flink internals) exception.

      RocksDBStateBackend:

      Caused by: java.lang.IllegalArgumentException: Key group 0 is not in KeyGroupRange{startKeyGroup=32, endKeyGroup=63}.
      at org.apache.flink.runtime.state.KeyGroupRangeOffsets.computeKeyGroupIndex(KeyGroupRangeOffsets.java:142)
      at org.apache.flink.runtime.state.KeyGroupRangeOffsets.setKeyGroupOffset(KeyGroupRangeOffsets.java:104)
      at org.apache.flink.contrib.streaming.state.snapshot.RocksFullSnapshotStrategy$SnapshotAsynchronousPartCallable.writeKVStateData(RocksFullSnapshotStrategy.java:319)
      at org.apache.flink.contrib.streaming.state.snapshot.RocksFullSnapshotStrategy$SnapshotAsynchronousPartCallable.writeSnapshotToOutputStream(RocksFullSnapshotStrategy.java:261)
      

      HeapStateBackend:

      2021-08-16 16:28:54	java.lang.NullPointerException: null
      2021-08-16 16:28:54		at org.apache.flink.runtime.state.heap.StateTable.put(StateTable.java:351) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.runtime.state.heap.StateTable.put(StateTable.java:159) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.runtime.state.heap.HeapListState.add(HeapListState.java:98) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:422) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:191) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:204) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:174) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:396) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:617) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:581) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570) ~[flink-dist_2.11-1.12.3.jar:1.12.3]
      2021-08-16 16:28:54		at java.lang.Thread.run(Unknown Source) ~[?:?]
      

        Attachments

          Activity

            People

            • Assignee:
              dmvk David Morávek
              Reporter:
              dmvk David Morávek
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: