Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21413

TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend

    XMLWordPrintableJSON

    Details

      Description

      Take the #TtlMapState as an example,

       

      public Map<UK, TtlValue<UV>> getUnexpiredOrNull(@Nonnull Map<UK, TtlValue<UV>> ttlValue) {
              Map<UK, TtlValue<UV>> unexpired = new HashMap<>();
              TypeSerializer<TtlValue<UV>> valueSerializer =
                      ((MapSerializer<UK, TtlValue<UV>>) original.getValueSerializer()).getValueSerializer();
              for (Map.Entry<UK, TtlValue<UV>> e : ttlValue.entrySet()) {
                      if (!expired(e.getValue())) {
                              // we have to do the defensive copy to update the value
                              unexpired.put(e.getKey(), valueSerializer.copy(e.getValue()));
                      }
              }
              return ttlValue.size() == unexpired.size() ? ttlValue : unexpired;
      }
      

       

      The returned value will never be null and the #StateEntry will exists forever, which leads to memory leak if the key's range of the stream is very large. Below we can see that 20+ millison uncleared TtlStateMap could take up several GB memory.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wind_ljy Jiayi Liao
                Reporter:
                wind_ljy Jiayi Liao
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: