Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-3089

State API Should Support Data Expiration (State TTL)

    XMLWordPrintableJSON

Details

    Description

      In some usecases (webanalytics) there is a need to have a state per visitor on a website (i.e. keyBy(sessionid) ).
      At some point the visitor simply leaves and no longer creates new events (so a special 'end of session' event will not occur).

      The only way to determine that a visitor has left is by choosing a timeout, like "After 30 minutes no events we consider the visitor 'gone'".

      Only after this (chosen) timeout has expired should we discard this state.

      In the Trigger part of Windows we can set a timer and close/discard this kind of information. But that introduces the buffering effect of the window (which in some scenarios is unwanted).

      What I would like is to be able to set a timeout on a specific state which I can update afterwards.

      This makes it possible to create a map function that assigns the right value and that discards the state automatically.

      Attachments

        Issue Links

          depends upon

          New Feature - A new feature of the product, which has yet to be developed. FLINK-12005 [State TTL] Event time support

          • Not a Priority -
          • Open

          New Feature - A new feature of the product, which has yet to be developed. FLINK-9510 State TTL API with cleanup on explicit access

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. FLINK-10471 State TTL cleanup using RocksDb compaction filter

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. FLINK-10473 State TTL incremental cleanup using Heap backend key iterator

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-9938 State TTL cleanup of full state snapshot upon checkpointing

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-12476 [State TTL] Consider setting a default background cleanup strategy in StateTtlConfig

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-14898 Enable background cleanup of state with TTL by default

          • Major - Major loss of function.
          • Closed
          is depended upon by

          New Feature - A new feature of the product, which has yet to be developed. FLINK-4428 Method map/flatMapWithState may need a eviction policy

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-3946 State API Should Support Data Expiration

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-16686 [State TTL] Make user class loader available in native RocksDB compaction thread

          • Not a Priority -
          • Open

          New Feature - A new feature of the product, which has yet to be developed. FLINK-18467 Document what can be reconfigured for state with TTL between job restarts

          • Not a Priority -
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-15606 Deprecate enable default background cleanup of state with TTL

          • Major - Major loss of function.
          • Closed
          mentioned in

          Page Loading...

          Activity

            People

              azagrebin Andrey Zagrebin
              nielsbasjes Niels Basjes
              Votes:
              9 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: