Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-12005

[State TTL] Event time support

    XMLWordPrintableJSON

Details

    Description

      The event time is opted for in StateTtlConfig by setting TtlTimeCharacteristic.EventTime.

      To enable event time support, the updated watermark needs to be passed to the state backend, shared with TTL state wrappers and additional cleanup strategies (snapshot transformers and compaction filter).

      Event time provider

      Additional implementation of TtlTimeProvider, which holds current watermark, needs to be passed to the state backend at the moment of its creation in StreamTaskStateInitializerImpl. There several ways to update watermark in this implementation of TtlTimeProvider:

      • in InternalTimeServiceManager.advanceWatermark explicitly
      • InternalTimeServiceManager/InternalTimerServiceImpl could be refactored to use shared EventTimeService which holds current updatable watermark and wrapped by TtlTimeProvider

      The TTL state wrapping factory should create TTL state wrappers and snapshot transformers with TtlTimeProvider selected by TtlTimeCharacteristic.

      RocksDB TTL compaction filter

      The RocksDB TTL compaction filter factory needs to get selected TtlTimeProvider when it gets configured. There are two ways:

      • make it volatile and settable in RocksDbTtlCompactFiltersManager.TimeProviderWrapper, track it in RocksDbTtlCompactFiltersManager along with FlinkCompactionFilterFactory to configure later before configuring FlinkCompactionFilterFactory.
      • Move FlinkCompactionFilter.TimeProvider from FlinkCompactionFilterFactory to ConfigHolder and set selected TtlTimeProvider with the Config.

      The second option does not use volatile variable and should be more performant but needs changing RocksDB java client and either releasing new version FRocksDB or Flink RocksDB extensions.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              azagrebin Andrey Zagrebin
              Votes:
              1 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated: