Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-22806

The folder /checkpoint/shared state of FLINKSQL is getting bigger and bigger

    XMLWordPrintableJSON

    Details

      Description

      I have added the parameter :
      1、tableEnv.getConfig().setIdleStateRetention: 3600 (one hour),
      2、state.checkpoints.num-retained:3
      3、sql:

      // demo:  
          select count(1),  LISTAGG(concat(m,n)) from tabeA group by a, b, time_minute
      
      //details:
      CREATE TABLE user_behavior (
         `request_ip` STRING,
         `request_time` BIGINT,
         `header` STRING ,
         // timestamp  converted to specific per minute   
         `t_min` as cast(`request_time`-(`request_time` + 28800000)%60000 as BIGINT),
         `ts` as TO_TIMESTAMP(FROM_UNIXTIME(`request_time`/1000-28800,'yyyy-MM-dd HH:mm:ss')),
         WATERMARK FOR `ts` AS `ts` - INTERVAL '60' MINUTE) 
      with (
         'connector' = 'kafka',
         ........ 
      );
      
      
      CREATE TABLE blackhole_table (
         `cnt` BIGINT,
         `lists` STRING
      ) WITH (
       'connector' = 'blackhole'
      );
      
      
      insert into blackhole_table 
      select 
          count(*) as cnt, 
          LISTAGG(concat(`request_ip`, `header`, cast(`request_time` as STRING))) as lists
      from user_behavior 
      group by `request_ip`,`header`,`t_min`;
      

      4、state.backend: rocksdb
      5、state.backend.incremental is true

      I set the checkpoint state for one hour, but the size of the folder directory /checkpoint/shared is still growing. I observed it for two days and guessed that there was expired data in the /checkpoint/shared folder that had not been cleared?

      What else can limit the growth of state?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              YUJIANBO YUJIANBO
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: