Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-22356

Filesystem/Hive partition file is not committed when watermark is applied on rowtime of TIMESTAMP_LTZ type

    XMLWordPrintableJSON

    Details

      Description

      set execution.checkpointing.interval = 10s;
      set table.local-time-zone = 'Asia/Shanghai';
      
      create table mysource (
        ms bigint,
        ts as to_timestamp_ltz(ms, 3),
        watermark for ts as ts - interval '0.001' second
      ) with (
        'connector' = 'socket',
        'format' = 'json',
        'hostname' = '127.0.0.1',
        'port' = '9999'
      );
      
      CREATE TABLE fs_table2 (
          ms bigint,
        dt STRING,
        `hour` STRING,
        `mm` string
      ) PARTITIONED BY (dt, `hour`, `mm`) WITH (
        'connector'='filesystem',
        'path'='/Users/wuchong/Downloads/fs_table2',
        'format'='csv',
        'sink.partition-commit.delay'='1min',
        'sink.partition-commit.policy.kind'='success-file',
        'sink.rolling-policy.rollover-interval' = '30s',
        'sink.rolling-policy.check-interval' = '30s',
        'sink.partition-commit.trigger'='partition-time',
        'partition.time-extractor.timestamp-pattern' = '$dt $hour:$mm:00'
      );
      
      insert into  fs_table2
      SELECT ms,
      DATE_FORMAT(ts, 'yyyy-MM-dd'), DATE_FORMAT(ts, 'HH'), DATE_FORMAT(ts, 'mm')
      FROM mysource;
      

      Enther some data in socket:

      > nc -lk 9999
      {"ms": 1618839600000}
      {"ms": 1618839600123}
      {"ms": 1618839600456}
      {"ms": 1618839600789}
      {"ms": 1618839660000}
      {"ms": 1618839660123}
      {"ms": 1618839660456}
      {"ms": 1618839660789}
      {"ms": 1618839720000}
      {"ms": 1618839780000}
      {"ms": 1618839840000}
      {"ms": 1618839900000}
      {"ms": 1618839960000}
      {"ms": 1618840020000}
      

      However, all the files are not committed (not _SUCCESS file):

      ➜  hour=21 tree
      .
      ├── mm=40
      │   └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-0
      ├── mm=41
      │   └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-1
      ├── mm=42
      │   └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-2
      ├── mm=43
      │   └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-3
      ├── mm=44
      │   └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-4
      ├── mm=45
      │   └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-5
      ├── mm=46
      │   └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-6
      └── mm=47
          └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-7
      
      8 directories, 8 files
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Leonard Xu Leonard Xu
                Reporter:
                jark Jark Wu
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: