Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10875

Add `toTableWithTimestamp` method in `DataStreamConversions`

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: 1.7.3
    • Component/s: Table SQL / API
    • Labels:
      None
    • Release Note:
      Hide
      Did not find a better way.
      close JIRA first
      Show
      Did not find a better way. close JIRA first

      Description

      Currently we convert a `DataStream` to a `Table` by  `DataStreamConversions#toTable`, e.g.:

      // Without TimeAttribute
      ...
      val stream = env.fromCollection(...)
      val tab = stream.toTable(tEnv, 'a, 'b, 'c)
      val result = tab.select('a, 'b)
      ....
      
      // With TimeAttribute
      ...
      val stream = env.fromCollection(...).assignTimestampsAndWatermarks(...)
      val tab = stream.toTable(tEnv, 'a, 'b, 'c, 'ts.rowtime)
      val result = tab.window(Session withGap 5.milli on 'ts as 'w)
      ...

      I think the fieldNames parameter in the `toTable` method is reasonable in the conversion without the time attribute, because the fieldNames will actually correspond to the fields of the physical table, but when applied to the conversion with the time attribute, the time attribute column is silently added to the table. This feeling is very Magical, so I recommend adding a method that allows the user to display the time attribute added to the physical table: `toTableWithTimestamp`, which is automatically named to the time attribute column named by user input and TimeCharacteristic, eg:

      env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
      ...
      val table = stream.toTableWithTimestamp(tEnv, 'count, 'size, 'name, 'ts)
        .window(Tumble over 2.rows on 'ts as 'w)
      ...
      

      In the example above the flink will mark `ts` ad a `RowtimeAttribute`.

      What do you think ? 

        Attachments

          Activity

            People

            • Assignee:
              sunjincheng121 sunjincheng
              Reporter:
              sunjincheng121 sunjincheng
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: