Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21145

Flink Temporal Join Hive optimization

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 1.12.0
    • None
    • Connectors / Hive
    • None

    Description

      When flink temporal join hive dimension table, the latest partition data will be loaded into task memory in full, which will lead to high memory overhead. In fact, sometimes the latest full data is not required. You can add options like options in future versions. Is the dimension table data filtered?
      For example, select * from dim /'streaming-source.partition.include' ='latest' condition='fild1=ab'/ filter the latest partition data as long as fild1=ab

      Attachments

        Issue Links

          is duplicated by

          Sub-task - The sub-task of the issue FLINK-18779 Support the SupportsFilterPushDown interface for LookupTableSource

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              HideOnBush HideOnBush
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: