Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-32486

FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

    XMLWordPrintableJSON

Details

    • Hide
      We introduce runtime filter for batch jobs in 1.18, which is designed to improve join performance. It will dynamically generate filter conditions for certain Join queries at runtime to reduce the amount of scanned or shuffled data, avoid unnecessary I/O and network transmission, and speed up the query. Its working principle is building a filter(e.g. bloom filter) based on the data on the small table side(build side) first, then pass this filter to the large table side(probe side) to filter the irrelevant data on it, this can reduce the data reaching the join and improve performance.
      Show
      We introduce runtime filter for batch jobs in 1.18, which is designed to improve join performance. It will dynamically generate filter conditions for certain Join queries at runtime to reduce the amount of scanned or shuffled data, avoid unnecessary I/O and network transmission, and speed up the query. Its working principle is building a filter(e.g. bloom filter) based on the data on the small table side(build side) first, then pass this filter to the large table side(probe side) to filter the irrelevant data on it, this can reduce the data reaching the join and improve performance.

    Description

      This is an umbrella ticket forĀ FLIP-324

      Attachments

        Issue Links

          Activity

            People

              wanglijie Lijie Wang
              wanglijie Lijie Wang
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: