Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-5066

Support hoodie source metaclient cache for flink planner

    XMLWordPrintableJSON

Details

    Description

      Change Logs

      Flink Table Planner will invoke `HoodieTableSource.copy()` when applying its `RelOptRule` such as
      `PushPartitionIntoTableSourceScanRule`, thus result in multiple meta client instantiation, which will
      affect the performance of starting job quickly.

              // apply push down
              DynamicTableSource dynamicTableSource = tableSourceTable.tableSource().copy();
              PartitionPushDownSpec partitionPushDownSpec =
                      new PartitionPushDownSpec(remainingPartitions);
              partitionPushDownSpec.apply(dynamicTableSource, SourceAbilityContext.from(scan));
      

      Here we promote that meta client might be cached to reused.

      Impact

      speed up flink sql job start by avoiding unnecessary meta client repeated creation.

      Attachments

        Issue Links

          Activity

            People

              chenshizhi Shizhi Chen
              chenshizhi Shizhi Chen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: