Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-48817

MultiInsert is split to multiple sql executions, resulting in no exchange reuse

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 4.0.0, 3.5.1
    • 4.0.0
    • SQL

    Description

      MultiInsert is split to multiple sql executions, resulting in no exchange reuse.

       

      Reproduce sql:

      create table wangzhen_t1(c1 int);
      create table wangzhen_t2(c1 int);
      create table wangzhen_t3(c1 int);
      
      insert into wangzhen_t1 values (1), (2), (3);
      
      from (select /*+ REPARTITION(3) */ c1 from wangzhen_t1)
      insert overwrite table wangzhen_t2 select c1
      insert overwrite table wangzhen_t3 select c1; 

       

      In Spark 3.1, there is only one SQL execution and there is a reuse exchange.

       

      However, in Spark 3.5, it was split to multiple executions and there was no ReuseExchange.

       

      Attachments

        1. image-2024-07-05-14-59-35-340.png
          27 kB
          Zhen Wang
        2. image-2024-07-05-14-59-55-291.png
          24 kB
          Zhen Wang
        3. image-2024-07-05-15-00-01-805.png
          15 kB
          Zhen Wang
        4. image-2024-07-05-15-00-09-181.png
          16 kB
          Zhen Wang
        5. image-2024-07-05-15-00-17-693.png
          5 kB
          Zhen Wang
        6. image-2024-07-05-16-42-01-973.png
          29 kB
          Zhen Wang
        7. image-2024-07-05-16-42-17-817.png
          34 kB
          Zhen Wang
        8. image-2024-07-05-16-42-27-033.png
          36 kB
          Zhen Wang
        9. image-2024-07-05-16-42-34-738.png
          36 kB
          Zhen Wang
        10. image-2024-07-05-16-42-46-500.png
          4 kB
          Zhen Wang

        Issue Links

          Activity

            People

              wforget Zhen Wang
              wforget Zhen Wang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: