Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24705

Spark.sql.adaptive.enabled=true is enabled and self-join query

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.2.1, 2.3.1
    • 2.4.0
    • SQL
    • None

    Description

      smilegator
      When loading data using jdbc and enabling spark.sql.adaptive.enabled=true , for example loading a tableA table, unexpected results can occur when you use the following query.

      For example:
      device_loc table comes from the jdbc data source
      select tv_a.imei
      from ( select a.imei,a.speed from device_loc a) tv_a
      inner join ( select a.imei,a.speed from device_loc a ) tv_b on tv_a.imei = tv_b.imei
      group by tv_a.imei

      When the cache tabel device_loc is executed before this query is executed, everything is fine,However, if you do not execute cache table, unexpected results will occur, resulting in failure to execute.

      Remarks:Attachment records the stack when the error occurred

      Attachments

        1. Error stack.txt
          5 kB
          cheng dai

        Activity

          People

            maropu Takeshi Yamamuro
            daic cheng dai
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: