Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36749

The count result of the dimension table filed changes as `exector.memory` changes.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.1.3
    • None
    • Input/Output
    • None

    Description

      hi~, every one!

      Here‘s a very strange questions!!! 

      The meaning of this sql is count the number of the specified columns in each table after joining the table。as follows:

       

      SELECT cast(COUNT(DISTINCT tps.prod_siginst_id) AS STRING) AS siginst_cnt,
      cast(COUNT(DISTINCT qpl.list_id) AS STRING) AS list_cnt,
      cast(count(DISTINCT if(tb.brand_source=1,tps.prod_siginst_id,NULL)) AS STRING) AS domestic_siginst_cnt,
      cast(count(DISTINCT if(tb.brand_source=2,tps.prod_siginst_id,NULL)) AS STRING) AS import_siginst_cnt,
      cast(count(DISTINCT if(qpl.list_name NOT LIKE'%un_normal%',tps.prod_siginst_id,NULL)) AS STRING) AS standard_cnt,
      cast(count(DISTINCT if(qpl.list_name LIKE'%un_normal%',tps.prod_siginst_id,NULL)) AS STRING) AS nostandard_cnt
      FROM tableA tbi
      LEFT JOIN tableB tps ON tbi.prod_inst_id=tps.prod_inst_id
      LEFT JOIN tableC qpl ON tbi.prod_type_id=qpl.list_id
      LEFT JOIN tableD ON tps.brand_id=tb.brand_id
      WHERE tbi.prod_status=1
      AND tbi.prod_sell_status=1
      AND tb.recommend_flag=1;

       

      and the phenomenon of the question is if i add memory for executor, the count result of the tableC field(list_id,list_name) will changes as well. until the executor‘s memory is big enough, the result doesn't change.

       

      TableC is a dimensional table and the amount of data is fixed.

       

      In my opinions, this job should failed rather than output an incorrect count result if executor is insufficient memory.

      Could you please help me check whether this is a bug of spark itself or something wrong with my sql writing?

       

      here is log of this job.

       

      Attachments

        1. corrent_result.log
          113 kB
          LanYang
        2. wrong_result.log
          114 kB
          LanYang

        Activity

          People

            Unassigned Unassigned
            Lanyoung LanYang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: