[SPARK-26767] Filter on a dropDuplicates dataframe gives inconsistency result

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: SQL
• Labels: bulk-closed

Environment

To repeat the problem,

(1) create a csv file with records holding same values for a subset of columns (e.g. colA, colB, colC).

(2) read the csv file as a spark dataframe and then use dropDuplicates to dedup the subset of columns (i.e. dropDuplicates(["colA", "colB", "colC"]))

(3) select the resulting dataframe with where clause. (i.e. df.where("colA = 'A' and colB='B' and colG='G' and colH='H').show(100,False))

 

=> When (3) is rerun, it gives different number of resulting rows.

• Assignee: Unassigned
• Reporter: Jeffrey
• Created: 29/Jan/19 10:38
• Updated: 12/Dec/22 18:10
• Resolved: 25/May/21 01:41

Description

To repeat the problem,

(1) create a csv file with records holding same values for a subset of columns (e.g. colA, colB, colC).

(2) read the csv file as a spark dataframe and then use dropDuplicates to dedup the subset of columns (i.e. dropDuplicates(["colA", "colB", "colC"]))

(3) select the resulting dataframe with where clause. (i.e. df.where("colA = 'A' and colB='B' and colG='G' and colH='H').show(100,False))

 

=> When (3) is rerun, it gives different number of resulting rows.

Issue Links

• is related to: SPARK-27213 Unexpected results when filter is used after distinct