[SPARK-32515] Distinct Function Weird Bug

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.4.6
• Component/s: SQL

Environment

Window 10 and Mac, both have the same issues.

Using Scala version 2.11.12

Python 3.6.10

java version "1.8.0_261"

• Assignee: Unassigned
• Reporter: Jayce Jiang
• Created: 02/Aug/20 01:20
• Updated: 12/Dec/22 18:10
• Resolved: 07/Aug/20 01:48

Description

A weird spark display and counting error. When I was loading in my CSV file into spark and trying to do check all distinct value from a column inside of a dataframe. Everything I try in spark resulted in a wrong answer. But if I convert my spark dataframe into pandas dataframe, it works. Please help. This bug only happens in this one CSV file, all my other CSV files work properly. Here are the pictures.