[SPARK-46862] Incorrect count() of a dataframe loaded from CSV datasource

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0, 3.5.1, 3.4.3
• Component/s: SQL
• Labels: correctness pull-request-available
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 25/Jan/24 14:14
• Updated: 27/Feb/24 03:25
• Resolved: 26/Jan/24 08:03

Description

The example below portraits the issue:

>>> df=spark.read.option("multiline", "true").option("header", "true").option("escape", '"').csv("es-939111-data.csv")
>>> df.count()
4
>>> df.cache()
DataFrame[jobID: string, Name: string, City: string, Active: string]
>>> df.count()
5

Issue Links

• links to: GitHub Pull Request #44872
• links to: GitHub Pull Request #44910