[SPARK-30792] Dataframe .limit() performance improvements

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.4.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Nathan Grand
• Created: 11/Feb/20 17:18
• Updated: 25/May/21 01:50
• Resolved: 25/May/21 01:38

Description

It seems that

.limit()

is much less efficient than it could be/one would expect when reading a large dataset from parquet:

val sample = spark.read.parquet("/Some/Large/Data.parquet").limit(1000)
// Do something with sample ...

This might take hours, depending on the size of the data.

By comparison,

spark.read.parquet("/Some/Large/Data.parquet").show(1000)

is essentially instant.