[SPARK-28082] Add a note to DROPMALFORMED mode of CSV for column pruning

• Type: Documentation
• Status: Resolved
• Priority: Trivial
• Resolution: Duplicate
• Affects Version/s: 3.0.0
• Component/s: Documentation, PySpark, SQL
• Assignee: Unassigned
• Reporter: L. C. Hsieh
• Created: 17/Jun/19 15:25
• Updated: 12/Dec/22 18:10
• Resolved: 18/Jun/19 00:32

Description

This is inspired by SPARK-28058.

When using DROPMALFORMED mode, corrupted records aren't dropped if malformed columns aren't read. This behavior is due to CSV parser column pruning. Current doc of DROPMALFORMED doesn't mention the effect of column pruning. Users will be confused by the fact that DROPMALFORMED mode doesn't work as expected.

Issue Links

• duplicates: SPARK-28058 Reading csv with DROPMALFORMED sometimes doesn't drop malformed records
• links to: [Github] Pull Request #24894 (viirya)
• links to: GitHub Pull Request #24894