[SPARK-13792] Limit logging of bad records

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: SQL
• Target Version/s: 2.0.0
• Assignee: Reynold Xin
• Reporter: Hossein Falaki
• Created: 10/Mar/16 02:19
• Updated: 21/Jun/16 05:27
• Resolved: 21/Jun/16 04:46

Description

Currently in PERMISSIVE and DROPMALFORMED modes we log any record that is going to be ignored. This can generate a lot of logs with large datasets. A better idea is to log the first record and the number of subsequent records for each partition.

Issue Links

• links to: [Github] Pull Request #12173 (maropu)
• links to: [Github] Pull Request #13795 (rxin)
• links to: [Github] Pull Request #13800 (rxin)