[SPARK-8118] Turn off noisy log output produced by Parquet 1.7.0

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 1.4.1, 1.5.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Cheng Lian
• Reporter: Cheng Lian
• Created: 05/Jun/15 00:49
• Updated: 21/May/19 04:17
• Resolved: 21/May/19 04:17

Description

Parquet 1.7.0 renames package name to "org.apache.parquet", need to adjust ParquetRelation.enableLogForwarding accordingly to avoid noisy log output.

A better approach than simply muting these log lines is to redirect Parquet logs via SLF4J, so that we can handle them consistently. In general these logs are very useful. Esp. when used to diagnosing Parquet memory issue and filter push-down.

Issue Links

• is duplicated by: SPARK-9631 Giant pile of parquet log when trying to read local data
• is duplicated by: SPARK-13735 Log for parquet relation reading files is too verbose
• is related to: SPARK-16221 Redirect Parquet JUL logger via SLF4J for WRITE operations
• is related to: SPARK-13735 Log for parquet relation reading files is too verbose
• links to: [Github] Pull Request #6670 (liancheng)
• links to: [Github] Pull Request #8196 (liancheng)