[SPARK-24442] Add configuration parameter to adjust the numbers of records and the characters per row before truncation when a user runs.show()

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.2.0, 2.3.0
• Component/s: SQL
• Labels: bulk-closed
• Flags: Patch
• Assignee: Unassigned
• Reporter: Andrew K Long
• Created: 31/May/18 16:34
• Updated: 12/Dec/22 18:11
• Resolved: 08/Oct/19 05:43

Description

Currently the number of characters displayed when a user runs the .show() function on a data frame is hard coded. The current default is too small when used with wider console widths.  This fix will add two parameters.

 

parameter: "spark.show.default.number.of.rows" default: "20"

parameter: "spark.show.default.truncate.characters.per.column" default: "20"

 

This change will be backwords compatible and will not break any existing functionality nor change the default display characteristics.

Issue Links

• links to: [Github] Pull Request #22162 (AndrewKL)
• links to: GitHub Pull Request #26404