[SPARK-32699] Add percentage of missingness to df.summary()

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 3.0.0
• Component/s: ML
• Assignee: Unassigned
• Reporter: Chengyin Eng
• Created: 25/Aug/20 14:42
• Updated: 12/Dec/22 18:11
• Resolved: 26/Aug/20 04:27

Description

 

In df.summary(), we are returned counts of non-nulls for each column. It would be really helpful to have a percentage of non-nulls, since percentage of missingness is often an indicator for data scientists to discard the columns.