[SPARK-43239] Remove null_counts from info()

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Pandas API on Spark
• Assignee: Bjørn Jørgensen
• Reporter: Bjørn Jørgensen
• Created: 22/Apr/23 20:13
• Updated: 24/Apr/23 00:14
• Resolved: 24/Apr/23 00:14

Description

df.info() is broken now.
It prints

TypeError Traceback (most recent call last)
Cell In[12], line 1
----> 1 F05.info()

File /opt/spark/python/pyspark/pandas/frame.py:12167, in DataFrame.info(self, verbose, buf, max_cols, null_counts)
12163 count_func = self.count
12164 self.count = ( # type: ignore[assignment]
12165 lambda: count_func()._to_pandas() # type: ignore[assignment, misc, union-attr]
12166 )
> 12167 return pd.DataFrame.info(
12168 self, # type: ignore[arg-type]
12169 verbose=verbose,
12170 buf=buf,
12171 max_cols=max_cols,
12172 memory_usage=False,
12173 null_counts=null_counts,
12174 )
12175 finally:
12176 del self._data

TypeError: DataFrame.info() got an unexpected keyword argument 'null_counts'