[SPARK-36707] Support to specify index type and name in pandas API on Spark

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: PySpark
• Labels: release-notes
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 10/Sep/21 06:34
• Updated: 12/Dec/22 17:51
• Resolved: 07/Oct/21 09:36

Description

See https://koalas.readthedocs.io/en/latest/user_guide/typehints.html.

pandas API on Spark currently there's no way to specify the index type and name in the output when you apply an arbitrary function, which forces to create the default index:

>>> def transform(pdf) -> pd.DataFrame["id": int, "A": int]:
...     pdf['A'] = pdf.id + 1
...     return pdf
...
>>> ps.range(5).koalas.apply_batch(transform)

   id   A
0   0   1
1   1   2
2   2   3
3   3   4
4   4   5

We should have a way to specify the index.

Sub-Tasks

• Support numpy.typing for annotating ArrayType: Resolved
• Support new syntax for specifying index type and name: Resolved
• Support new syntax in function apply APIs: Resolved
• Support multi-index in new syntax: Resolved
• Document new syntax for specifying index type: Resolved
• Error when list of data type tuples has len = 1: Resolved