[SPARK-12782] reindex() columns in DataFrame

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Component/s: SQL
• Labels: bulk-closed feature pandas
• Assignee: Unassigned
• Reporter: Mahmoud Lababidi
• Created: 12/Jan/16 19:40
• Updated: 21/May/19 04:34
• Resolved: 21/May/19 04:34

Description

There is a nice feature in Pandas DataFrames that allows you to reindex columns. Example: a column has 2006-01-01, 2006-01-02, 2006-01-04. If you perform reindex() the missing 2006-01-03 will be filled in.

This seems necessary to implement in Spark.

http://pandas.pydata.org/pandas-docs/version/0.17.1/generated/pandas.DataFrame.reindex.html