[SPARK-42022] createDataFrame should autogenerate missing column names

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 12/Jan/23 06:03
• Updated: 08/Mar/23 00:36
• Resolved: 08/Mar/23 00:36

Description

pyspark/sql/tests/test_types.py:233 (TypesParityTests.test_infer_schema_not_enough_names)
['col1', '_2'] != ['col1']

Expected :['col1']
Actual   :['col1', '_2']
<Click to see difference>

self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests testMethod=test_infer_schema_not_enough_names>

    def test_infer_schema_not_enough_names(self):
        df = self.spark.createDataFrame([["a", "b"]], ["col1"])
>       self.assertEqual(df.columns, ["col1", "_2"])

../test_types.py:236: AssertionError

Issue Links

• links to: [Github] Pull Request #40310 (ueshin)