[SPARK-42426] insertInto fails when the column names are different from the table columns

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Assignee: Takuya Ueshin
• Reporter: Takuya Ueshin
• Created: 13/Feb/23 22:57
• Updated: 16/Feb/23 00:29
• Resolved: 16/Feb/23 00:29

Description

File "/.../python/pyspark/sql/connect/readwriter.py", line 518, in pyspark.sql.connect.readwriter.DataFrameWriter.insertInto
Failed example:
    df.selectExpr("age AS col1", "name AS col2").write.insertInto("tblA")
Exception raised:
    Traceback (most recent call last):
      File "/.../lib/python3.9/doctest.py", line 1334, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.readwriter.DataFrameWriter.insertInto[3]>", line 1, in <module>
        df.selectExpr("age AS col1", "name AS col2").write.insertInto("tblA")
      File "/.../python/pyspark/sql/connect/readwriter.py", line 477, in insertInto
        self.saveAsTable(tableName)
      File "/.../python/pyspark/sql/connect/readwriter.py", line 495, in saveAsTable
        self._spark.client.execute_command(self._write.command(self._spark.client))
      File "/.../python/pyspark/sql/connect/client.py", line 553, in execute_command
        self._execute(req)
      File "/.../python/pyspark/sql/connect/client.py", line 648, in _execute
        self._handle_error(rpc_error)
      File "/.../python/pyspark/sql/connect/client.py", line 718, in _handle_error
        raise convert_exception(info, status.message) from None
    pyspark.errors.exceptions.connect.AnalysisException: Cannot resolve 'age' given input columns: [col1, col2].

Issue Links

• links to: [Github] Pull Request #40024 (ueshin)
• links to: [Github] Pull Request #40024 (ueshin)