[SPARK-41473] Implement `functions.format_number`

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect, PySpark
• Assignee: Ruifeng Zheng
• Reporter: Xinrong Meng
• Created: 09/Dec/22 22:49
• Updated: 27/Dec/22 06:17
• Resolved: 27/Dec/22 06:16

Description

Currently, `functions.format_number` has data type mismatch issue as shown below:

>>> from pyspark.sql.connect.session import SparkSession as RemoteSparkSession
>>> cspark = RemoteSparkSession.builder.remote().getOrCreate()
>>> from pyspark.sql.connect import functions as CF
>>> cspark.createDataFrame([(5,)], ['a']).select(CF.format_number('a', 4))
22/12/09 14:46:52 ERROR SparkConnectService: Error analyzing plan.
org.apache.spark.sql.AnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "format_number(a, 4)" due to data type mismatch: Parameter 2 requires the ("INT" or "STRING") type, however "4" has the type "BIGINT".;
'Project [unresolvedalias(format_number(a#2L, 4), None)]
+- Project [0#0L AS a#2L]
   +- LocalRelation [0#0L]
...

The jira aims to fix the data type mismatch issue and implement `functions.format_number`.

Issue Links

• links to: [Github] Pull Request #39229 (zhengruifeng)
• links to: [Github] Pull Request #39229 (zhengruifeng)