[SPARK-28522] Pass dynamic parameters to custom file input format

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.3.0
• Component/s: Java API
• Assignee: Unassigned
• Reporter: Ayan Mukherjee
• Created: 26/Jul/19 09:15
• Updated: 12/Dec/22 18:10
• Resolved: 27/Jul/19 06:42

Description

We have  developed a custom file input format and calling it in pyspark using newAPIHadoopFile option. It appears there is no option to pass parameters dynamically to the custom format.

 

rdd2 = sc.newAPIHadoopFile("/abcd/efgh/i1.txt", "com.test1.TEST2.TESTInputFormat", "org.apache.hadoop.io.Text", "org.apache.hadoop.io.NullWritable")