[PIG-4585] Use newAPIHadoopRDD instead of newAPIHadoopFile

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: spark-branch
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Mohit Sabharwal
• Reporter: Mohit Sabharwal
• Created: 03/Jun/15 22:52
• Updated: 21/Jun/17 09:18
• Resolved: 11/Jun/15 13:16

Description

LoadConverter currently uses SparkContext.newAPIHadoopFile which won't work for non-filesystem based input sources, like HBase.

newAPIHadoopFile assumes a FileInputFormat and attempts to verify this in the constructor, which fails for HBaseTableInputFormat (which is not a FileInputFormat)

  NewFileInputFormat.setInputPaths(job, path)

Issue Links

• links to: review board