[SPARK-25348] Data source for binary files

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Weichen Xu
• Reporter: Xiangrui Meng
• Created: 05/Sep/18 19:11
• Updated: 28/Apr/19 15:56
• Resolved: 16/Apr/19 22:42

Description

It would be useful to have a data source implementation for binary files, which can be used to build features to load images, audio, and videos.

Microsoft has an implementation at https://github.com/Azure/mmlspark/tree/master/src/io/binary. It would be great if we can merge it into Spark main repo.

cc: mhamilton and imatiach

Proposed API:

Format name: "binaryFile"

Schema:

• content: BinaryType
• status (following Hadoop FIleStatus):
  • path: StringType
  • modificationTime: Timestamp
  • length: LongType (size limit 2GB)

Options:

• pathGlobFilter: only include files with path matching the glob pattern

Input partition size can be controlled by common SQL confs: maxPartitionBytes and openCostInBytes

Issue Links

• blocks: SPARK-27534 Do not load `content` column in binary data source if it is not selected
• blocks: SPARK-27472 Docuement binary file data source in Spark user guide
• blocks: SPARK-27473 Support filter push down for status fields in binary file data source
• relates to: SPARK-22666 Spark datasource for image format
• links to: GitHub Pull Request #24354