[SPARK-12214] Spark to provide an API to save user-defined metadata as part of Sequence File header

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.3.0, 1.4.0, 1.5.0
• Component/s: Spark Core
• Labels: Sequence bulk-closed header metadata sequencefile
• Assignee: Unassigned
• Reporter: Naveen Pishe
• Created: 08/Dec/15 21:11
• Updated: 21/May/19 04:36
• Resolved: 21/May/19 04:36

Description

There is requirement to save "user-defined" Metadata as part of Sequence File "Header" using Spark.

To write a User-defined metadata as part of Sequence File using regular Hadoop API's I pass the metadata object to SequenceFile.Writer constructor which when creates a SequenceFile ensures the metadata is part of the sequence file header.

Currently Spark's JavaPairRDD Api provides methods to save an RDD to an SequenceFile format, but I don't see any API which can either give the SequenceFile.writer or a method where in I can pass the user-defined metadata so as to be written as part of sequence file header.

The enhancement request an API to implement the same