Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12214

Spark to provide an API to save user-defined metadata as part of Sequence File header

    Details

      Description

      There is requirement to save "user-defined" Metadata as part of Sequence File "Header" using Spark.

      To write a User-defined metadata as part of Sequence File using regular Hadoop API's I pass the metadata object to SequenceFile.Writer constructor which when creates a SequenceFile ensures the metadata is part of the sequence file header.

      Currently Spark's JavaPairRDD Api provides methods to save an RDD to an SequenceFile format, but I don't see any API which can either give the SequenceFile.writer or a method where in I can pass the user-defined metadata so as to be written as part of sequence file header.

      The enhancement request an API to implement the same

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              npishe Naveen Pishe
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: