Hadoop Common
  1. Hadoop Common
  2. HADOOP-732

SequenceFile's header should allow to store metadata in the form of key/value pairs

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: io
    • Labels:
      None

      Description

      The sequence file currently stores a fixed list of metadata attributes, such as key/value class names,
      compression method, etc. To make sequence file more self descriptable, it should allow to store a list of key/value pairs. One particular attribute of interest is to indicate whether the key/value classes are actually hadoop record classes,
      if so, store the DDls for the records. This way, we may create tools to extract DDl from a sequence file and
      then generate necessary classes. It also make it possible to provide an interpretive version of Hadoop record.
      This way, even in the situation where Hadoop or the application does not have the necessary classes,
      a sequence file of Hadoop records can be read and deserialized "interpretively".

      1. seqFileMetadata.patch.2
        24 kB
        Runping Qi
      2. seqFileMetadata.patch
        32 kB
        Runping Qi

        Activity

        Hide
        Owen O'Malley added a comment -

        Would the meta-data key/value pairs be required to be strings? That would simplify everything. It isn't clear what the right API for this would be.

        Show
        Owen O'Malley added a comment - Would the meta-data key/value pairs be required to be strings? That would simplify everything. It isn't clear what the right API for this would be.
        Hide
        Runping Qi added a comment -

        Strings sgould be fine.

        Show
        Runping Qi added a comment - Strings sgould be fine.
        Hide
        Runping Qi added a comment -

        Attached is a patch for this issue.

        SequenceFile has a new header — a TreeMap<Text, Text> object wrapped in a class, Metadata, implementing Writable interface. To accomodate this, the version number is bumped up to 6.

        The Reader class has a new member variable for the metadata. A method is also added for returning the metadata object. The new code can read the files of old versions.

        New constructors of various Writer classes are added to take a metadata object as their last parameter. New createWriter static functions with metadata as the last
        parameter are also introduced. They are all backward compatible. A new unit test is added to TestSequenceFile for testing writing/reading sequence files with metadata.
        All unit tests passed.

        Show
        Runping Qi added a comment - Attached is a patch for this issue. SequenceFile has a new header — a TreeMap<Text, Text> object wrapped in a class, Metadata, implementing Writable interface. To accomodate this, the version number is bumped up to 6. The Reader class has a new member variable for the metadata. A method is also added for returning the metadata object. The new code can read the files of old versions. New constructors of various Writer classes are added to take a metadata object as their last parameter. New createWriter static functions with metadata as the last parameter are also introduced. They are all backward compatible. A new unit test is added to TestSequenceFile for testing writing/reading sequence files with metadata. All unit tests passed.
        Hide
        Runping Qi added a comment -

        Reattach the patch

        Show
        Runping Qi added a comment - Reattach the patch
        Hide
        Hadoop QA added a comment -

        +1, because http://issues.apache.org/jira/secure/attachment/12349495/seqFileMetadata.patch applied and successfully tested against trunk revision r499156.

        Show
        Hadoop QA added a comment - +1, because http://issues.apache.org/jira/secure/attachment/12349495/seqFileMetadata.patch applied and successfully tested against trunk revision r499156.
        Hide
        Doug Cutting added a comment -

        This looks good. My only concern is that it adds yet more SequenceFile.createWriter() signatures. Until HADOOP-938 is resolved, I'd prefer this only add a single new createWriter() signature, one that includes all possible options, including this new option.

        Show
        Doug Cutting added a comment - This looks good. My only concern is that it adds yet more SequenceFile.createWriter() signatures. Until HADOOP-938 is resolved, I'd prefer this only add a single new createWriter() signature, one that includes all possible options, including this new option.
        Hide
        Runping Qi added a comment -

        Knock off a few createWriter methods.
        seqFileMetadata.patch.2 contains the new patch.

        Show
        Runping Qi added a comment - Knock off a few createWriter methods. seqFileMetadata.patch.2 contains the new patch.
        Hide
        Doug Cutting added a comment -

        I just committed this. Thanks, Runping!

        Show
        Doug Cutting added a comment - I just committed this. Thanks, Runping!

          People

          • Assignee:
            Runping Qi
            Reporter:
            Runping Qi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development