Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1669

Add support for columnar event serializer in HDFS

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Sinks+Sources

    Description

      Motivation:
      Columnar storage is preferred for better performance and compression for low-latency analytical workloads. Avro 1.7.2 supports column-major file format [1]
      and we can implement AbstractTrevniAvroEventSerializer (as like AbstractAvroEventSerializer). HDFSSink can have serializer type to store events in Trevni column-major file format.

      [1] http://avro.apache.org/docs/current/trevni/spec.html
      https://github.com/cutting/trevni

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-3585 Integrate Trevni as another columnar oriented file format

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Patch Available
          links to

          Web Link Pig load/store support for Trevni format

          Activity

            People

              mubarakseyed Mubarak Seyed
              mubarakseyed Mubarak Seyed
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: