Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3585

Integrate Trevni as another columnar oriented file format

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Minor
    • Resolution: Unresolved
    • 0.10.0
    • None
    • Serializers/Deserializers
    • None

    Description

      add new avro module trevni as another columnar format.New columnar format need a columnar SerDe,seems fastutil is a good choice.the shark project use fastutil library as columnar serde library but it seems too large (almost 15m) for just a few primitive array collection.

      Attachments

        1. futurama_episodes.avro
          3 kB
          Mark Wagner
        2. HIVE-3585.1.patch.txt
          55 kB
          Mark Wagner
        3. HIVE-3585.2.patch.txt
          77 kB
          Sean Busbey
        4. HIVE-3585.3.patch.txt
          77 kB
          Edward Capriolo

        Issue Links

        is blocked by

        Bug - A problem which impairs or prevents the functions of the product. HIVE-5302 PartitionPruner logs warning on Avro non-partitioned data

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HIVE-3953 Reading of partitioned Avro data fails because of missing properties

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HIVE-4789 FetchOperator fails on partitioned Avro data

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed
        is related to

        New Feature - A new feature of the product, which has yet to be developed. HIVE-895 Add SerDe for Avro serialized data

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. AVRO-1229 Trevni should support Boolean fields

        • Major - Major loss of function.
        • Closed
        relates to

        New Feature - A new feature of the product, which has yet to be developed. FLUME-1669 Add support for columnar event serializer in HDFS

        • Major - Major loss of function.
        • Open

        New Feature - A new feature of the product, which has yet to be developed. AVRO-806 add a column-major file format

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. AVRO-1241 improve trevni performance on string deserialization

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. PIG-3015 Rewrite of AvroStorage

        • Major - Major loss of function.
        • Closed
        There are no Sub-Tasks for this issue.

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mwagner Mark Wagner Assign to me
            gemini5201314 alex gemini
            Votes:
            14 Vote for this issue
            Watchers:
            31 Start watching this issue

            Dates

              Created:
              Updated:

              Slack

                Issue deployment