Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1343

add an interface in RCFile to support concatenation of two files without (de)compression

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.6.0
    • 0.8.0
    • None

    Description

      If two files are concatenated, we need to read each record in these files and write them back to the destination file. The IO cost is mostly unavoidable due to the lack of append functionality in HDFS. However the CPU cost could be significantly reduced by avoiding compression and decompression of the files.

      The File Format layer should provide API that implement the block-level concatenation.

      Attachments

        1. HIVE-1343.1.patch
          5 kB
          He Yongqiang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            he yongqiang He Yongqiang Assign to me
            nzhang Ning Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment