Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-77

Improvements in ByteBuffer read path

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersStop watchingWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.9.0
    • parquet-mr
    • None

    Description

      For Apache Drill, we are looking to pass in a buffer that we have already allocated (in this case from Direct memory), wrapped in a ByteBuffer.
      The current effort to allow a ByteBuffer read path is great except that the interface allocates the memory and there is no way for an application to pass in memory that has been allocated or (even better) to provide an allocator.
      Additionally, it would be great to be able to use the same approach while decompressing.
      As a starting point here is a patch on top of the ByteBuffer read effort that adds a function in CompatibilityUtils and also adds a ByteBuffer path for the Snappy Decompressor. The latter requires Hadoop 2.3 though, so some discussion on this would be called for.

      Please let me have any feedback and I can make changes/additions.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jaltekruse Jason Altekruse Assign to me
            parthc Parth Chandra
            Votes:
            0 Vote for this issue
            Watchers:
            6 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment