Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-2171

Implement vectored IO in parquet file format

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.14.0
    • parquet-mr
    • None

    Description

      We recently added a new feature called vectored IO in Hadoop for improving read performance for seek heavy readers. Spark Jobs and others which uses parquet will greatly benefit from this api. Details can be found here 

      https://github.com/apache/hadoop/commit/e1842b2a749d79cbdc15c524515b9eda64c339d5

      https://issues.apache.org/jira/browse/HADOOP-18103

      https://issues.apache.org/jira/browse/HADOOP-11867

      Attachments

        Issue Links

          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. PARQUET-2277 Bump hadoop.version from 3.2.3 to 3.3.5

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          New Feature - A new feature of the product, which has yet to be developed. SPARK-44116 Utilize Hadoop vectorized APIs

          • Major - Major loss of function.
          • Open
          is related to

          Sub-task - The sub-task of the issue HADOOP-19101 Vectored Read into off-heap buffer broken in fallback implementation

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-19098 Vector IO: consistent specified rejection of overlapping ranges

          • Major - Major loss of function.
          • Resolved
          is superceded by

          Improvement - An improvement or enhancement to an existing feature or task. PARQUET-2486 Improve Parquet IO Performance within cloud datalakes

          • Major - Major loss of function.
          • In Progress
          relates to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-18103 High performance vectored read API in Hadoop

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #1103

          Web Link GitHub Pull Request #1139

          Web Link GitHub Pull Request #1330

          Activity

            People

              stevel@apache.org Steve Loughran
              mthakur Mukund Thakur
              Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: