Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18103

High performance vectored read API in Hadoop

    XMLWordPrintableJSON

Details

    Description

      Add support for multiple ranged vectored read api in PositionedReadable. The default iterates through the ranges to read each synchronously, but the intent is that FSDataInputStream subclasses can make more efficient readers especially object stores implementation.

      Attachments

        1. Vectored Read API for Hadoop FS.pdf
          463 kB
          Mukund Thakur

        Issue Links

          causes

          Sub-task - The sub-task of the issue HADOOP-18854 add options to disable range merging of vectored io

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18322 Yetus build failure in branch-3.3.

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. ORC-1787 Failure on readDiskRangesVectored with concurrent record readers

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-18287 Provide a shim library for modern FS APIs

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. PARQUET-2171 Implement vectored IO in parquet file format

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18855 VectorIO API tuning/stabilization

          • Major - Major loss of function.
          • In Progress

          Improvement - An improvement or enhancement to an existing feature or task. ORC-1570 Add `supportVectoredIO` API to `HadoopShimsCurrent` and use it

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDDS-7298 Ozone support for VectorIO APIs in Hadoop

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #4476

          There are no Sub-Tasks for this issue.

          Activity

            People

              mthakur Mukund Thakur
              mthakur Mukund Thakur
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 33h
                  33h