Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27589 Spark file source V2
  3. SPARK-23817

Create file source V2 framework and migrate ORC read path

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      Migrate ORC file format read path to data source V2. 

      Supports:

      1. Scan ColumnarBatch
      2. Scan UnsafeRow
      3. Push down filters
      4. Push down required columns

      Not supported( due to limitation of data source V2):

      1. Read multiple file path
      2. Read bucketed file.

       

      Attachments

        Issue Links

          Activity

            People

              Gengliang.Wang Gengliang Wang
              Gengliang.Wang Gengliang Wang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: