Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27589 Spark file source V2
  3. SPARK-23817

Create file source V2 framework and migrate ORC read path

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      Migrate ORC file format read path to data source V2. 

      Supports:

      1. Scan ColumnarBatch
      2. Scan UnsafeRow
      3. Push down filters
      4. Push down required columns

      Not supported( due to limitation of data source V2):

      1. Read multiple file path
      2. Read bucketed file.

       

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. SPARK-26865 DataSourceV2Strategy should push normalized filters

          • Major - Major loss of function.
          • Resolved
          is a child of

          Umbrella - An overarching type made of sub-tasks SPARK-23507 Migrate file-based data sources to data source v2

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #20933 [Github] Pull Request #20933 (gengliangwang)

          Web Link GitHub Pull Request #20933

          Web Link GitHub Pull Request #23383

          Web Link GitHub Pull Request #27816

          Activity

            People

              Gengliang.Wang Gengliang Wang
              Gengliang.Wang Gengliang Wang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: