Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27589 Spark file source V2
  3. SPARK-23817

Create file source V2 framework and migrate ORC read path

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      Migrate ORC file format read path to data source V2. 

      Supports:

      1. Scan ColumnarBatch
      2. Scan UnsafeRow
      3. Push down filters
      4. Push down required columns

      Not supported( due to limitation of data source V2):

      1. Read multiple file path
      2. Read bucketed file.

       

      Attachments

        Issue Links

        causes

        Bug - A problem which impairs or prevents the functions of the product. SPARK-26865 DataSourceV2Strategy should push normalized filters

        • Major - Major loss of function.
        • Resolved
        is a child of

        Umbrella - An overarching type made of sub-tasks SPARK-23507 Migrate file-based data sources to data source v2

        • Major - Major loss of function.
        • Resolved
        links to

        Pull request #20933 [Github] Pull Request #20933 (gengliangwang)

        Web Link GitHub Pull Request #20933

        Web Link GitHub Pull Request #23383

        Web Link GitHub Pull Request #27816

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Gengliang.Wang Gengliang Wang
            Gengliang.Wang Gengliang Wang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment