Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-42774

Expose VectorTypes API for DataSourceV2 Batch Scans

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.3.2
    • None
    • SQL
    • None

    Description

      SparkPlan's vectorType's attribute can be used to specialize codegen however BatchScanExecBase does not override this so we DSv2 sources do not get any benefit of concrete class dispatch.

      This proposes adding an override to BatchScanExecBase which delegates to a new default method on PartitionReaderFactory to expose vectoryTypes:

      {{
      default Optional<Iterable<String>> getVectorTypes()

      { return Optional.empty(); }

      }}

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            emkornfield Micah Kornfield
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: