Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41279 Feature parity: DataFrame API in Spark Connect
  3. SPARK-40586

Decouple plan transformation and validation on server side

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Connect
    • None

    Description

      Project connect, from some perspectives, can be thought as replacing the SQL parser to generate a parsed (but the difference that is unresolved) plan, then the plan is passed to the analyzer. This means that connect should also do validation on the proto as there are many in-validate parser cases that analyzer does not expect to see, which potentially could cause problems if connect only pass through the proto (of course have it translated) to analyzer.

      Meanwhile I think this is a good idea to decouple the validation and transformation so that we have two stages:
      stage 1: proto validation. For example validate if necessary fields are populated or not.
      stage 2: transformation, which convert the proto to a plan with assumption that the plan is valid parsed version of the plan.

      Attachments

        Activity

          People

            Unassigned Unassigned
            amaliujia Rui Wang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: