Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41279 Feature parity: DataFrame API in Spark Connect
  3. SPARK-40586

Decouple plan transformation and validation on server side

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Connect

    Description

      Project connect, from some perspectives, can be thought as replacing the SQL parser to generate a parsed (but the difference that is unresolved) plan, then the plan is passed to the analyzer. This means that connect should also do validation on the proto as there are many in-validate parser cases that analyzer does not expect to see, which potentially could cause problems if connect only pass through the proto (of course have it translated) to analyzer.

      Meanwhile I think this is a good idea to decouple the validation and transformation so that we have two stages:
      stage 1: proto validation. For example validate if necessary fields are populated or not.
      stage 2: transformation, which convert the proto to a plan with assumption that the plan is valid parsed version of the plan.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              amaliujia Rui Wang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: