Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41305 Connect Proto Completeness
  3. SPARK-41919

Unify the schema or datatype in protos

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Connect
    • None

    Description

      this ticket only focus on the protos sent from client to server.

      we normally use

        oneof schema {
      
          DataType datatype = 2;
      
          // Server will use Catalyst parser to parse this string to DataType.
          string datatype_str = 3;
        }
      

      to represent a schema or datatype.

      actually, we can simplify it with just a string. In the server, we can easily parse a DDL-formatted schema or a JSON formatted one.

        // (Optional) The schema of local data.
        // It should be either a DDL-formatted type string or a JSON string.
        //
        // The server side will update the column names and data types according to this schema.
        // If the 'data' is not provided, then this schema will be required.
        optional string schema = 2;
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            podongfeng Ruifeng Zheng
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: