Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12474

support deserialization for physical plan and hive logical plan from JSON string

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Incomplete
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:

      Description

      SPARK-12321 add a framework based on reflection that can serialize TreeNode to JSON and deserialize it back. However, it can't handle all corner cases and we bypass them in the test, see https://github.com/apache/spark/pull/10311/files#diff-238d584c15e16c24f49a40bcf163fe13R190 and https://github.com/apache/spark/pull/10311/files#diff-238d584c15e16c24f49a40bcf163fe13R212.

      known corner cases:
      1. ExpressionEncoder
      2. BaseRelation
      3. hive logical plan
      4. physical plan

      The framework is in catalyst module and may not be able to handle corner cases from other modules, an idea is defining a JsonSerializable trait and implement it for corner cases.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cloud_fan Wenchen Fan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: