Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19615

Provide Dataset union convenience for divergent schema

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.1.0
    • None
    • SQL

    Description

      Creating a union DataFrame over two sources that have different schema definitions is surprisingly complex. Provide a version of the union method that will create a infer a target schema as the result of merging the sources. Automatically add extend either side with null columns for any missing columns that are nullable.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21316 Dataset Union output is not consistent with the column sequence

          • Major - Major loss of function.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-21043 Add unionByName API to Dataset

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              ndimiduk Nick Dimiduk
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: