Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9874

UnionAll operation on DataFrame doesn't check for column names

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.4.0
    • None
    • SQL
    • None

    Description

      UnionAll operation in dataFrame checks only for the column dataType. For example if df1 has a field id of type String and df2 has a field city of type String then, unionAll appends both dataFrames one after another.
      This should not be allowed. Either it should create combined schema or it should throw error.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. SPARK-9813 Incorrect UNION ALL behavior

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-15918 unionAll returns wrong result when two dataframes has schema in different order

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              raghavendra.pandey Raghavendra Kumar Pandey
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: