Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37344

split function behave differently between spark 2.3 and spark 3.2

    XMLWordPrintableJSON

Details

    • Bug
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.1.1, 3.1.2, 3.2.0
    • None
    • SQL

    Description

      while use split function in sql, it behave differently between 2.3 and 3.2, which cause incorrect problem.

      we can use this sql to reproduce this problem:

       

      create table split_test ( id int,name string)

      insert into split_test values(1,"abc;def")

      explain extended select split(name,'\\\\;') from split_test

       

      spark3:

      spark-sql> Explain extended select split(name,'\\\\;') from split_test;

      == Parsed Logical Plan ==

      'Project [unresolvedalias('split('name, \\;), None)]

      +- 'UnresolvedRelation [split_test], [], false

       

      spark2:

       

      spark-sql> Explain extended select split(name,'\\\\;') from split_test;

      == Parsed Logical Plan ==

      'Project [unresolvedalias('split('name, \;), None)]

      +- 'UnresolvedRelation split_test

       

      It looks like the deal of escape is different

      Attachments

        Activity

          People

            Unassigned Unassigned
            oceaneast ocean
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: