Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-46837 String function support (parent)
  3. SPARK-49204

Handle surrogate pairs properly

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 4.0.0
    • 4.0.0
    • SQL

    Description

      Fix the following string expressions to handle one-to-many case mapping properly:

      • StringReplace
      • StringInstr
      • StringLocate
      • SubstringIndex
      • StringTrim
      • StringTrimLeft
      • StringTrimRight

      ย 

      Examples of incorrect results (underย ICU collations):

      StringReplace("๐Ÿ˜„a", "a", "b") // returns: "๐Ÿ˜„ab" (incorrect), instead of: "๐Ÿ˜„b" (correct)

      ย 

      StringInstr("๐Ÿ˜„a", "a") // returns: 3 (incorrect), instead of: 2 (correct)

      ย 

      StringLocate("a", "๐Ÿ˜„a") // returns: 3 (incorrect), instead of: 2 (correct)

      ย 

      SubstringIndex("๐Ÿ˜„a", "a") // returns: "๐Ÿ˜„a" (incorrect), instead of: "๐Ÿ˜„" (correct)

      ย 

      StringTrim("๐Ÿ˜„", "๐Ÿ˜„") // returns: "๐Ÿ˜„" (incorrect), instead of: "" (correct)

      ย 

      StringTrimLeft("๐Ÿ˜„", "๐Ÿ˜„") // returns: "๐Ÿ˜„" (incorrect), instead of: "" (correct)

      ย 

      StringTrimRight("๐Ÿ˜„", "๐Ÿ˜„") // returns: "๐Ÿ˜„" (incorrect), instead of: "" (correct)

      Attachments

        Activity

          People

            uros-db Uroลก Bojaniฤ‡
            uros-db Uroลก Bojaniฤ‡
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: