Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23094

Json Readers choose wrong encoding when bad records are present and fail

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.1
    • 2.4.0
    • SQL
    • None

    Description

      The cases described in SPARK-16548 and SPARK-20549 handled the JsonParser code paths for expressions but not the readers. We should also cover reader code paths reading files with bad characters.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-23428 Revert [SPARK-23094] Fix invalid character handling in JsonDataSource

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-20549 java.io.CharConversionException: Invalid UTF-32 in JsonToStructs

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-16548 java.io.CharConversionException: Invalid UTF-32 character prevents me from querying my data

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Pull request #20302 [Github] Pull Request #20302 (brkyvz)

          Pull request #20614 [Github] Pull Request #20614 (gatorsmile)

          Pull request #20937 [Github] Pull Request #20937 (MaxGekk)

          Pull request #21254 [Github] Pull Request #21254 (gatorsmile)

          Activity

            People

              maxgekk Max Gekk
              brkyvz Burak Yavuz
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: