Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19918

Use TextFileFormat in implementation of JsonFileFormat

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.0
    • Component/s: SQL
    • Labels:
      None

      Description

      If we use Dataset for initial loading when inferring the schema, there are advantages. Please refer SPARK-18362

      It seems JSON one was supposed to be fixed together but missed according to https://github.com/apache/spark/pull/15813

      A similar problem also affects the JSON file format and this patch originally fixed that as well, but I've decided to split that change into a separate patch so as not to conflict with changes in another JSON PR.

      Also, this affects some functionalities because it does not use FileScanRDD. This problem is described in SPARK-19885 (but it was CSV's case).

        Attachments

          Activity

            People

            • Assignee:
              hyukjin.kwon Hyukjin Kwon
              Reporter:
              hyukjin.kwon Hyukjin Kwon
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: