Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34679

inferTimestamp option is missing from the list of options in DataFrameReader.json.

    XMLWordPrintableJSON

    Details

      Description

      inferTimestamp option is missing in the list of options in DataFrameReader.json method in the API docs missing from the [Scaladocs here|DataFrameReader.json].

      Simiarly in the [Pyspark docs|pyspark.sql.DataFrameReader.json] as well.

      However we have this blurb in the [migration guide|Spark 3.0 to 3.0.1 migration guide]

      • In Spark 3.0, JSON datasource and JSON function schema_of_json infer TimestampType from string values if they match to the pattern defined by the JSON option timestampFormat. Since version 3.0.1, the timestamp type inference is disabled by default. Set the JSON option inferTimestamp to true to enable such type inference.

      We should add this in the documentation as well as there is a possibility that the Data Engineers might not be aware of this option.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              P7hB Prashanth Babu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 0.5h
                0.5h
                Remaining:
                Remaining Estimate - 0.5h
                0.5h
                Logged:
                Time Spent - Not Specified
                Not Specified