Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16825

Replace hive.default.fileformat by spark.sql.default.fileformat

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 2.0.0
    • None
    • SQL
    • None

    Description

      Currently, we are using hive.default.fileformat for specifying the default file format in CREATE TABLE statement. Multiple issues exist:

      • This parameter value is not from `hive-site.xml`. Thus, even if users change the hive.default.fileformat in `hive-site.xml`, Spark will ignore it. To change the parameter values, users have to use Spark interface, (e.g., by a SET command or API).
      • This parameter is not documented.
      • This parameter value will not be sent to Hive metastore. It is being used by Spark internals when processing CREATE TABLE statement.
      • This parameter is case sensitive.

      Since this is being used by Spark only, it does not make sense to use a parameter starting from `hive`. we might follow the other Hive-related parameters and introduce a new Spark parameter here. It should be public. Thus, how about replacing `hive.default.fileformat` by `spark.sql.default.fileformat`. we also should make it case insensitive.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              smilegator Xiao Li
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: