Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23514

Replace spark.sparkContext.hadoopConfiguration by spark.sessionState.newHadoopConf()

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.4.0
    • SQL
    • None

    Description

      Check all the places where we directly use spark.sparkContext.hadoopConfiguration. Instead, in some scenarios, it makes more sense to call spark.sessionState.newHadoopConf() which blends in settings from SQLConf.

      Attachments

        Activity

          People

            juliuszsompolski Juliusz Sompolski
            smilegator Xiao Li
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: