Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22938

Assert that SQLConf.get is accessed only on the driver.

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.1
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Assert if code tries to access SQLConf.get on executor.
      This can lead to hard to detect bugs, where the executor will read fallbackConf, falling back to default config values, ignoring potentially changed non-default configs.
      If a config is to be passed to executor code, it needs to be read on the driver, and passed explicitly.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                juliuszsompolski Juliusz Sompolski
                Reporter:
                juliuszsompolski Juliusz Sompolski
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: