Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27589 Spark file source V2
  3. SPARK-27136

Remove data source option check_files_exist

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      The data source option check_files_exist is introduced in In https://github.com/apache/spark/pull/23383 when the file source V2 framework is implemented. In the PR, FileIndex was created as a member of FileTable, so that we could implement partition pruning like 0f9fcab in the future. At that time FileIndexes will always be created for file writes, so we needed the option to decide whether to check file existence.

      After https://github.com/apache/spark/pull/23774, the option is not needed anymore. This PR is to clean the option.

      Attachments

        Issue Links

          Activity

            People

              Gengliang.Wang Gengliang Wang
              Gengliang.Wang Gengliang Wang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: