Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22366

Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      There's an existing flag "spark.sql.files.ignoreCorruptFiles" that will quietly ignore attempted reads from files that have been corrupted, but it still allows the query to fail on missing files. Being able to ignore missing files too is useful in some replication scenarios.

      We should add a "spark.sql.files.ignoreMissingFiles" to fill out the functionality.

        Attachments

          Activity

            People

            • Assignee:
              joseph.torres Jose Torres
              Reporter:
              joseph.torres Jose Torres
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: