Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1272

Don't fail job if some local directories are buggy

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Shuffle
    • Labels:
      None

      Description

      If Spark cannot create shuffle directories inside of a local directory it might make sense to just log an error and continue, provided that at least one valid shuffle directory exists. Otherwise if a single disk is wonky the entire job can fail.

      The down side is that this might mask failures if the person actually misconfigures the local directories to point to the wrong disk(s).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              pwendell Patrick Wendell
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: